Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c7643fb4d0 | ||
|
|
882a9f6808 | ||
|
|
aa57c1a894 | ||
|
|
2a48751283 | ||
|
|
6083903ccb | ||
|
|
ee9eb353fb | ||
|
|
df597bb1b1 | ||
|
|
23ad0ca317 | ||
|
|
71d5804eaf | ||
|
|
5ef141eb17 | ||
|
|
fadbe10420 | ||
|
|
17ea2f4110 | ||
|
|
603513a15e | ||
|
|
5e0592381a | ||
|
|
ef89f77ea7 | ||
|
|
9e2584ec8b | ||
|
|
a5dfc1ee99 | ||
|
|
7e72126a07 | ||
|
|
80a65b013f | ||
|
|
54ec72da17 | ||
|
|
d82b8efdd7 | ||
|
|
27b3c24065 | ||
|
|
7e9bfbc2be | ||
|
|
f5410bd253 | ||
|
|
98522cecba | ||
|
|
3138e782bd | ||
|
|
393b77dfd2 | ||
|
|
cfdd3435bc | ||
|
|
f3a6b4f715 | ||
|
|
5c888ff948 | ||
|
|
03e0d84380 | ||
|
|
a5603d8f01 | ||
|
|
988c4899d4 | ||
|
|
77e9973ff5 | ||
|
|
625263dd48 | ||
|
|
42bc482ed8 | ||
|
|
aea24353ad | ||
|
|
d1493de31a | ||
|
|
93725f42b3 | ||
|
|
f80148c5db | ||
|
|
541067e6b2 | ||
|
|
8f0b1982c2 | ||
|
|
a405c5ed35 | ||
|
|
b232033b11 | ||
|
|
4a33898a6c | ||
|
|
21c98bd485 | ||
|
|
a651c81dfb | ||
|
|
0951d909c9 | ||
|
|
69effb542d | ||
|
|
e9314ea1ee | ||
|
|
ad9563412d | ||
|
|
f71c7ccce4 | ||
|
|
f541566714 | ||
|
|
006bbcef28 | ||
|
|
8889a4ceff | ||
|
|
56dfc6d0b9 | ||
|
|
f1ae1ddb36 | ||
|
|
c21c48e61e | ||
|
|
22cd17d064 | ||
|
|
48ef6d34cd | ||
|
|
317f36b456 | ||
|
|
7a8c6bd2e3 | ||
|
|
3c035ccef3 | ||
|
|
a6ec644ecb | ||
|
|
b7f4b7e148 | ||
|
|
ed1a20a960 | ||
|
|
511d8dca93 | ||
|
|
29aefc3b6f | ||
|
|
2e69215920 | ||
|
|
b93e8481f9 | ||
|
|
51ed5c0b2a | ||
|
|
9f9043c037 | ||
|
|
a25f139281 | ||
|
|
52600b28ab | ||
|
|
7cf388367e | ||
|
|
13830865f6 | ||
|
|
010b9affc7 | ||
|
|
ed5eed6d54 | ||
|
|
14f340370c | ||
|
|
50e0c5a786 | ||
|
|
649bbafa17 | ||
|
|
b7c19592e5 | ||
|
|
6c115c1394 | ||
|
|
6d79709cc0 | ||
|
|
0336f42e2a | ||
|
|
0f4a527337 | ||
|
|
1d92627935 |
No files matched your search
@@ -227,9 +227,18 @@ See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
|
|||||||
|
|
||||||
## External Exposure Policy (MANDATORY)
|
## External Exposure Policy (MANDATORY)
|
||||||
|
|
||||||
Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
|
Every Caddy-proxied service is in exactly one of three postures: behind the
|
||||||
Authentik embedded outpost) or explicitly internal-only — no public DNS name,
|
auth chain (Anubis :8923 → Authentik embedded outpost), **exempt by
|
||||||
no vhost. New services default to internal-only until guarded.
|
construction** because it authenticates itself (`auth.`, git smart-HTTP, Gitea,
|
||||||
|
OpenWebUI, Jellyfin, Plex, Audiobookshelf, the *arr stack), or internal-only —
|
||||||
|
no public DNS name, no vhost. New services default to internal-only until guarded.
|
||||||
|
|
||||||
|
The guard chain is a **browser-only device**: Anubis demands a JavaScript
|
||||||
|
proof-of-work and the outpost works from a session cookie, so any non-browser
|
||||||
|
client — a native app, a podcast RSS reader, an OPDS client, `git`, a
|
||||||
|
service-to-service call — is answered with challenge HTML where it expected
|
||||||
|
JSON or a socket payload. A service that must serve such a client cannot sit
|
||||||
|
behind the chain; it has to own its authentication.
|
||||||
|
|
||||||
When guarding a new vhost — provider first, then route:
|
When guarding a new vhost — provider first, then route:
|
||||||
|
|
||||||
@@ -248,6 +257,10 @@ When guarding a new vhost — provider first, then route:
|
|||||||
Exempt categories (document the reason in the vhost comment): the IdP itself
|
Exempt categories (document the reason in the vhost comment): the IdP itself
|
||||||
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.
|
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.
|
||||||
|
|
||||||
|
Unguarding a vhost is the mirror image of the procedure above — the route flips
|
||||||
|
first and the IdP rows go last — and `serverAppConfigs-docs/Infrastructure/Caddy.md`
|
||||||
|
*Unguard a vhost* owns the steps.
|
||||||
|
|
||||||
## Deletion & Trash Policy (MANDATORY)
|
## Deletion & Trash Policy (MANDATORY)
|
||||||
|
|
||||||
Never `rm`. Move deletions into trash so they stay recoverable:
|
Never `rm`. Move deletions into trash so they stay recoverable:
|
||||||
@@ -735,7 +748,12 @@ linked runbooks.
|
|||||||
**Ops**
|
**Ops**
|
||||||
|
|
||||||
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
||||||
Use `pkill -x <exact-name>`.
|
Use `pkill -x <exact-name>`, or the bracket trick — `pkill -f "serve[-]script.sh"` —
|
||||||
|
when the pattern has to be named: the bracket makes the pattern stop matching the
|
||||||
|
command line that carries it. Quote it: a zsh remote glob-expands an unquoted `[s]`
|
||||||
|
and the kill silently never runs. Re-hit 2026-11-02 —
|
||||||
|
`ssh host 'pkill -f jfshowswatch.py; …'` killed the remote shell mid-command, and
|
||||||
|
ssh exits **255 with zero output**, which reads exactly like a network fault.
|
||||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
||||||
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
||||||
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
||||||
@@ -818,6 +836,7 @@ per-attempt page and no per-swap ledger anywhere else.
|
|||||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||||
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||||
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||||
|
| Duplicate films and the purge hold (Dankserver) | `serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md` |
|
||||||
|
|
||||||
There is no page for a retired endpoint and no change diary. A payload swap
|
There is no page for a retired endpoint and no change diary. A payload swap
|
||||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||||
@@ -860,6 +879,6 @@ history.
|
|||||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||||
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
|
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Deployed on Dankserver 2026-10-31, re-verified 2026-11-01: both quadlets bind the view, not the raw array. A restart for any reason discards that server's in-flight library scan, so re-trigger the refresh — see [[Media Metadata View]].** |
|
||||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||||
+33
-7
@@ -22,13 +22,16 @@
|
|||||||
# CHALLENGE everyone else) → Caddy's internal listener http://:8016 (auth
|
# CHALLENGE everyone else) → Caddy's internal listener http://:8016 (auth
|
||||||
# decision; Host headers survive every hop) → authenticated traffic is
|
# decision; Host headers survive every hop) → authenticated traffic is
|
||||||
# forwarded back to http://:8017 and routed straight to the app.
|
# forwarded back to http://:8017 and routed straight to the app.
|
||||||
# Apps with their own auth (Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) and the
|
# Apps with their own auth (Gitea, OpenWebUI, Jellyfin, Plex, Audiobookshelf,
|
||||||
# Authentik IdP itself (auth.franksdank.cloud — its login flow must never sit
|
# *arr, Pi-hole) and the Authentik IdP itself (auth.franksdank.cloud — its login
|
||||||
# behind a challenge) are exempt by construction: their vhosts never route
|
# flow must never sit behind a challenge) are exempt by construction: their
|
||||||
# through Anubis. ComfyUI left this exempt set on 2026-08-31 and Grafana left
|
# vhosts never route through Anubis. The chain is browser-only by nature — a
|
||||||
# it on 2026-09-01 (OAuth2 relying-party mode against authentik): both now sit
|
# native app client, a podcast RSS reader or a git client solves no JavaScript
|
||||||
# behind the guard chain; Grafana authenticates users itself via the OAuth2
|
# proof, so anything that must serve one has to own its own authentication.
|
||||||
# code flow, so no identity header is forwarded to it.
|
# ComfyUI left this exempt set on 2026-08-31 and Grafana left it on 2026-09-01
|
||||||
|
# (OAuth2 relying-party mode against authentik): both now sit behind the guard
|
||||||
|
# chain; Grafana authenticates users itself via the OAuth2 code flow, so no
|
||||||
|
# identity header is forwarded to it.
|
||||||
# The pihole vhost keeps its basicauth edge; the hash placeholder below is
|
# The pihole vhost keeps its basicauth edge; the hash placeholder below is
|
||||||
# rendered by deploy.sh from CADDY_BASICAUTH_HASH (host .env, git-ignored) into
|
# rendered by deploy.sh from CADDY_BASICAUTH_HASH (host .env, git-ignored) into
|
||||||
# /home/dank/Containers/caddy/Caddyfile, which the container mounts.
|
# /home/dank/Containers/caddy/Caddyfile, which the container mounts.
|
||||||
@@ -264,6 +267,29 @@ files.franksdank.cloud {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# --- Audiobookshelf (exempt by construction: own auth) --------------------
|
||||||
|
# Caddy → app: no Anubis, no outpost. The guard chain answers a native client's
|
||||||
|
# /ping, /api/login and /socket.io/ handshake with challenge HTML, so behind it
|
||||||
|
# the phone app parsed a non-object where the socket `init` payload belongs and
|
||||||
|
# podcast RSS/OPDS feeds broke the same way. ABS authenticates itself (local
|
||||||
|
# accounts plus authentik OIDC), so it joins jelly/plex/*arr outside the chain
|
||||||
|
# (2026-11-03); the IdP rows that guarded this name — proxy provider
|
||||||
|
# `audiobooks-proxy` and application "Audiobookshelf Guard" — are retired with
|
||||||
|
# the change, after the route flips. Accepted exposure: /audiobookshelf/api/login
|
||||||
|
# is internet-visible with no rate limiter on either side (this Caddy build has
|
||||||
|
# no rate_limit module; ABS 2.37.2 ships no lockout) — the same class every other
|
||||||
|
# own-auth vhost here lives with, so shared accounts need long unique passwords.
|
||||||
|
# ROUTER_BASE_PATH=/audiobookshelf, so the bare root is a convenience hop onto it.
|
||||||
|
# X-Forwarded-Proto must say https: ABS derives its OIDC origin from
|
||||||
|
# x-forwarded-proto://Host and this hop is plain http, so without the pin its own
|
||||||
|
# https callback is a 400 "Invalid callback URL - must be same-origin".
|
||||||
|
audiobooks.franksdank.cloud {
|
||||||
|
redir / /audiobookshelf/ permanent
|
||||||
|
reverse_proxy 192.168.0.235:8000 {
|
||||||
|
header_up X-Forwarded-Proto https
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
# --- Jellyfin ---
|
# --- Jellyfin ---
|
||||||
jelly.franksdank.cloud {
|
jelly.franksdank.cloud {
|
||||||
reverse_proxy 192.168.0.235:8096
|
reverse_proxy 192.168.0.235:8096
|
||||||
|
|||||||
@@ -26,6 +26,7 @@
|
|||||||
192.168.0.143 romm.franksdank.cloud
|
192.168.0.143 romm.franksdank.cloud
|
||||||
192.168.0.143 files.franksdank.cloud
|
192.168.0.143 files.franksdank.cloud
|
||||||
192.168.0.143 jelly.franksdank.cloud
|
192.168.0.143 jelly.franksdank.cloud
|
||||||
|
192.168.0.143 audiobooks.franksdank.cloud
|
||||||
192.168.0.143 radarr.franksdank.cloud
|
192.168.0.143 radarr.franksdank.cloud
|
||||||
192.168.0.143 lidarr.franksdank.cloud
|
192.168.0.143 lidarr.franksdank.cloud
|
||||||
192.168.0.143 readarr.franksdank.cloud
|
192.168.0.143 readarr.franksdank.cloud
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# Quadlet unit: Audiobookshelf audiobook server (rootless, boot-persistent).
|
||||||
|
#
|
||||||
|
# Install path on dankserver: ~/.config/containers/systemd/audiobookshelf.container
|
||||||
|
# Unit name: audiobookshelf.service (podman 5.x drops the container- prefix)
|
||||||
|
#
|
||||||
|
# Everything this server writes is its own state — the SQLite library, accounts,
|
||||||
|
# playback progress, cached covers — and it lives under
|
||||||
|
# /home/dank/Containers/audiobookshelf on the NVMe, so no metadata byte ever
|
||||||
|
# lands on the spindle. The audio itself is bound READ-ONLY off the array: the
|
||||||
|
# only sanctioned write path into /mnt/entropy is the overlay view
|
||||||
|
# (scripts/media-view.sh), Audiobookshelf has no need of it, and a read-only bind
|
||||||
|
# makes a stray sidecar write structurally impossible rather than discouraged.
|
||||||
|
#
|
||||||
|
# The digest is the linux/amd64 manifest of the tag, which is what podman
|
||||||
|
# verifies and what RepoDigests records on this single-arch host. Pinning the
|
||||||
|
# manifest-list digest instead would let a re-pull resolve to a different image.
|
||||||
|
# The upstream version tags stop at 2.19.0 (built 2025-02-02) while `latest` is
|
||||||
|
# newer by a year and a half, so `latest@digest` is the only pin that names the
|
||||||
|
# build actually running — same reasoning as the jellyfin and romm units.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Audiobookshelf audiobook server container
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Container]
|
||||||
|
ContainerName=audiobookshelf
|
||||||
|
Image=ghcr.io/advplyr/audiobookshelf:latest@sha256:37c2057e150694970442579815c646fe6bdc01c95b9d1547c4c9068be8844ef3
|
||||||
|
Network=pasta
|
||||||
|
# The names are the image's own: the server reads PORT, CONFIG_PATH, METADATA_PATH
|
||||||
|
# and ROUTER_BASE_PATH (measured against the digest below — `podman inspect` shows
|
||||||
|
# PORT=80 CONFIG_PATH=/config METADATA_PATH=/metadata in the image ENV, and the
|
||||||
|
# server's source greps for exactly these). The ABS_* family printed by the server
|
||||||
|
# README is the npm package's naming and is INERT here: with ABS_PORT=8000 the
|
||||||
|
# server listened on :80 under base path /audiobookshelf, so the unit sat
|
||||||
|
# `active (running)` while every probe to :8000 was refused (measured 2026-11-02,
|
||||||
|
# first deploy).
|
||||||
|
# ROUTER_BASE_PATH MUST be /audiobookshelf: the SPA in /app/client/dist/index.html is
|
||||||
|
# built with `routerBasePath:"/audiobookshelf"` baked in, so it asks for its assets under
|
||||||
|
# that prefix no matter what the server is mounted at. With `/` the server answers the
|
||||||
|
# page but every script request falls through to the SPA fallback and comes back as
|
||||||
|
# text/html — Chrome refuses to execute it and the UI never leaves "Loading…"
|
||||||
|
# (measured 2026-11-03; the API and /healthcheck were fine, which is why an API-only
|
||||||
|
# verification missed it).
|
||||||
|
Environment=PORT=8000
|
||||||
|
Environment=ROUTER_BASE_PATH=/audiobookshelf
|
||||||
|
Environment=CONFIG_PATH=/config
|
||||||
|
Environment=METADATA_PATH=/metadata
|
||||||
|
Environment=TZ=America/Vancouver
|
||||||
|
PublishPort=8000:8000
|
||||||
|
Volume=/home/dank/Containers/audiobookshelf/config:/config
|
||||||
|
Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata
|
||||||
|
Volume=/mnt/entropy/Audiobooks:/audiobooks:ro
|
||||||
|
# The image is Alpine: it ships wget and no curl, so the health check must name
|
||||||
|
# the binary that exists or the container is unhealthy while serving perfectly.
|
||||||
|
# /healthcheck answers `OK` at the root AND under the base path (measured on both
|
||||||
|
# base paths), so the probe below is correct whatever ROUTER_BASE_PATH says.
|
||||||
|
HealthCmd=wget -q -O /dev/null http://127.0.0.1:8000/healthcheck
|
||||||
|
HealthInterval=30s
|
||||||
|
HealthTimeout=10s
|
||||||
|
HealthRetries=3
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
# Refuse to start until the array is a real mount — with the bind source missing,
|
||||||
|
# podman invents an empty directory and Audiobookshelf records an empty library as
|
||||||
|
# truth. See the header of scripts/wait-for-mount.sh for the blind-boot incident.
|
||||||
|
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /mnt/entropy
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
# The first boot scans the whole audiobook tree. Weighted LOW (default 100) so a
|
||||||
|
# scan loses to playback when the array is contended. Proportional share, not a
|
||||||
|
# cap: inert while the disk is idle.
|
||||||
|
IOWeight=10
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
Executable
+309
@@ -0,0 +1,309 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# audiobookshelf-init.sh — make an Audiobookshelf server usable, from nothing.
|
||||||
|
#
|
||||||
|
# Why this is a script and not a browser session (live 2026-11-02, Dankserver):
|
||||||
|
# a freshly deployed Audiobookshelf answers /healthcheck with 200 while it has no
|
||||||
|
# user, no library and no scan — "healthy" says nothing about usefulness, and a
|
||||||
|
# service that only works after someone remembers to click through a first-run
|
||||||
|
# wizard is not deployed, it is installed. Everything below is the API the React
|
||||||
|
# client itself calls, so the same procedure runs unattended and lands in git.
|
||||||
|
#
|
||||||
|
# The API facts this script depends on, read off the pinned image rather than
|
||||||
|
# guessed (dist-server/server/Server.js inside the digest this repo pins):
|
||||||
|
# * the creation route is POST /init with {"newRoot":{"username","password"}}.
|
||||||
|
# There is NO /setup route in 2.37.x — POST /setup answers "Unauthorized",
|
||||||
|
# which reads exactly like an auth fault and is instead a missing route.
|
||||||
|
# * /status (no /api prefix) reports {"isInit": <has a root user>}; every route
|
||||||
|
# under /api needs the bearer token that POST /login returns.
|
||||||
|
# * the library count is NOT a field on the library DTO; the door is
|
||||||
|
# GET /api/libraries/{id}/items -> {"total": N}.
|
||||||
|
#
|
||||||
|
# The credential is generated once into a mode-600 host file and never enters
|
||||||
|
# the repo, the URL, or a process argument list. Same pattern as calibre-server.
|
||||||
|
#
|
||||||
|
# Single sign-on (authentik) is armed by the same run, and only when the credential
|
||||||
|
# file carries ABS_OIDC_CLIENT_SECRET, so a local-only deployment is unchanged by a
|
||||||
|
# re-run. These OIDC facts came from the same pinned image, and each one is a trap:
|
||||||
|
# * isOpenIDAuthSettingsValid demands issuer + authorization + token + userinfo +
|
||||||
|
# jwks + clientID + clientSecret + signingAlgorithm at once. Miss one and the
|
||||||
|
# server SILENTLY strips 'openid' from authActiveAuthMethods and falls back to
|
||||||
|
# ['local'] while answering 200 everywhere — so the only honest check is what
|
||||||
|
# /status says AFTER the write, which is what this script does.
|
||||||
|
# * the algorithm is HS256, not the image's RS256 default: no provider on the
|
||||||
|
# fleet's authentik carries a signing key, so the client secret IS the signature
|
||||||
|
# key (scripts/authentik-audiobookshelf-sso.sh). Left at RS256, every id_token
|
||||||
|
# fails verification and the failure looks like a broken IdP.
|
||||||
|
# * the auth block lives on PATCH /api/auth-settings. PATCH /api/settings has a
|
||||||
|
# 20-key allowlist (PATCHABLE_SETTINGS_KEYS) with NO auth key in it, so writing
|
||||||
|
# the OIDC config there answers 200 and changes nothing — measured on this host
|
||||||
|
# 2026-11-02. The keys /api/auth-settings accepts are exactly its own
|
||||||
|
# authenticationSettings getters; anything else is skipped with a debug line.
|
||||||
|
# * authOpenIDSubfolderForRedirectURLs must be sent explicitly. The 2.17.4 migration only
|
||||||
|
# fills it on a DB that ALREADY had OIDC on, so a database created before SSO leaves it
|
||||||
|
# UNDEFINED and the server builds redirect_uri
|
||||||
|
# "http://host/undefined/auth/openid/callback" — authentik answers the authorize request
|
||||||
|
# with "Redirect URI Error" (measured 2026-11-02). This script derives the value from
|
||||||
|
# ABS_URL's path, so it cannot drift from the unit's ROUTER_BASE_PATH.
|
||||||
|
# * 'local' stays in authActiveAuthMethods on purpose: if authentik is ever down,
|
||||||
|
# the root account must still be able to sign into its own server.
|
||||||
|
#
|
||||||
|
# usage: audiobookshelf-init.sh # create what is missing, then scan
|
||||||
|
# ABS_SCAN=0 audiobookshelf-init.sh # bootstrap without a scan
|
||||||
|
# ABS_OIDC=0 audiobookshelf-init.sh # never touch the auth methods
|
||||||
|
#
|
||||||
|
# environment (defaults are the deployed shape):
|
||||||
|
# ABS_URL server base URL INCLUDING the mounted base path
|
||||||
|
# (http://127.0.0.1:8000/audiobookshelf)
|
||||||
|
# ABS_SECRET credential file (/home/dank/.config/secrets/audiobookshelf)
|
||||||
|
# ABS_LIBRARY library name to ensure (Audiobooks)
|
||||||
|
# ABS_FOLDER folder to bind inside the server (/audiobooks)
|
||||||
|
# ABS_SCAN 1 triggers a scan (default), 0 skips it
|
||||||
|
# ABS_OIDC auto (default) arms SSO when the credential file has a client
|
||||||
|
# secret, 1 requires one, 0 leaves the auth methods alone
|
||||||
|
# ABS_OIDC_ISSUER authentik issuer URL for this client
|
||||||
|
# ABS_OIDC_CLIENT_ID provider client id (audiobookshelf)
|
||||||
|
# ABS_OIDC_ALG id_token signing algorithm the IdP actually uses (HS256)
|
||||||
|
# ABS_OIDC_SUBFOLDER override for the redirect-URI subfolder (default: ABS_URL's path)
|
||||||
|
#
|
||||||
|
# exit 0 every step is satisfied (created or already present)
|
||||||
|
# exit 1 a step was refused or the server misbehaved (named on stderr)
|
||||||
|
# exit 2 usage / unreadable credential file
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ABS_URL=${ABS_URL:-http://127.0.0.1:8000/audiobookshelf}
|
||||||
|
ABS_SECRET=${ABS_SECRET:-/home/dank/.config/secrets/audiobookshelf}
|
||||||
|
ABS_LIBRARY=${ABS_LIBRARY:-Audiobooks}
|
||||||
|
ABS_FOLDER=${ABS_FOLDER:-/audiobooks}
|
||||||
|
ABS_SCAN=${ABS_SCAN:-1}
|
||||||
|
ABS_OIDC=${ABS_OIDC:-auto}
|
||||||
|
ABS_OIDC_ISSUER=${ABS_OIDC_ISSUER:-https://auth.franksdank.cloud/application/o/audiobookshelf/}
|
||||||
|
ABS_OIDC_CLIENT_ID=${ABS_OIDC_CLIENT_ID:-audiobookshelf}
|
||||||
|
ABS_OIDC_ALG=${ABS_OIDC_ALG:-HS256}
|
||||||
|
|
||||||
|
die() { printf 'refused: %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
case "$ABS_SCAN" in
|
||||||
|
0 | 1) ;;
|
||||||
|
*) printf 'refused: ABS_SCAN wants 0 or 1, got %s\n' "$ABS_SCAN" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$ABS_OIDC" in
|
||||||
|
auto | 0 | 1) ;;
|
||||||
|
*) printf 'refused: ABS_OIDC wants auto, 0 or 1, got %s\n' "$ABS_OIDC" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# ---- credential -------------------------------------------------------------
|
||||||
|
umask 077
|
||||||
|
if [ ! -f "$ABS_SECRET" ]; then
|
||||||
|
mkdir -p "$(dirname "$ABS_SECRET")"
|
||||||
|
user=${ABS_ADMIN_USER:-frank}
|
||||||
|
# `tr … | head -c 24` is the obvious line and it is a trap under
|
||||||
|
# `set -o pipefail`: head exits at 24 bytes, tr takes SIGPIPE, the pipeline
|
||||||
|
# status is 141 and set -e kills the script before the credential is ever
|
||||||
|
# written (measured 2026-11-02). od reads a FIXED byte count and exits on its
|
||||||
|
# own, so no consumer closes a pipe early; bash slices the length.
|
||||||
|
pass=$(od -An -tx1 -N 32 /dev/urandom | tr -d ' \n')
|
||||||
|
pass=${pass:0:24}
|
||||||
|
[ "${#pass}" -eq 24 ] || die "could not generate a password (no /dev/urandom)"
|
||||||
|
printf 'ABS_ADMIN_USER=%s\nABS_ADMIN_PASSWORD=%s\n' "$user" "$pass" > "$ABS_SECRET"
|
||||||
|
printf 'credential: created %s\n' "$ABS_SECRET"
|
||||||
|
fi
|
||||||
|
[ -r "$ABS_SECRET" ] || die "$ABS_SECRET exists but is not readable by $(id -un)"
|
||||||
|
chmod 600 "$ABS_SECRET"
|
||||||
|
|
||||||
|
# ---- everything else, over HTTP ---------------------------------------------
|
||||||
|
# python3 rather than curl so the password and the bearer token never appear in
|
||||||
|
# any argv (they would be world-readable in /proc/<pid>/cmdline otherwise).
|
||||||
|
ABS_URL="$ABS_URL" ABS_SECRET="$ABS_SECRET" ABS_LIBRARY="$ABS_LIBRARY" \
|
||||||
|
ABS_FOLDER="$ABS_FOLDER" ABS_SCAN="$ABS_SCAN" ABS_OIDC="$ABS_OIDC" \
|
||||||
|
ABS_OIDC_ISSUER="$ABS_OIDC_ISSUER" ABS_OIDC_CLIENT_ID="$ABS_OIDC_CLIENT_ID" \
|
||||||
|
ABS_OIDC_ALG="$ABS_OIDC_ALG" python3 - <<'PY'
|
||||||
|
import json, os, sys, urllib.error, urllib.parse, urllib.request
|
||||||
|
|
||||||
|
BASE = os.environ["ABS_URL"].rstrip("/")
|
||||||
|
LIB = os.environ["ABS_LIBRARY"]
|
||||||
|
FOLDER = os.environ["ABS_FOLDER"]
|
||||||
|
SCAN = os.environ["ABS_SCAN"] == "1"
|
||||||
|
cred = {}
|
||||||
|
for line in open(os.environ["ABS_SECRET"]):
|
||||||
|
if "=" in line:
|
||||||
|
k, _, v = line.strip().partition("=")
|
||||||
|
cred[k] = v
|
||||||
|
user = cred.get("ABS_ADMIN_USER") or "frank"
|
||||||
|
password = cred.get("ABS_ADMIN_PASSWORD") or ""
|
||||||
|
if not password:
|
||||||
|
sys.exit("refused: %s has no ABS_ADMIN_PASSWORD" % os.environ["ABS_SECRET"])
|
||||||
|
|
||||||
|
|
||||||
|
def call(method, path, body=None, token=None, want_json=True):
|
||||||
|
data = None
|
||||||
|
headers = {"X-Requested-With": "audiobookshelf-init"}
|
||||||
|
if body is not None:
|
||||||
|
data = json.dumps(body).encode()
|
||||||
|
headers["Content-Type"] = "application/json"
|
||||||
|
if token:
|
||||||
|
headers["Authorization"] = "Bearer " + token
|
||||||
|
req = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=60) as r:
|
||||||
|
raw = r.read()
|
||||||
|
if not raw or not want_json:
|
||||||
|
return r.status, raw
|
||||||
|
try:
|
||||||
|
return r.status, json.loads(raw)
|
||||||
|
except ValueError:
|
||||||
|
# Express answers sendStatus() with the STATUS TEXT as the body:
|
||||||
|
# POST /init and the scan route both reply "OK" (measured on the
|
||||||
|
# pinned 2.37.2 image). A successful call must never be judged by
|
||||||
|
# whether its body happens to parse as JSON.
|
||||||
|
return r.status, raw.decode("utf-8", "replace")[:200]
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
raw = e.read()
|
||||||
|
try:
|
||||||
|
payload = json.loads(raw)
|
||||||
|
except Exception:
|
||||||
|
payload = raw.decode("utf-8", "replace")[:200]
|
||||||
|
return e.code, payload
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
# Not an HTTP answer at all: nothing listening, DNS dead, connection
|
||||||
|
# refused. HTTPError is a URLError subclass, so this arm must come after.
|
||||||
|
return 0, "unreachable (%s)" % e.reason
|
||||||
|
|
||||||
|
|
||||||
|
status, st = call("GET", "/status")
|
||||||
|
if status != 200 or not isinstance(st, dict):
|
||||||
|
sys.exit("refused: %s/status answered %s (%s) — is the service up?" % (BASE, status, st))
|
||||||
|
if st.get("isInit"):
|
||||||
|
print("account: already registered")
|
||||||
|
else:
|
||||||
|
code, body = call("POST", "/init", {"newRoot": {"username": user, "password": password}})
|
||||||
|
if code != 200:
|
||||||
|
sys.exit("refused: POST /init answered %s (%s)" % (code, body))
|
||||||
|
print("account: created (%s)" % user)
|
||||||
|
|
||||||
|
code, login = call("POST", "/login", {"username": user, "password": password})
|
||||||
|
if code != 200 or not isinstance(login, dict) or not login.get("user", {}).get("token"):
|
||||||
|
sys.exit("refused: login answered %s (%s) — the credential file does not match the server"
|
||||||
|
% (code, login))
|
||||||
|
token = login["user"]["token"]
|
||||||
|
print("token: obtained (%s)" % login["user"].get("type", "user"))
|
||||||
|
|
||||||
|
code, libs = call("GET", "/api/libraries", token=token)
|
||||||
|
if code != 200 or not isinstance(libs, dict) or "libraries" not in libs:
|
||||||
|
sys.exit("refused: GET /api/libraries answered %s (%s)" % (code, libs))
|
||||||
|
lib = next((l for l in libs["libraries"] if l.get("name") == LIB), None)
|
||||||
|
if lib:
|
||||||
|
print("library: exists %s (%s)" % (LIB, lib["id"]))
|
||||||
|
else:
|
||||||
|
body = {
|
||||||
|
"name": LIB,
|
||||||
|
"provider": "audiobook",
|
||||||
|
"icon": "book",
|
||||||
|
"folders": [{"fullPath": FOLDER}],
|
||||||
|
"settings": {
|
||||||
|
"coverEncoding": "jpg",
|
||||||
|
"skipMatchingMediaWithAsin": False,
|
||||||
|
"skipMatchingMediaWithIsbn": False,
|
||||||
|
"autoDetectCrossoverAudiobooks": False,
|
||||||
|
"ignoreAudiobookMetadata": False,
|
||||||
|
"useAudiobookFileMetadata": False,
|
||||||
|
},
|
||||||
|
"providerMetadata": {},
|
||||||
|
}
|
||||||
|
code, made = call("POST", "/api/libraries", body, token=token)
|
||||||
|
if code != 200:
|
||||||
|
sys.exit("refused: POST /api/libraries answered %s (%s)" % (code, made))
|
||||||
|
lib = made
|
||||||
|
print("library: created %s (%s) folder %s" % (LIB, lib["id"], FOLDER))
|
||||||
|
|
||||||
|
if SCAN:
|
||||||
|
code, body = call("POST", "/api/libraries/%s/scan" % lib["id"], {}, token=token,
|
||||||
|
want_json=False)
|
||||||
|
if code != 200:
|
||||||
|
sys.exit("refused: scan answered %s (%s)" % (code, body))
|
||||||
|
print("scan: accepted")
|
||||||
|
else:
|
||||||
|
print("scan: skipped (ABS_SCAN=0)")
|
||||||
|
|
||||||
|
code, items = call("GET", "/api/libraries/%s/items?limit=1" % lib["id"], token=token)
|
||||||
|
if code == 200 and isinstance(items, dict):
|
||||||
|
print("items: %s (a fresh scan raises this from 0; watch it, not /healthcheck)"
|
||||||
|
% items.get("total"))
|
||||||
|
|
||||||
|
# ---- single sign-on (authentik) --------------------------------------------------
|
||||||
|
# The provider this points at is provisioned by scripts/authentik-audiobookshelf-sso.sh;
|
||||||
|
# the two share exactly one value, the client secret in the credential file.
|
||||||
|
mode = os.environ["ABS_OIDC"]
|
||||||
|
oidc_secret = cred.get("ABS_OIDC_CLIENT_SECRET") or ""
|
||||||
|
if mode == "0":
|
||||||
|
print("oidc: skipped (ABS_OIDC=0)")
|
||||||
|
elif not oidc_secret:
|
||||||
|
if mode == "1":
|
||||||
|
sys.exit("refused: ABS_OIDC=1 but %s has no ABS_OIDC_CLIENT_SECRET — run "
|
||||||
|
"scripts/authentik-audiobookshelf-sso.sh first" % os.environ["ABS_SECRET"])
|
||||||
|
print("oidc: not configured (no ABS_OIDC_CLIENT_SECRET in the credential file)"
|
||||||
|
" — local auth only")
|
||||||
|
else:
|
||||||
|
client_id = os.environ["ABS_OIDC_CLIENT_ID"]
|
||||||
|
issuer = os.environ["ABS_OIDC_ISSUER"].rstrip("/") + "/"
|
||||||
|
# redirect_uri = <origin><subfolder>/auth/openid/callback, and the subfolder IS the
|
||||||
|
# base path the app is mounted under — taken from ABS_URL so unit and config agree.
|
||||||
|
subfolder = os.environ.get("ABS_OIDC_SUBFOLDER")
|
||||||
|
if subfolder is None:
|
||||||
|
subfolder = urllib.parse.urlparse(os.environ["ABS_URL"]).path.rstrip("/")
|
||||||
|
origin = issuer.split("/application/")[0]
|
||||||
|
body = {
|
||||||
|
"authActiveAuthMethods": ["local", "openid"],
|
||||||
|
"authOpenIDIssuerURL": issuer,
|
||||||
|
"authOpenIDAuthorizationURL": origin + "/application/o/authorize/",
|
||||||
|
"authOpenIDTokenURL": origin + "/application/o/token/",
|
||||||
|
"authOpenIDUserInfoURL": origin + "/application/o/userinfo/",
|
||||||
|
"authOpenIDJwksURL": issuer + "jwks/",
|
||||||
|
"authOpenIDLogoutURL": issuer + "end-session/",
|
||||||
|
"authOpenIDClientID": client_id,
|
||||||
|
"authOpenIDClientSecret": oidc_secret,
|
||||||
|
"authOpenIDTokenSigningAlgorithm": os.environ["ABS_OIDC_ALG"],
|
||||||
|
"authOpenIDSubfolderForRedirectURLs": subfolder,
|
||||||
|
"authOpenIDButtonText": "Sign in with Authentik",
|
||||||
|
# not auto-launch: the login page keeps offering the local form, which is the
|
||||||
|
# break-glass path when the IdP is unreachable.
|
||||||
|
"authOpenIDAutoLaunch": False,
|
||||||
|
# a household user's first SSO sign-in creates their account; the group claim
|
||||||
|
# decides the role, and a user in neither group is denied (fail closed).
|
||||||
|
"authOpenIDAutoRegister": True,
|
||||||
|
"authOpenIDGroupClaim": "abs_groups",
|
||||||
|
# The app's own scheme lives HERE, never in the IdP: ABS swaps it for
|
||||||
|
# <base>/auth/openid/mobile-redirect before the provider sees a redirect_uri.
|
||||||
|
"authOpenIDMobileRedirectURIs": ["audiobookshelf://oauth"],
|
||||||
|
}
|
||||||
|
code, res = call("PATCH", "/api/auth-settings", body, token=token)
|
||||||
|
if code != 200:
|
||||||
|
sys.exit("refused: PATCH /api/auth-settings answered %s (%s)" % (code, res))
|
||||||
|
# Verify twice: what was stored, and what the server will honour. A config that
|
||||||
|
# misses any of the eight required OIDC values is accepted with 200 and then
|
||||||
|
# silently dropped at the next load, leaving authMethods=['local'].
|
||||||
|
code, stored = call("GET", "/api/auth-settings", token=token)
|
||||||
|
if code != 200 or not isinstance(stored, dict):
|
||||||
|
sys.exit("refused: GET /api/auth-settings answered %s (%s)" % (code, stored))
|
||||||
|
for key, want in (("authOpenIDIssuerURL", issuer),
|
||||||
|
("authOpenIDTokenSigningAlgorithm", os.environ["ABS_OIDC_ALG"]),
|
||||||
|
("authOpenIDSubfolderForRedirectURLs", subfolder),
|
||||||
|
("authOpenIDButtonText", "Sign in with Authentik")):
|
||||||
|
if stored.get(key) != want:
|
||||||
|
sys.exit("refused: stored %s=%r, expected %r" % (key, stored.get(key), want))
|
||||||
|
if sorted(stored.get("authActiveAuthMethods") or []) != ["local", "openid"]:
|
||||||
|
sys.exit("refused: stored authActiveAuthMethods=%r, expected local+openid "
|
||||||
|
"('openid' is dropped when the block is incomplete)"
|
||||||
|
% stored.get("authActiveAuthMethods"))
|
||||||
|
code, after = call("GET", "/status")
|
||||||
|
methods = after.get("authMethods") if isinstance(after, dict) else None
|
||||||
|
if not isinstance(methods, list) or "openid" not in methods:
|
||||||
|
sys.exit("refused: /status reports authMethods=%s after a stored config that "
|
||||||
|
"looks complete — restart the service, the strategy is only armed at "
|
||||||
|
"load for a config that was invalid when the server read it" % methods)
|
||||||
|
print("oidc: armed (issuer %s | alg %s | claim abs_groups | redirect subfolder %r | "
|
||||||
|
"client %s | secret %s chars | authMethods %s)"
|
||||||
|
% (issuer, os.environ["ABS_OIDC_ALG"], subfolder, client_id, len(oidc_secret),
|
||||||
|
methods))
|
||||||
|
PY
|
||||||
|
exit 0
|
||||||
Executable
+264
@@ -0,0 +1,264 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# authentik-audiobookshelf-sso.sh — provision the authentik side of Audiobookshelf SSO.
|
||||||
|
#
|
||||||
|
# Why a script: this state lives in authentik's DATABASE, not in any repo file, and a
|
||||||
|
# blueprint re-import or a rebuild-from-scratch reverts hand-made provider rows
|
||||||
|
# silently. Same reasoning (and same door) as
|
||||||
|
# authentik-set-login-session-duration.sh: ssh into Dankclaw, `ak shell` from stdin.
|
||||||
|
# Re-run this after such a rebuild.
|
||||||
|
#
|
||||||
|
# The contract with Audiobookshelf, read off the pinned image (2.37.2,
|
||||||
|
# dist-server/server/auth/OidcAuthStrategy.js + objects/settings/ServerSettings.js)
|
||||||
|
# rather than a forum, because each of these is a dead-login trap:
|
||||||
|
#
|
||||||
|
# * Two URIs per origin ABS is reached at — <base>/auth/openid/callback and
|
||||||
|
# <base>/auth/openid/mobile-redirect — for EVERY entry in BASES (the LAN address
|
||||||
|
# and the public vhost). ABS SWAPS the app's own audiobookshelf://oauth for the
|
||||||
|
# mobile-redirect URL before the IdP ever sees a redirect_uri, so registering the
|
||||||
|
# app scheme with authentik — the obvious move — is what makes phone logins fail.
|
||||||
|
# The app scheme belongs in ABS's authOpenIDMobileRedirectURIs list (its built-in
|
||||||
|
# default), not here. The mobile URI exists on every origin because the app sends
|
||||||
|
# the one belonging to the door it was configured with: the vhost started serving
|
||||||
|
# non-browser clients on 2026-11-03 (it left the guard chain), so a phone pointed
|
||||||
|
# at the public name sends the public mobile-redirect URI, and STRICT matching
|
||||||
|
# answers an unregistered one with "Redirect URI Error".
|
||||||
|
# * <base> CARRIES THE BASE PATH. ABS builds redirect_uri as
|
||||||
|
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback, and that subfolder is
|
||||||
|
# the ROUTER_BASE_PATH the container is mounted at — /audiobookshelf, which is also the only
|
||||||
|
# base path the shipped SPA can boot under. Left unset (the fresh-DB default) the URL
|
||||||
|
# becomes "/undefined/…"; a provider registered without the prefix mismatches the same way
|
||||||
|
# (measured 2026-11-02, "Redirect URI Error"). audiobookshelf-init.sh derives its copy from
|
||||||
|
# ABS_URL, so the two sides each move with one variable.
|
||||||
|
# * ABS requests the scopes "openid profile email <GroupClaim> [<PermsClaim>]": the
|
||||||
|
# CLAIM NAME is a scope name. So authentik needs a scope mapping whose scope_name
|
||||||
|
# is exactly the claim name, or the claim never arrives and the callback dies on
|
||||||
|
# "Group claim <name> not found in userinfo".
|
||||||
|
# * the claim's VALUES must be ABS role names — the callback picks the first of
|
||||||
|
# ['admin','user','guest'] present in the list. Group names like
|
||||||
|
# audiobookshelf_admins are therefore translated by the mapping, never passed
|
||||||
|
# through. A user in neither group gets [] and is denied (fail closed).
|
||||||
|
# * tokens are HS256 here: no provider on this box carries a signing_key (census
|
||||||
|
# 2026-11-02), which makes the client secret the signature key. ABS's own default
|
||||||
|
# is RS256, so the ABS side MUST be set to HS256 or every id_token fails
|
||||||
|
# verification.
|
||||||
|
# * provider.authorization_flow must be non-NULL or /application/o/authorize/
|
||||||
|
# answers a 500 with 'NoneType' object has no attribute 'slug' (measured on
|
||||||
|
# grafana, same authentik build).
|
||||||
|
# * provider.grant_types MUST contain authorization_code. authentik 2026.8 defaults an
|
||||||
|
# ORM-created provider to the EMPTY list, and /application/o/authorize/ then refuses the
|
||||||
|
# code flow with oauthlib's generic "The request is otherwise malformed"; the only line
|
||||||
|
# that names the cause is the server log's 'Invalid grant_type for provider'. The
|
||||||
|
# redirect-URI check runs FIRST, so a provider carrying both faults shows only the URI
|
||||||
|
# error until the URI is fixed (measured 2026-11-02, where the login survived the URI fix
|
||||||
|
# and died here).
|
||||||
|
# * property_mappings must carry the built-in OpenID scope mappings ('openid', 'profile',
|
||||||
|
# 'email') alongside the claim mapping. A provider holding only the claim mapping grants a
|
||||||
|
# token scoped shorter than the scopes ABS requests, and ABS's /userinfo call then dies
|
||||||
|
# 'insufficient_scope (The request requires higher privileges than provided by the access
|
||||||
|
# token)' after authentik has already issued a good code (measured 2026-11-02).
|
||||||
|
#
|
||||||
|
# Idempotent: every object is created or asserted. Safe to re-run.
|
||||||
|
#
|
||||||
|
# usage: ABS_OIDC_CLIENT_SECRET=<secret> authentik-audiobookshelf-sso.sh
|
||||||
|
# (the secret is the value stored in
|
||||||
|
# /home/dank/.config/secrets/audiobookshelf on Dankserver — get it with
|
||||||
|
# ssh dankserver awk -F= '/^ABS_OIDC_CLIENT_SECRET=/{print $2}' \
|
||||||
|
# ~/.config/secrets/audiobookshelf)
|
||||||
|
#
|
||||||
|
# environment:
|
||||||
|
# ABS_OIDC_CLIENT_SECRET REQUIRED. The provider's client secret.
|
||||||
|
# ABS_OIDC_CLIENT_ID provider/client id (audiobookshelf)
|
||||||
|
# ABS_OIDC_GROUPS_CLAIM claim ABS reads for roles (abs_groups)
|
||||||
|
# ABS_OIDC_USERNAME authentik user made admin (akadmin)
|
||||||
|
# ABS_OIDC_USERS space-separated authentik usernames given the USER group
|
||||||
|
# (default: nobody — an empty list is fail-closed, and the
|
||||||
|
# household decides who listens to audiobooks)
|
||||||
|
# ABS_E2E_ADMIN authentik user kept in the ADMIN group for the Playwright gate
|
||||||
|
# (qa-e2e-admin — tests/e2e/specs/23-audiobookshelf-sso.spec.ts
|
||||||
|
# signs in as this identity, so a re-run restores the fixture)
|
||||||
|
# ABS_BASE ABS URL as the browser sees it, base path included
|
||||||
|
# (http://192.168.0.235:8000/audiobookshelf — the LAN
|
||||||
|
# origin, reached directly, which is also what the phone
|
||||||
|
# app is configured with)
|
||||||
|
# ABS_EXTRA_BASES space-separated additional origins, base path included
|
||||||
|
# (default: https://audiobooks.franksdank.cloud/audiobookshelf
|
||||||
|
# — the guarded vhost, dankclaw/Caddyfile).
|
||||||
|
# ABS builds its redirect_uri from the ORIGIN of the
|
||||||
|
# request, so each origin the app is served at needs its
|
||||||
|
# own registered URI — an unregistered origin is a
|
||||||
|
# redirect_uri_mismatch at the IdP, before any credential.
|
||||||
|
# Set it empty to register the LAN origin alone.
|
||||||
|
# AUTHENTIK_HOST ssh target (dankclaw)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SECRET="${ABS_OIDC_CLIENT_SECRET:?set ABS_OIDC_CLIENT_SECRET (see the header)}"
|
||||||
|
[ "${#SECRET}" -ge 32 ] || { echo "refused: ABS_OIDC_CLIENT_SECRET is too short to be a secret" >&2; exit 2; }
|
||||||
|
|
||||||
|
CLIENT_ID="${ABS_OIDC_CLIENT_ID:-audiobookshelf}"
|
||||||
|
CLAIM="${ABS_OIDC_GROUPS_CLAIM:-abs_groups}"
|
||||||
|
ADMIN_GROUP="audiobookshelf_admins"
|
||||||
|
USER_GROUP="audiobookshelf_users"
|
||||||
|
SSO_USER="${ABS_OIDC_USERNAME:-akadmin}"
|
||||||
|
MEMBERS="${ABS_OIDC_USERS:-}"
|
||||||
|
E2E_ADMIN="${ABS_E2E_ADMIN:-qa-e2e-admin}"
|
||||||
|
BASE="${ABS_BASE:-http://192.168.0.235:8000/audiobookshelf}"
|
||||||
|
EXTRA_BASES="${ABS_EXTRA_BASES:-https://audiobooks.franksdank.cloud/audiobookshelf}"
|
||||||
|
HOST="${AUTHENTIK_HOST:-dankclaw}"
|
||||||
|
|
||||||
|
# The claim expression: translate group membership into ABS role names. It is stored
|
||||||
|
# verbatim so a later read of the provider shows exactly this policy.
|
||||||
|
read -r -d '' EXPR <<PYEXPR || true
|
||||||
|
groups = {g.name for g in user.groups.all()}
|
||||||
|
roles = []
|
||||||
|
if "$ADMIN_GROUP" in groups:
|
||||||
|
roles.append("admin")
|
||||||
|
if groups & {"$ADMIN_GROUP", "$USER_GROUP"}:
|
||||||
|
roles.append("user")
|
||||||
|
return {"$CLAIM": roles}
|
||||||
|
PYEXPR
|
||||||
|
|
||||||
|
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
|
||||||
|
from authentik.core.models import Application, Group, User
|
||||||
|
from authentik.flows.models import Flow
|
||||||
|
from authentik.providers.oauth2.models import (
|
||||||
|
ClientType, OAuth2Provider, RedirectURI, RedirectURIMatchingMode, RedirectURIType, ScopeMapping)
|
||||||
|
|
||||||
|
ADMIN_GROUP, USER_GROUP = "$ADMIN_GROUP", "$USER_GROUP"
|
||||||
|
CLAIM, CLIENT_ID, SECRET, BASE = "$CLAIM", "$CLIENT_ID", "$SECRET", "$BASE"
|
||||||
|
BASES = [BASE] + """$EXTRA_BASES""".split()
|
||||||
|
SSO_USER, E2E_ADMIN = "$SSO_USER", "$E2E_ADMIN"
|
||||||
|
MEMBERS = """$MEMBERS""".split()
|
||||||
|
EXPR = '''$EXPR'''
|
||||||
|
# The expression reads user.groups, NOT user.ak_groups: authentik 2026.8 logs
|
||||||
|
# "User.ak_groups is deprecated … use User.groups" on every evaluation of the old
|
||||||
|
# name (measured on this run), and a stored mapping that leans on a deprecated
|
||||||
|
# accessor is a future upgrade failure sitting inside the login path.
|
||||||
|
|
||||||
|
for name in (ADMIN_GROUP, USER_GROUP):
|
||||||
|
Group.objects.get_or_create(name=name)
|
||||||
|
admin_g, user_g = Group.objects.get(name=ADMIN_GROUP), Group.objects.get(name=USER_GROUP)
|
||||||
|
|
||||||
|
# --- the claim mapping (scope_name IS the claim name ABS requests) ---------------
|
||||||
|
mapping = ScopeMapping.objects.filter(name="audiobookshelf-groups").first()
|
||||||
|
if mapping is None:
|
||||||
|
mapping = ScopeMapping.objects.create(name="audiobookshelf-groups", scope_name=CLAIM,
|
||||||
|
expression=EXPR,
|
||||||
|
description="Audiobookshelf role names from group membership")
|
||||||
|
print("mapping: created (scope_name %s)" % CLAIM)
|
||||||
|
elif mapping.scope_name != CLAIM or mapping.expression != EXPR:
|
||||||
|
mapping.scope_name, mapping.expression = CLAIM, EXPR
|
||||||
|
mapping.save(update_fields=["scope_name", "expression"])
|
||||||
|
print("mapping: scope/expression updated (scope_name %s)" % CLAIM)
|
||||||
|
else:
|
||||||
|
print("mapping: already current")
|
||||||
|
|
||||||
|
# Prove the expression runs and yields an ABS role BEFORE anyone tries to log in:
|
||||||
|
# authentik execs the expression inside a function whose context has `user`.
|
||||||
|
_src = "def _probe(user, request=None):\n" + "".join(" %s\n" % ln for ln in EXPR.splitlines())
|
||||||
|
_ns = {}
|
||||||
|
exec(_src, _ns)
|
||||||
|
|
||||||
|
# --- provider --------------------------------------------------------------------
|
||||||
|
# One callback pair per origin (LAN address and the public vhost): ABS derives the
|
||||||
|
# browser callback from the request origin and the mobile callback from the door the
|
||||||
|
# app was configured with, so each origin needs both or that door cannot sign in.
|
||||||
|
uris = [RedirectURI(matching_mode=RedirectURIMatchingMode.STRICT, url="%s%s" % (b, p))
|
||||||
|
for b in BASES
|
||||||
|
for p in ("/auth/openid/callback", "/auth/openid/mobile-redirect")]
|
||||||
|
authz_flow = Flow.objects.get(slug="default-provider-authorization-implicit-consent")
|
||||||
|
|
||||||
|
# Three things beyond the name, each one a silent dead login: the grant-type list (2026.8
|
||||||
|
# defaults it empty), the built-in OpenID scope mappings beside the claim mapping, and the
|
||||||
|
# redirect URIs with the base path baked in.
|
||||||
|
GRANTS = ["authorization_code", "refresh_token"]
|
||||||
|
BUILTIN_SCOPES = ["authentik default OAuth Mapping: OpenID 'openid'",
|
||||||
|
"authentik default OAuth Mapping: OpenID 'profile'",
|
||||||
|
"authentik default OAuth Mapping: OpenID 'email'"]
|
||||||
|
scoped = {m.pk: m for m in ScopeMapping.objects.filter(name__in=BUILTIN_SCOPES)}
|
||||||
|
scoped[mapping.pk] = mapping
|
||||||
|
_have = {m.name for m in scoped.values()}
|
||||||
|
if len(_have) < len(BUILTIN_SCOPES):
|
||||||
|
print("scope mappings: NOT PROVIDED by this authentik install: %s"
|
||||||
|
% ", ".join(sorted(set(BUILTIN_SCOPES) - _have)))
|
||||||
|
|
||||||
|
p = OAuth2Provider.objects.filter(name="audiobookshelf").first()
|
||||||
|
if p is None:
|
||||||
|
p = OAuth2Provider(name="audiobookshelf", client_id=CLIENT_ID,
|
||||||
|
client_type=ClientType.CONFIDENTIAL, client_secret=SECRET,
|
||||||
|
authorization_flow=authz_flow, grant_types=list(GRANTS))
|
||||||
|
p.redirect_uris = uris
|
||||||
|
p.save()
|
||||||
|
p.property_mappings.set(list(scoped.values()))
|
||||||
|
print("provider: created (grant_types %s)" % GRANTS)
|
||||||
|
else:
|
||||||
|
changed = []
|
||||||
|
if p.client_id != CLIENT_ID:
|
||||||
|
p.client_id = CLIENT_ID
|
||||||
|
changed.append("client_id")
|
||||||
|
if p.client_secret != SECRET:
|
||||||
|
p.client_secret = SECRET
|
||||||
|
changed.append("client_secret")
|
||||||
|
if p.authorization_flow_id != authz_flow.pk:
|
||||||
|
p.authorization_flow = authz_flow
|
||||||
|
changed.append("authorization_flow")
|
||||||
|
if list(p.grant_types) != GRANTS:
|
||||||
|
p.grant_types = list(GRANTS)
|
||||||
|
changed.append("grant_types")
|
||||||
|
if [(u.url, str(u.matching_mode)) for u in p.redirect_uris] != \
|
||||||
|
[(u.url, str(u.matching_mode)) for u in uris]:
|
||||||
|
p.redirect_uris = uris
|
||||||
|
changed.append("redirect_uris")
|
||||||
|
if changed:
|
||||||
|
p.save()
|
||||||
|
if sorted(m.pk for m in p.property_mappings.all()) != sorted(scoped):
|
||||||
|
p.property_mappings.set(list(scoped.values()))
|
||||||
|
changed.append("property_mappings")
|
||||||
|
print("provider: asserted (%s)" % (", ".join(changed) or "nothing to change"))
|
||||||
|
|
||||||
|
# --- application (the launchpad entry; binds the provider) ------------------------
|
||||||
|
app = Application.objects.filter(slug="audiobookshelf").first()
|
||||||
|
if app is None:
|
||||||
|
app = Application.objects.create(name="audiobookshelf", slug="audiobookshelf",
|
||||||
|
provider=p, meta_launch_url=BASE)
|
||||||
|
print("application: created")
|
||||||
|
elif app.provider_id != p.pk or app.meta_launch_url != BASE:
|
||||||
|
app.provider, app.meta_launch_url = p, BASE
|
||||||
|
app.save(update_fields=["provider", "meta_launch_url"])
|
||||||
|
print("application: asserted")
|
||||||
|
else:
|
||||||
|
print("application: already bound")
|
||||||
|
|
||||||
|
# --- membership --------------------------------------------------------------------
|
||||||
|
root = User.objects.get(username=SSO_USER)
|
||||||
|
root.groups.add(admin_g)
|
||||||
|
# The deploy gate's Playwright spec signs in as this identity, so the admin fixture is
|
||||||
|
# provisioned with the rest: a blueprint re-import that drops it fails the gate, not a phone.
|
||||||
|
e2e = User.objects.filter(username=E2E_ADMIN).first()
|
||||||
|
if e2e is None:
|
||||||
|
print("e2e fixture: authentik user %s absent — the e2e SSO spec will fail" % E2E_ADMIN)
|
||||||
|
else:
|
||||||
|
e2e.groups.add(admin_g)
|
||||||
|
print("e2e fixture: %s in %s" % (E2E_ADMIN, ADMIN_GROUP))
|
||||||
|
print("admin: %s in %s" % (SSO_USER, ADMIN_GROUP))
|
||||||
|
for name in MEMBERS:
|
||||||
|
u = User.objects.get(username=name)
|
||||||
|
u.groups.add(user_g)
|
||||||
|
print("users: %s in %s" % (", ".join(MEMBERS) or "(none — fail closed)", USER_GROUP))
|
||||||
|
|
||||||
|
# --- verification: the claim the callback will actually receive ---------------------
|
||||||
|
for probe in [root] + ([e2e] if e2e else []) + [User.objects.get(username=n) for n in MEMBERS]:
|
||||||
|
print("claim for %s -> %s" % (probe.username, _ns["_probe"](user=probe, request=None)))
|
||||||
|
p.refresh_from_db()
|
||||||
|
print("provider %s | client_id %s | flow %s | grants %s | uris %s" % (
|
||||||
|
p.name, p.client_id, p.authorization_flow.slug, list(p.grant_types),
|
||||||
|
[u.url for u in p.redirect_uris]))
|
||||||
|
print("scopes: %s" % sorted(m.scopemapping.scope_name for m in p.property_mappings.all()))
|
||||||
|
PYEOF
|
||||||
|
|
||||||
|
cat <<DONE
|
||||||
|
|
||||||
|
Next: arm the Audiobookshelf side (issuer, client, HS256, claim) —
|
||||||
|
ssh dankserver 'ABS_OIDC=1 bash /home/dank/serverAppConfigs/scripts/audiobookshelf-init.sh'
|
||||||
|
It reads the same secret from /home/dank/.config/secrets/audiobookshelf and verifies
|
||||||
|
/status answers openid among its auth methods.
|
||||||
|
DONE
|
||||||
Executable
+95
@@ -0,0 +1,95 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Census and repair the OAuth2 defaults on every authentik PROXY provider.
|
||||||
|
#
|
||||||
|
# A proxy provider built by hand in `ak shell` never runs
|
||||||
|
# ProxyProvider.set_oauth_defaults() — the REST serializer calls it, the ORM does
|
||||||
|
# not — so the row looks perfectly configured (external_host, internal_host,
|
||||||
|
# application bound, embedded-outpost claim present) while three OAuth facts are
|
||||||
|
# blank. Each one kills the guarded login in a different, unrelated-looking way,
|
||||||
|
# all BEFORE any credential is asked for:
|
||||||
|
#
|
||||||
|
# * redirect_uris empty -> authentik answers its own page:
|
||||||
|
# "Redirect URI Error — The request fails due to a missing, invalid, or
|
||||||
|
# mismatching redirection URI (redirect_uri)". Measured on the Audiobookshelf
|
||||||
|
# guard provider, 2026-11-03.
|
||||||
|
# * grant_types empty -> the authorize endpoint refuses the outpost's
|
||||||
|
# authorization-code request with a bodiless HTTP 400 (Chromium renders
|
||||||
|
# "This page isn't working / HTTP ERROR 400"). Same provider, one step
|
||||||
|
# later, after the URIs were filled in by hand.
|
||||||
|
# * authorization_flow NULL -> the authorize view crashes building its flow
|
||||||
|
# planner: AttributeError 'NoneType' object has no attribute 'slug', HTTP
|
||||||
|
# 500. That is what authentik-grafana-sso-fix.sh exists for.
|
||||||
|
#
|
||||||
|
# So: a guard that shows ANY of those three pages is not an Anubis problem and
|
||||||
|
# not a Caddy problem — run this script before touching the route.
|
||||||
|
#
|
||||||
|
# Repair calls the VENDOR's own set_oauth_defaults() (grant_types = authz code +
|
||||||
|
# client credentials + password, confidential client, the stock scope mappings,
|
||||||
|
# redirect_uris derived from external_host by _get_callback_url) and points the
|
||||||
|
# authorization flow at default-provider-authorization-implicit-consent, the flow
|
||||||
|
# every working provider here uses. Providers that already match are untouched,
|
||||||
|
# so a deliberate per-provider deviation is never silently flattened.
|
||||||
|
#
|
||||||
|
# This is authentik DB state. It is NOT in any deployable file: a blueprint
|
||||||
|
# re-import, or an ak-shell re-creation of a provider, reverts these facts and
|
||||||
|
# the symptom returns verbatim. Re-run after any such rebuild. Idempotent.
|
||||||
|
#
|
||||||
|
# Usage: authentik-proxy-provider-defaults.sh [--check]
|
||||||
|
# no argument repair what drifts, print one line per provider
|
||||||
|
# --check census only, exit 1 if anything drifts (gate use)
|
||||||
|
# override host: AUTHENTIK_HOST=…
|
||||||
|
set -euo pipefail
|
||||||
|
HOST="${AUTHENTIK_HOST:-dankclaw}"
|
||||||
|
MODE="${1:-}"
|
||||||
|
case "$MODE" in
|
||||||
|
"" | --check) : ;;
|
||||||
|
*) echo "usage: $(basename "$0") [--check]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
|
||||||
|
from authentik.flows.models import Flow
|
||||||
|
from authentik.providers.proxy.models import ProxyProvider, _get_callback_url
|
||||||
|
|
||||||
|
CHECK = $( [ "$MODE" = "--check" ] && echo True || echo False )
|
||||||
|
FLOW = "default-provider-authorization-implicit-consent"
|
||||||
|
WANT_GRANTS = ["authorization_code", "client_credentials", "password"]
|
||||||
|
|
||||||
|
flow = Flow.objects.filter(slug=FLOW).first()
|
||||||
|
if not flow and not CHECK:
|
||||||
|
raise SystemExit("flow %s is missing; nothing was changed" % FLOW)
|
||||||
|
|
||||||
|
drift = 0
|
||||||
|
for p in ProxyProvider.objects.all().order_by("pk"):
|
||||||
|
want_uris = [u.url for u in _get_callback_url(p.external_host)]
|
||||||
|
have_uris = [u.url for u in p.redirect_uris]
|
||||||
|
bad = []
|
||||||
|
if [str(g) for g in p.grant_types] != WANT_GRANTS:
|
||||||
|
bad.append("grant_types=%s" % [str(g) for g in p.grant_types])
|
||||||
|
if have_uris != want_uris:
|
||||||
|
bad.append("redirect_uris=%s" % have_uris)
|
||||||
|
if p.authorization_flow_id is None:
|
||||||
|
bad.append("authorization_flow=None")
|
||||||
|
if not bad:
|
||||||
|
print("ok %s (ext %s)" % (p.name, p.external_host))
|
||||||
|
continue
|
||||||
|
drift += 1
|
||||||
|
if CHECK:
|
||||||
|
print("DRIFT %s: %s" % (p.name, "; ".join(bad)))
|
||||||
|
continue
|
||||||
|
print("fix %s: %s" % (p.name, "; ".join(bad)))
|
||||||
|
p.set_oauth_defaults()
|
||||||
|
if p.authorization_flow_id is None:
|
||||||
|
p.authorization_flow = flow
|
||||||
|
p.save()
|
||||||
|
after = ProxyProvider.objects.get(pk=p.pk)
|
||||||
|
print(" -> grants %s, %d redirect URIs, flow %s"
|
||||||
|
% ([str(g) for g in after.grant_types], len(after.redirect_uris),
|
||||||
|
after.authorization_flow and after.authorization_flow.slug))
|
||||||
|
|
||||||
|
print("providers checked, drift: %d" % drift)
|
||||||
|
raise SystemExit(1 if (CHECK and drift) else 0)
|
||||||
|
PYEOF
|
||||||
|
|
||||||
|
echo "Verify: walk the guarded vhost in a browser — Anubis solves, authentik"
|
||||||
|
echo "renders its login form (no Redirect URI Error, no HTTP 400), and the"
|
||||||
|
echo "callback lands back on the app."
|
||||||
@@ -60,6 +60,7 @@ SERVICE_ORDER = (
|
|||||||
"calibre-server",
|
"calibre-server",
|
||||||
"plex",
|
"plex",
|
||||||
"tautulli",
|
"tautulli",
|
||||||
|
"audiobookshelf",
|
||||||
"prowlarr",
|
"prowlarr",
|
||||||
"qbittorrent",
|
"qbittorrent",
|
||||||
"radarr",
|
"radarr",
|
||||||
@@ -123,6 +124,7 @@ DANKSERVER_SERVICES = {
|
|||||||
"calibre-server",
|
"calibre-server",
|
||||||
"plex",
|
"plex",
|
||||||
"tautulli",
|
"tautulli",
|
||||||
|
"audiobookshelf",
|
||||||
"prowlarr",
|
"prowlarr",
|
||||||
"qbittorrent",
|
"qbittorrent",
|
||||||
"radarr",
|
"radarr",
|
||||||
@@ -420,6 +422,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
"copyparty": {"copyparty"},
|
"copyparty": {"copyparty"},
|
||||||
"plex": {"plex"},
|
"plex": {"plex"},
|
||||||
"tautulli": {"tautulli"},
|
"tautulli": {"tautulli"},
|
||||||
|
"audiobookshelf": {"audiobookshelf"},
|
||||||
}
|
}
|
||||||
_add(services, *quadlet_services.get(parts[1], set()))
|
_add(services, *quadlet_services.get(parts[1], set()))
|
||||||
if len(parts) >= 2 and parts[1] == "host":
|
if len(parts) >= 2 and parts[1] == "host":
|
||||||
|
|||||||
@@ -95,6 +95,7 @@ service_is_allowed() {
|
|||||||
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
|
dankserver:audiobookshelf|\
|
||||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||||
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||||
@@ -1212,6 +1213,10 @@ ensure_media_view() {
|
|||||||
# scripts/wait-for-mount.sh, which waits and then refuses the start. A
|
# scripts/wait-for-mount.sh, which waits and then refuses the start. A
|
||||||
# relayout must be followed by `media-view.sh reap`
|
# relayout must be followed by `media-view.sh reap`
|
||||||
# (see the Media Relayout page), which this is not.
|
# (see the Media Relayout page), which this is not.
|
||||||
|
# A fresh host has no upper base, and `install` refuses to invent one — so
|
||||||
|
# without this the media deploy dies on a rebuilt machine with
|
||||||
|
# "put it on the SSD first" (measured 2026-10-31).
|
||||||
|
run_privileged bash scripts/media-view.sh ensure-base
|
||||||
run_privileged bash scripts/media-view.sh install
|
run_privileged bash scripts/media-view.sh install
|
||||||
bash scripts/media-view.sh verify
|
bash scripts/media-view.sh verify
|
||||||
}
|
}
|
||||||
@@ -1296,6 +1301,37 @@ deploy_calibre_web() {
|
|||||||
systemctl --user restart calibre-web.service
|
systemctl --user restart calibre-web.service
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Audiobookshelf keeps its whole state (SQLite library, accounts, progress,
|
||||||
|
# cached covers) on the NVMe under /home/dank/Containers/audiobookshelf; the
|
||||||
|
# audio is a read-only bind off the array, so the only thing the deployer has to
|
||||||
|
# create is the two state roots. mkdir names the owner — letting podman invent
|
||||||
|
# the directories leaves the SQLite file unwritable by the container user on a
|
||||||
|
# rootless first boot. No `systemctl --user enable`: a generated Quadlet unit
|
||||||
|
# cannot be enabled, and WantedBy=default.target is what carries it across a
|
||||||
|
# reboot.
|
||||||
|
deploy_audiobookshelf() {
|
||||||
|
cp "dankserver/audiobookshelf/quadlet/audiobookshelf.container" "$quadlet_unit_dir/"
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
mkdir -p /home/dank/Containers/audiobookshelf/config \
|
||||||
|
/home/dank/Containers/audiobookshelf/metadata
|
||||||
|
systemctl --user restart audiobookshelf.service
|
||||||
|
# /healthcheck answers once the HTTP server is bound, at the root whatever
|
||||||
|
# ROUTER_BASE_PATH says (measured on both base paths); the library scan that
|
||||||
|
# follows runs in the background and must not gate the deploy.
|
||||||
|
wait_for_health audiobookshelf http://127.0.0.1:8000/healthcheck 90
|
||||||
|
# A server with no root user and no library is "healthy" and useless, so the
|
||||||
|
# bootstrapper runs after the health gate and its failure fails the deploy:
|
||||||
|
# it creates the account (credential in the mode-600 host secret file), the
|
||||||
|
# Audiobooks library over the /audiobooks bind, and fires the scan. Every
|
||||||
|
# step is idempotent, so a re-deploy costs one incremental scan.
|
||||||
|
# The bootstrapper has to speak the same base path the unit mounts (the API, the
|
||||||
|
# login route and the OIDC redirect subfolder all hang off it), so the URL is read
|
||||||
|
# out of the unit file here instead of being restated in the script's default.
|
||||||
|
abs_base=$(awk -F= '/^Environment=ROUTER_BASE_PATH=/{print $3}' \
|
||||||
|
dankserver/audiobookshelf/quadlet/audiobookshelf.container)
|
||||||
|
ABS_URL="http://127.0.0.1:8000${abs_base}" scripts/audiobookshelf-init.sh
|
||||||
|
}
|
||||||
|
|
||||||
# The content server's single admin account and the desktop's basic-auth
|
# The content server's single admin account and the desktop's basic-auth
|
||||||
# credential (same pair) are bootstrapped exactly once: the env file is the
|
# credential (same pair) are bootstrapped exactly once: the env file is the
|
||||||
# plaintext store (mode 600), the userdb's pw column is plaintext too, so
|
# plaintext store (mode 600), the userdb's pw column is plaintext too, so
|
||||||
@@ -2172,6 +2208,7 @@ deploy_service() {
|
|||||||
copyparty) deploy_copyparty ;;
|
copyparty) deploy_copyparty ;;
|
||||||
plex) deploy_plex ;;
|
plex) deploy_plex ;;
|
||||||
tautulli) deploy_tautulli ;;
|
tautulli) deploy_tautulli ;;
|
||||||
|
audiobookshelf) deploy_audiobookshelf ;;
|
||||||
alloy) deploy_alloy ;;
|
alloy) deploy_alloy ;;
|
||||||
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
||||||
gpu-exporter) deploy_gpu_exporter ;;
|
gpu-exporter) deploy_gpu_exporter ;;
|
||||||
|
|||||||
@@ -148,6 +148,7 @@ valid_service() {
|
|||||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
|
dankserver:audiobookshelf|\
|
||||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||||
dankserver:prowlarr|dankserver:frank-archon|\
|
dankserver:prowlarr|dankserver:frank-archon|\
|
||||||
|
|||||||
@@ -35,7 +35,8 @@
|
|||||||
# Subcommands
|
# Subcommands
|
||||||
# plan print the resolved shape and every pending change, touch nothing
|
# plan print the resolved shape and every pending change, touch nothing
|
||||||
# install fetch+verify the plugin, then write the keys (backing up first)
|
# install fetch+verify the plugin, then write the keys (backing up first)
|
||||||
# verify prove the plugin and both keys are actually in place
|
# verify prove the plugin and both keys are in place, and that no library
|
||||||
|
# is a silent file index (providers off / image fetchers empty)
|
||||||
#
|
#
|
||||||
# Nothing here restarts Jellyfin: the deployer owns that
|
# Nothing here restarts Jellyfin: the deployer owns that
|
||||||
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
||||||
@@ -134,7 +135,18 @@ text = escape(val)
|
|||||||
if pat.search(src):
|
if pat.search(src):
|
||||||
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
|
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
|
||||||
else:
|
else:
|
||||||
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
|
# The plugin owns this file and writes it under ITS OWN root element
|
||||||
|
# (TheTVDB's is <PluginConfiguration>), and its serializer omits a
|
||||||
|
# null-valued setting entirely — so the element being filled can be
|
||||||
|
# absent outright, and a hardcoded </Configuration> anchor then matches
|
||||||
|
# nothing. Anchor on the document's real root instead.
|
||||||
|
out = src
|
||||||
|
root = re.search(r"<([A-Za-z_][\w.:-]*)[^>]*>", src)
|
||||||
|
if root and ("</%s>" % root.group(1)) in src:
|
||||||
|
name = root.group(1)
|
||||||
|
out = src.replace(
|
||||||
|
"</%s>" % name, f" <{elem}>{text}</{elem}>\n</{name}>", 1
|
||||||
|
)
|
||||||
if out == src:
|
if out == src:
|
||||||
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
|
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
|
||||||
|
|
||||||
@@ -340,6 +352,64 @@ cmd_install() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# A library whose options.xml carries EnableInternetProviders=false, or an empty
|
||||||
|
# ImageFetchers list beside a populated MetadataFetchers list, is a SILENT FILE
|
||||||
|
# INDEX: it indexes every file, answers 204 to every refresh, stores no provider
|
||||||
|
# id, plot, rating or artwork, and logs nothing — so the item count reads like a
|
||||||
|
# healthy scan. Jellyfin 12 keeps these options in root/default/<Library>/options.xml
|
||||||
|
# (it was library.xml) and stores NONE of them in the database, so this audit is
|
||||||
|
# the only place the state can be read without a browser session.
|
||||||
|
verify_library_options() {
|
||||||
|
local root="$JF_ROOT/root/default" f lib problems
|
||||||
|
if [ ! -d "$root" ]; then
|
||||||
|
printf '= no library options under %s (nothing audited)\n' "$root"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local rc=0
|
||||||
|
for f in "$root"/*/options.xml; do
|
||||||
|
[ -f "$f" ] || continue
|
||||||
|
lib="$(basename "$(dirname "$f")")"
|
||||||
|
problems="$(python3 -s - "$f" <<'PY'
|
||||||
|
import sys, xml.dom.minidom as X
|
||||||
|
root = X.parse(sys.argv[1]).documentElement
|
||||||
|
def child(el, name):
|
||||||
|
for n in el.childNodes:
|
||||||
|
if n.nodeType == n.ELEMENT_NODE and n.tagName == name:
|
||||||
|
return n
|
||||||
|
def flag(el, name):
|
||||||
|
n = child(el, name)
|
||||||
|
return None if n is None else (n.firstChild.data.strip() if n.firstChild else "")
|
||||||
|
def names(el, name):
|
||||||
|
n = child(el, name)
|
||||||
|
return None if n is None else [s.firstChild.data.strip() for s in n.childNodes
|
||||||
|
if s.nodeType == s.ELEMENT_NODE and s.firstChild]
|
||||||
|
bad = []
|
||||||
|
ip = flag(root, "EnableInternetProviders")
|
||||||
|
if ip is not None and ip.lower() != "true":
|
||||||
|
bad.append("EnableInternetProviders=%s" % (ip or "empty"))
|
||||||
|
types = child(root, "TypeOptions")
|
||||||
|
if types is not None:
|
||||||
|
for t in types.childNodes:
|
||||||
|
if t.nodeType != t.ELEMENT_NODE or t.tagName != "TypeOptions":
|
||||||
|
continue
|
||||||
|
meta = names(t, "MetadataFetchers") or []
|
||||||
|
if meta and not (names(t, "ImageFetchers") or []):
|
||||||
|
bad.append("%s: ImageFetchers empty while MetadataFetchers names %d"
|
||||||
|
% (flag(t, "Type") or "?", len(meta)))
|
||||||
|
print("; ".join(bad))
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
if [ -n "$problems" ]; then
|
||||||
|
printf 'SILENT INDEX %s: %s\n' "$lib" "$problems" >&2
|
||||||
|
rc=1
|
||||||
|
else
|
||||||
|
printf '= %s: library options fetch from providers\n' "$lib"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
cmd_verify() {
|
cmd_verify() {
|
||||||
check_root
|
check_root
|
||||||
local bad=0
|
local bad=0
|
||||||
@@ -378,6 +448,7 @@ cmd_verify() {
|
|||||||
printf '%s %s chars\n' "$elem" "${#n}"
|
printf '%s %s chars\n' "$elem" "${#n}"
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
verify_library_options || bad=1
|
||||||
[ "$bad" -eq 0 ] || return 1
|
[ "$bad" -eq 0 ] || return 1
|
||||||
printf 'providers configured\n'
|
printf 'providers configured\n'
|
||||||
}
|
}
|
||||||
|
|||||||
+10
-2
@@ -121,14 +121,14 @@ class Journal:
|
|||||||
|
|
||||||
def read_plan(path, root, force):
|
def read_plan(path, root, force):
|
||||||
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
|
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
|
||||||
mkdirs, keeps, moves = [], [], []
|
mkdirs, keeps, moves, bad = [], [], [], []
|
||||||
with open(path) as fh:
|
with open(path) as fh:
|
||||||
header = fh.readline()
|
header = fh.readline()
|
||||||
if not header.startswith("action\t"):
|
if not header.startswith("action\t"):
|
||||||
raise Refuse(f"{path} is not a plan (first column is "
|
raise Refuse(f"{path} is not a plan (first column is "
|
||||||
f"{header.split(chr(9))[0]!r}, expected 'action') —"
|
f"{header.split(chr(9))[0]!r}, expected 'action') —"
|
||||||
" regenerate it with scripts/media-plan.py")
|
" regenerate it with scripts/media-plan.py")
|
||||||
for line in fh:
|
for n, line in enumerate(fh, 2):
|
||||||
f = line.rstrip("\n").split("\t")
|
f = line.rstrip("\n").split("\t")
|
||||||
if f[0] == "mkdir":
|
if f[0] == "mkdir":
|
||||||
mkdirs.append(f[2])
|
mkdirs.append(f[2])
|
||||||
@@ -136,6 +136,14 @@ def read_plan(path, root, force):
|
|||||||
keeps.append(f[1])
|
keeps.append(f[1])
|
||||||
elif f[0] == "move":
|
elif f[0] == "move":
|
||||||
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
|
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
|
||||||
|
else:
|
||||||
|
bad.append(f"line {n}: unknown action {f[0]!r} — this tool"
|
||||||
|
" knows mkdir, keep and move (a rename is a move)")
|
||||||
|
if bad:
|
||||||
|
# A row this tool could not read is a plan bug, and a plan bug refuses:
|
||||||
|
# silently dropping it turns a typo into a no-op run that reports success.
|
||||||
|
head = "; ".join(bad[:3])
|
||||||
|
raise Refuse(head + (f" (+{len(bad) - 3} more)" if len(bad) > 3 else ""))
|
||||||
if not force:
|
if not force:
|
||||||
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
|
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
|
||||||
if p and not (p == root or p.startswith(root + os.sep)):
|
if p and not (p == root or p.startswith(root + os.sep)):
|
||||||
|
|||||||
+10
-4
@@ -78,10 +78,16 @@ one journal is one undo:
|
|||||||
|
|
||||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
|
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
|
||||||
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
|
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
|
||||||
--tvdb --tmdb --out /mnt/entropy/media-tv-plan.tsv \
|
--tvdb --tmdb --out /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||||
--park /mnt/entropy/media-park/TVShows
|
--park /mnt/entropy/NeedsProcessing/TVShows
|
||||||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
python3 scripts/media-apply.py apply \
|
||||||
--root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
|
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-tv-relayout.jsonl
|
||||||
|
|
||||||
|
Every plan, journal and log lives in /mnt/entropy/.media-campaign/ — the array
|
||||||
|
root holds libraries only. media-apply.py defaults its journal to
|
||||||
|
<root>/media-relayout.jsonl, so pass --journal explicitly or the run puts the
|
||||||
|
log back on the root.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import collections
|
import collections
|
||||||
|
|||||||
+13
-6
@@ -17,13 +17,20 @@ the executor's own path guard still has teeth, and give it its own journal —
|
|||||||
one journal is one undo, and a TV move must not be welded to a movie relayout:
|
one journal is one undo, and a TV move must not be welded to a movie relayout:
|
||||||
|
|
||||||
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
|
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
|
||||||
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
|
--library /mnt/entropy/TVShows \
|
||||||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
--out /mnt/entropy/.media-campaign/media-tv-plan.tsv
|
||||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
python3 scripts/media-apply.py apply \
|
||||||
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
|
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl
|
||||||
|
python3 scripts/media-apply.py verify \
|
||||||
|
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl
|
||||||
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
--journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl --apply
|
||||||
|
|
||||||
|
Plans and journals live in `/mnt/entropy/.media-campaign/`, never loose on the
|
||||||
|
array root: `media-apply.py` defaults its journal to `<root>/media-relayout.jsonl`,
|
||||||
|
so `--journal` is always explicit here.
|
||||||
|
|
||||||
Guards, and why each one is here:
|
Guards, and why each one is here:
|
||||||
|
|
||||||
|
|||||||
Executable
+475
@@ -0,0 +1,475 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""media-tvdupes.py — find duplicate TV episodes and plan their staging, one
|
||||||
|
EPISODE at a time.
|
||||||
|
|
||||||
|
A TV library grows the same show into several folders: a tagged season box, an
|
||||||
|
untagged packager drop, a `Complete` folder that is not complete. The folders
|
||||||
|
look like copies of each other and are not, so the unit of comparison is the
|
||||||
|
episode and never the folder. A file is droppable only when EVERY episode it
|
||||||
|
carries is also held, somewhere else in the same show, by a file at least as
|
||||||
|
large — Frank's rule: the same episode in two places keeps the larger one. A
|
||||||
|
file that holds one episode nothing else has stays, whatever else it carries.
|
||||||
|
|
||||||
|
It writes a plan and moves nothing; `scripts/media-apply.py` is the only thing
|
||||||
|
that renames. The plan stages into a dated hold, so the pass is undoable from
|
||||||
|
the journal and frees no space until the hold is purged.
|
||||||
|
|
||||||
|
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||||
|
--hold /mnt/entropy/.AGENTTRASH/20261102-tv-duplicates \
|
||||||
|
--out /mnt/entropy/.media-campaign/tv-dupes-plan.tsv
|
||||||
|
python3 scripts/media-apply.py apply \
|
||||||
|
--plan /mnt/entropy/.media-campaign/tv-dupes-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/tv-dupes.jsonl \
|
||||||
|
--metadata-view /var/lib/media-meta/view --apply
|
||||||
|
python3 scripts/media-apply.py verify \
|
||||||
|
--plan /mnt/entropy/.media-campaign/tv-dupes-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/tv-dupes.jsonl
|
||||||
|
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/.media-campaign/tv-dupes.jsonl --apply # empty folders only
|
||||||
|
scripts/media-view.sh reap # ghost sidecars above
|
||||||
|
|
||||||
|
Then both servers have to be told: the plan moves files on the raw array, and no
|
||||||
|
watcher crosses the metadata view — [[Media Relayout]] owns the refresh order.
|
||||||
|
|
||||||
|
Guards, and why each one is here:
|
||||||
|
|
||||||
|
* **Identity before geometry.** Two folders holding the same words are not the
|
||||||
|
same show: the 2026-11-02 census found `Avatar: The Last Airbender` (1999,
|
||||||
|
tvdb 74852) beside the 2024 remake (385925), `Star Trek: Voyager` beside
|
||||||
|
`Star Trek: The Animated Series`, and a German thriller Jellyfin had titled
|
||||||
|
`Millennium (2022) [tvdbid-472933]` beside `Millennium (1996)`. A folder's
|
||||||
|
identity is read from three places, in this order: its `[tvdbid-N]` tag, the
|
||||||
|
`tvshow.nfo` Jellyfin wrote — which lives in the metadata view, not on the
|
||||||
|
array, and is what `--meta` reads — and a `folder<TAB>id` map passed with
|
||||||
|
`--ids`. The tag outranks the derived layers: an id that contradicts a folder's
|
||||||
|
own tag is reported and ignored, never unioned — measured 2026-11-02, Jellyfin's
|
||||||
|
provider table carried a 2026 special's id on a folder tagged `[tvdbid-73838]`,
|
||||||
|
and unioning them makes one show of the 2000 series and the special, whose bigger
|
||||||
|
files then dominate the original's pilot. Two folders are one show when they share
|
||||||
|
an id, or when neither has one and name and year both agree. Two disjoint id sets
|
||||||
|
is an absolute stop: the pair is reported as a conflict and nothing under either is
|
||||||
|
staged. Same name with nothing to decide on is reported unresolved — the fix is the
|
||||||
|
tag, not a guess.
|
||||||
|
* **A folder-level verdict destroys content.** 10 of the 25 shows that looked
|
||||||
|
duplicated held COMPLEMENTARY halves: `Game of Thrones Complete` carries no
|
||||||
|
S01 and no S08, `Modern Family S05` is the only copy of season 5, `Loiter
|
||||||
|
Squad Uncensored` is the only copy of season 1. Judged per folder, 20 shows
|
||||||
|
and some 340 GB of unique episodes would have been staged; judged per episode
|
||||||
|
the true figure was a sixth of that.
|
||||||
|
* **A bare number is never an episode.** `256491.mp4` may be an episode id, a
|
||||||
|
poster, or a season, so a name that states no `SxxEyy` / `1x04` marker is
|
||||||
|
unkeyed: never staged, and counted in the report. A media server's own episode
|
||||||
|
table does know those files — pass it with `--witness` and its rows key
|
||||||
|
whatever the server has matched.
|
||||||
|
* **The audit re-reads the disk.** Sizes and the survivor's existence are
|
||||||
|
checked again when the plan is written, because the census is a snapshot: a
|
||||||
|
file that grew, vanished, or landed in the hold in between must fail the plan,
|
||||||
|
not surprise the executor.
|
||||||
|
* **A sidecar that shares an episode's stem follows it.** Subtitles and `.nfo`
|
||||||
|
left at the old coordinates are ghost bait for `media-view.sh reap`.
|
||||||
|
* A multi-episode file (`S01E01-02`) is dominated only when every episode in its
|
||||||
|
range is held elsewhere — and it competes for the survivor seat on each of
|
||||||
|
them, because the byte law compares files. A pack that states its range in
|
||||||
|
the name is a name both media servers parse the same way, so the pack winning
|
||||||
|
against two single-episode rips is the law applied, not a misread.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import collections
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||||
|
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||||
|
eng = importlib.util.module_from_spec(_spec)
|
||||||
|
_spec.loader.exec_module(eng)
|
||||||
|
|
||||||
|
# The executor's definition of a media file, plus the two the title engine omits:
|
||||||
|
# an audit that disagrees with media-apply.py is a plan that dies at its
|
||||||
|
# preflight instead of at the census.
|
||||||
|
VIDEO = tuple(sorted(set(eng.VIDEO_SUFFIX) | {".ts", ".wmv"}))
|
||||||
|
TAG_RE = re.compile(r"\[\s*tvdbid[\s._-]*(\d+)\s*\]", re.I)
|
||||||
|
YEAR_RE = re.compile(r"[(\[]\s*((?:19|20)\d\d)\s*[)\]]")
|
||||||
|
SEASON_EP = re.compile(r"(?i)\bs(?:eason)?[\s._-]*(\d{1,2})[\s._-]*e(?:p(?:isode)?)?"
|
||||||
|
r"[\s._-]*(\d{1,3})(?:[\s._-]*(?:-|to|\+)[\s._-]*e?"
|
||||||
|
r"[\s._-]*(\d{1,3}))?")
|
||||||
|
NXN = re.compile(r"(?i)(?<!\d)(\d{1,2})x(\d{1,3})(?!\d)")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--library", default="/mnt/entropy/TVShows",
|
||||||
|
help="the TV library to census (never written)")
|
||||||
|
ap.add_argument("--hold", required=True,
|
||||||
|
help="dated trash the plan stages into, e.g. "
|
||||||
|
"/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates")
|
||||||
|
ap.add_argument("--out", default=None,
|
||||||
|
help="plan TSV for scripts/media-apply.py (default: "
|
||||||
|
"tv-dupes-plan.tsv beside --library)")
|
||||||
|
ap.add_argument("--witness", default=None,
|
||||||
|
help="TSV of path<TAB>season<TAB>episode out of a media "
|
||||||
|
"server's episode table; keys files whose names carry"
|
||||||
|
" no marker")
|
||||||
|
ap.add_argument("--meta", default=None,
|
||||||
|
help="the merged metadata view (on the host "
|
||||||
|
"/var/lib/media-meta/view): tvshow.nfo lives in the"
|
||||||
|
" upper layer, so this is where an untagged folder's"
|
||||||
|
" tvdb id is readable")
|
||||||
|
ap.add_argument("--ids", default=None,
|
||||||
|
help="TSV of folder<TAB>tvdbid, one line per folder, for"
|
||||||
|
" identity a lookup settled and the folder name does"
|
||||||
|
" not state")
|
||||||
|
ap.add_argument("--json", default=None,
|
||||||
|
help="also dump the analysis as JSON")
|
||||||
|
return ap.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
ARGS = parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def episode_keys(name):
|
||||||
|
"""Every (season, episode) an episode name states.
|
||||||
|
|
||||||
|
`S01E03`, `Season 1 E3`, `1x04`, and a range `S01E01-02`, which expands so a
|
||||||
|
two-in-one file is only droppable when both halves are held elsewhere. A
|
||||||
|
bare number states nothing and yields nothing."""
|
||||||
|
keys = set()
|
||||||
|
for m in SEASON_EP.finditer(name):
|
||||||
|
s, first = int(m.group(1)), int(m.group(2))
|
||||||
|
keys.add((s, first))
|
||||||
|
if m.group(3):
|
||||||
|
keys.update((s, e) for e in range(first + 1, int(m.group(3)) + 1))
|
||||||
|
keys.update((int(m.group(1)), int(m.group(2))) for m in NXN.finditer(name))
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def folder_identity(folder):
|
||||||
|
"""(fold_sep name, year, tvdb ids) for a top-level show folder.
|
||||||
|
|
||||||
|
Tag and year come in both orders — `Name (2008) [tvdbid-84859]` and `Name
|
||||||
|
[tvdbid-84859] (2008)` are both in the library — so neither is assumed to
|
||||||
|
sit at the end, and both are cut out of the name folders compare on."""
|
||||||
|
ids = frozenset(int(x) for x in TAG_RE.findall(folder))
|
||||||
|
year = None
|
||||||
|
for _ in range(3):
|
||||||
|
y = YEAR_RE.search(folder)
|
||||||
|
if not y:
|
||||||
|
break
|
||||||
|
year = year or y.group(1)
|
||||||
|
folder = TAG_RE.sub(" ", f"{folder[:y.start()]} {folder[y.end():]}")
|
||||||
|
return eng.fold_sep(TAG_RE.sub(" ", folder)), year, ids
|
||||||
|
|
||||||
|
|
||||||
|
def nfo_id(path):
|
||||||
|
"""The tvdb id a `tvshow.nfo` states, or None.
|
||||||
|
|
||||||
|
Three shapes are in the library: Jellyfin's `<uniqueid type="tvdb">`, Kodi's
|
||||||
|
`<tvdbid>`, and a packager's `<id>` + `<id_type>` pair; some carry the id
|
||||||
|
only in the title line as `[tvdbid-N]`."""
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
text = fh.read(16384)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
for rx in (r"<tvdbid\s*>\s*(\d+)\s*<",
|
||||||
|
r"<uniqueid[^>]*type\s*=\s*[\"']?tvdb[^>]*>\s*(\d+)\s*<",
|
||||||
|
r"<id\s*>\s*(\d+)\s*</id>\s*<id_type>\s*(?:tvdb|thedotdb)",
|
||||||
|
r"\[\s*tvdbid[\s._-]*(\d+)\s*\]"):
|
||||||
|
m = re.search(rx, text, re.I)
|
||||||
|
if m:
|
||||||
|
return int(m.group(1))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def collect_ids(library, meta, ids_file):
|
||||||
|
"""top-level folder -> {tvdb ids} from the nfo layer and an operator map.
|
||||||
|
|
||||||
|
`meta` is the merged metadata view: Jellyfin's sidecars live in the upper
|
||||||
|
layer, so a folder's `tvshow.nfo` is not on the raw array. Reading the id
|
||||||
|
from there is what makes an untagged packager folder identifiable at all —
|
||||||
|
without it the pair stays unresolved and nothing under it is staged."""
|
||||||
|
out = collections.defaultdict(set)
|
||||||
|
if meta:
|
||||||
|
try:
|
||||||
|
tops = sorted(os.listdir(library))
|
||||||
|
except OSError:
|
||||||
|
tops = []
|
||||||
|
for folder in tops:
|
||||||
|
i = nfo_id(os.path.join(meta, folder, "tvshow.nfo"))
|
||||||
|
if i:
|
||||||
|
out[folder].add(i)
|
||||||
|
if ids_file:
|
||||||
|
with open(ids_file, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for line in fh:
|
||||||
|
cols = line.rstrip("\n").split("\t")
|
||||||
|
if len(cols) < 2 or not cols[1].strip().isdigit():
|
||||||
|
continue
|
||||||
|
out[cols[0]].add(int(cols[1].strip()))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def witness_rows(path):
|
||||||
|
"""absolute path -> {(season, episode)} from a media server's episode table."""
|
||||||
|
out = collections.defaultdict(set)
|
||||||
|
if not path:
|
||||||
|
return out
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for line in fh:
|
||||||
|
cols = line.rstrip("\n").split("\t")
|
||||||
|
if len(cols) < 3 or not cols[0].startswith("/"):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out[os.path.abspath(cols[0])].add((int(cols[1]), int(cols[2])))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def sidecars(path):
|
||||||
|
"""The files beside PATH carrying its exact stem — subtitles, nfo, artwork."""
|
||||||
|
folder, stem = os.path.split(path)
|
||||||
|
stem = os.path.splitext(stem)[0]
|
||||||
|
out = []
|
||||||
|
try:
|
||||||
|
entries = sorted(os.listdir(folder))
|
||||||
|
except OSError:
|
||||||
|
return out
|
||||||
|
for e in entries:
|
||||||
|
p = os.path.join(folder, e)
|
||||||
|
if p == path or os.path.islink(p) or not os.path.isfile(p):
|
||||||
|
continue
|
||||||
|
if os.path.splitext(e)[0] == stem:
|
||||||
|
out.append(p)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def scan(library, hold, witness):
|
||||||
|
"""Every video file under LIBRARY: relative path, size, episode keys.
|
||||||
|
|
||||||
|
A file keys from the witness when it has a row (a server matched it), and
|
||||||
|
from its own name otherwise; with neither it is unkeyed."""
|
||||||
|
files = []
|
||||||
|
for dirpath, dirnames, filenames in os.walk(library):
|
||||||
|
dirnames[:] = sorted(d for d in dirnames
|
||||||
|
if not d.startswith(".") and d != "@eaDir"
|
||||||
|
and os.path.abspath(os.path.join(dirpath, d)) != hold)
|
||||||
|
for f in sorted(filenames):
|
||||||
|
p = os.path.join(dirpath, f)
|
||||||
|
if os.path.islink(p) or not f.lower().endswith(VIDEO):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
size = os.path.getsize(p)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
files.append({"path": os.path.abspath(p),
|
||||||
|
"rel": os.path.relpath(os.path.abspath(p), library),
|
||||||
|
"size": size,
|
||||||
|
"keys": set(witness.get(os.path.abspath(p))
|
||||||
|
or episode_keys(os.path.splitext(f)[0]))})
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
def show_groups(folders, extra_ids):
|
||||||
|
"""(show groups, conflicts, unresolved, disagreements) over top-level folders.
|
||||||
|
|
||||||
|
A folder's id is its `[tvdbid-N]` tag; the nfo layer and `--ids` fill a folder
|
||||||
|
only when it carries no tag. The rule, in order of trust: an id in common makes
|
||||||
|
one show; two disjoint id sets are two shows, reported as a conflict and never
|
||||||
|
merged; no id on either side merges only on name AND year (the 1999 miniseries
|
||||||
|
and the 2015 series are not one show); one side ided and the other bare is
|
||||||
|
unresolved, because the fix is the tag.
|
||||||
|
|
||||||
|
The derived layers never overrule a tag. Measured 2026-11-02: Jellyfin's
|
||||||
|
provider table carried a 2026 special's tvdb id on the folder tagged
|
||||||
|
`[tvdbid-73838]`, and unioning the two made one show of the 2000 series and the
|
||||||
|
special — whose bigger files then dominate the original's pilot. A contradiction
|
||||||
|
is reported instead, because the tag is the operator's word and the provider
|
||||||
|
table is a guess that can be wrong.
|
||||||
|
"""
|
||||||
|
names = {f: folder_identity(f) for f in folders}
|
||||||
|
tags = {f: set(names[f][2]) for f in folders}
|
||||||
|
derived = {f: set(extra_ids.get(f, ())) for f in folders}
|
||||||
|
disagree = {f: (sorted(tags[f]), sorted(derived[f] - tags[f]))
|
||||||
|
for f in folders if tags[f] and derived[f] - tags[f]}
|
||||||
|
ids = {f: (tags[f] or derived[f]) for f in folders}
|
||||||
|
parent = {f: f for f in folders}
|
||||||
|
|
||||||
|
def find(f):
|
||||||
|
while parent[f] != f:
|
||||||
|
parent[f] = parent[parent[f]]
|
||||||
|
f = parent[f]
|
||||||
|
return f
|
||||||
|
|
||||||
|
conflicts, unresolved = [], []
|
||||||
|
order = sorted(folders)
|
||||||
|
for i, a in enumerate(order):
|
||||||
|
for b in order[i + 1:]:
|
||||||
|
na, ya, _ia = names[a]
|
||||||
|
nb, yb, _ib = names[b]
|
||||||
|
if na != nb:
|
||||||
|
continue
|
||||||
|
ia, ib = ids[a], ids[b]
|
||||||
|
if ia & ib:
|
||||||
|
parent[find(b)] = find(a)
|
||||||
|
elif ia and ib:
|
||||||
|
conflicts.append((a, b, f"tvdb {sorted(ia)} vs {sorted(ib)}"))
|
||||||
|
elif not ia and not ib and ya and ya == yb:
|
||||||
|
parent[find(b)] = find(a)
|
||||||
|
elif not ia and not ib:
|
||||||
|
conflicts.append((a, b,
|
||||||
|
f"same name, years {ya or '?'} vs {yb or '?'}"))
|
||||||
|
else:
|
||||||
|
unresolved.append((a, b, "one folder carries a tvdb id,"
|
||||||
|
" the other none"))
|
||||||
|
groups = collections.defaultdict(list)
|
||||||
|
for f in order:
|
||||||
|
groups[find(f)].append(f)
|
||||||
|
return sorted(groups.values()), conflicts, unresolved, disagree
|
||||||
|
|
||||||
|
|
||||||
|
def rank(rec):
|
||||||
|
"""Survivor order: biggest, then shortest path, then lexicographic.
|
||||||
|
|
||||||
|
A strict total order per key means a key's winner is never itself dominated,
|
||||||
|
so no chain of staging can cycle and exactly one copy of a byte-identical
|
||||||
|
pair is staged."""
|
||||||
|
return (-rec["size"], len(rec["path"]), rec["path"])
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
library = os.path.abspath(ARGS.library)
|
||||||
|
hold = os.path.abspath(ARGS.hold)
|
||||||
|
out = ARGS.out or os.path.join(os.path.dirname(library), "tv-dupes-plan.tsv")
|
||||||
|
if not os.path.isdir(library):
|
||||||
|
sys.exit(f"refused: {library} is not a directory")
|
||||||
|
if hold == library or hold.startswith(library + os.sep):
|
||||||
|
sys.exit(f"refused: --hold {hold} sits inside the library; a hold the"
|
||||||
|
" census walks is a plan that eats its own output")
|
||||||
|
|
||||||
|
files = scan(library, hold, witness_rows(ARGS.witness))
|
||||||
|
folders = sorted({r["rel"].split(os.sep)[0] for r in files})
|
||||||
|
ids = collect_ids(library, ARGS.meta, ARGS.ids)
|
||||||
|
groups, conflicts, unresolved, disagree = show_groups(folders, ids)
|
||||||
|
|
||||||
|
by_show = collections.defaultdict(list)
|
||||||
|
for members in groups:
|
||||||
|
for r in files:
|
||||||
|
if r["rel"].split(os.sep)[0] in members:
|
||||||
|
by_show[members[0]].append(r)
|
||||||
|
|
||||||
|
rows, audits, report, shows_json = [], [], [], []
|
||||||
|
for show in sorted(by_show):
|
||||||
|
recs = by_show[show]
|
||||||
|
holders = collections.defaultdict(list)
|
||||||
|
for r in recs:
|
||||||
|
for k in r["keys"]:
|
||||||
|
holders[k].append(r)
|
||||||
|
winner = {k: min(v, key=rank) for k, v in holders.items()}
|
||||||
|
dominated = [r for r in recs if r["keys"]
|
||||||
|
and all(winner[k] is not r for k in r["keys"])]
|
||||||
|
if not dominated:
|
||||||
|
continue
|
||||||
|
dup_keys = sorted({k for r in dominated for k in r["keys"]})
|
||||||
|
sole = sum(1 for v in holders.values() if len(v) == 1)
|
||||||
|
staged_folders = {os.path.dirname(r["rel"]) for r in dominated}
|
||||||
|
for r in dominated:
|
||||||
|
for k in r["keys"]:
|
||||||
|
t = winner[k]
|
||||||
|
try:
|
||||||
|
now_src = os.path.getsize(r["path"])
|
||||||
|
now_twin = os.path.getsize(t["path"])
|
||||||
|
except OSError:
|
||||||
|
audits.append(f"gone since the census: {r['rel']}"
|
||||||
|
f" or its survivor {t['rel']}")
|
||||||
|
break
|
||||||
|
if now_twin < now_src:
|
||||||
|
audits.append(f"survivor smaller on disk: {r['rel']}"
|
||||||
|
f" {now_src} > {t['rel']} {now_twin}")
|
||||||
|
for r in dominated:
|
||||||
|
dst = os.path.join(hold, r["rel"])
|
||||||
|
rows.append((r["path"], dst))
|
||||||
|
for side in sidecars(r["path"]):
|
||||||
|
rows.append((side, os.path.join(hold, os.path.relpath(side, library))))
|
||||||
|
shows_json.append({"show": show, "files": len(dominated),
|
||||||
|
"bytes": sum(r["size"] for r in dominated),
|
||||||
|
"episodes": len(dup_keys),
|
||||||
|
"sole-episode files left": sole,
|
||||||
|
"folders": sorted(staged_folders)})
|
||||||
|
n_ep, n_folder = len(dup_keys), len(staged_folders)
|
||||||
|
report.append(
|
||||||
|
f"{show}: stage {len(dominated)} files"
|
||||||
|
f" ({sum(r['size'] for r in dominated) / 2**30:.1f} GiB) — {n_ep}"
|
||||||
|
f" episode{n_ep != 1 and 's' or ''} held twice or more; {sole}"
|
||||||
|
f" episode{sole != 1 and 's' or ''} exist in one copy only, so"
|
||||||
|
f" {n_folder} folder{n_folder != 1 and 's' or ''} keep their files")
|
||||||
|
|
||||||
|
rows = sorted(set(rows))
|
||||||
|
dests = set()
|
||||||
|
for src, dst in rows:
|
||||||
|
if dst in dests:
|
||||||
|
audits.append(f"destination twice: {dst}")
|
||||||
|
dests.add(dst)
|
||||||
|
if os.path.exists(dst):
|
||||||
|
audits.append(f"hold target occupied: {dst}")
|
||||||
|
if not os.path.isfile(src):
|
||||||
|
audits.append(f"source missing: {src}")
|
||||||
|
|
||||||
|
unkeyed = collections.Counter(os.path.dirname(r["rel"]) for r in files
|
||||||
|
if not r["keys"])
|
||||||
|
if ARGS.json:
|
||||||
|
with open(ARGS.json, "w") as fh:
|
||||||
|
json.dump({"shows": shows_json,
|
||||||
|
"conflicts": [list(c) for c in conflicts],
|
||||||
|
"unresolved": [list(u) for u in unresolved],
|
||||||
|
"disagreement": {f: [t, d] for f, (t, d) in disagree.items()},
|
||||||
|
"unkeyed": dict(unkeyed)}, fh, indent=1)
|
||||||
|
|
||||||
|
for line in report:
|
||||||
|
print(line)
|
||||||
|
for a, b, why in conflicts:
|
||||||
|
print(f"CONFLICT (never merged): {a} | {b} — {why}")
|
||||||
|
for a, b, why in unresolved:
|
||||||
|
print(f"UNRESOLVED (tag it, then re-run): {a} | {b} — {why}")
|
||||||
|
for f, (tagged, extra) in sorted(disagree.items()):
|
||||||
|
print(f"DISAGREE (the tag wins): {f} — tag tvdb {tagged},"
|
||||||
|
f" derived tvdb {extra}")
|
||||||
|
if unkeyed:
|
||||||
|
print(f"unkeyed, never staged: {sum(unkeyed.values())} files in"
|
||||||
|
f" {len(unkeyed)} folders name no SxxEyy and have no --witness row;"
|
||||||
|
" most: " + ", ".join(f"{d} ({n})" for d, n in unkeyed.most_common(5)))
|
||||||
|
|
||||||
|
# The gate runs before the plan is written: a plan file that exists is a plan
|
||||||
|
# somebody will point the executor at, so a failed audit leaves no artifact.
|
||||||
|
if audits:
|
||||||
|
print(f"AUDIT FAILED ({len(audits)})", file=sys.stderr)
|
||||||
|
for a in audits[:10]:
|
||||||
|
print(f" {a}", file=sys.stderr)
|
||||||
|
print(f"refused: no plan written to {out}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
mkdirs = sorted({os.path.dirname(dst) for _s, dst in rows},
|
||||||
|
key=lambda p: (p.count(os.sep), p))
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
fh.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in mkdirs:
|
||||||
|
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for src, dst in rows:
|
||||||
|
fh.write(f"move\t{src}\t{dst}\ttvdup\n")
|
||||||
|
|
||||||
|
held = sum(os.path.getsize(s) for s, _d in rows if os.path.isfile(s))
|
||||||
|
print(f"plan: {len(rows)} moves ({held / 2**30:.1f} GiB) into"
|
||||||
|
f" {len(mkdirs)} hold folders -> {out}")
|
||||||
|
print("audit: every staged file has a survivor in the same show holding each of"
|
||||||
|
" its episodes at >= its size on disk, one destination each")
|
||||||
|
if rows:
|
||||||
|
root = os.path.dirname(library)
|
||||||
|
print(f"next: python3 scripts/media-apply.py apply --plan {out} --root"
|
||||||
|
f" {root} --journal {os.path.join(root, 'tv-dupes.jsonl')}"
|
||||||
|
" --metadata-view /var/lib/media-meta/view (dry run; add --apply)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
+18
-1
@@ -60,7 +60,7 @@ die() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
|
printf 'usage: %s plan|ensure-base|verify|install|reap|status [options]\n' "$0" >&2
|
||||||
exit 2
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -98,6 +98,22 @@ require_upper_base() {
|
|||||||
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# The SSD placement of the metadata layer is a human choice, so `install` refuses
|
||||||
|
# to invent a base directory: a mistyped --upper-base must not quietly create a
|
||||||
|
# tree on whatever filesystem the caller landed on. The deployer owns the
|
||||||
|
# documented location, so it calls this first — measured 2026-10-31, when a
|
||||||
|
# fresh-host `install` died with "put it on the SSD first" and nothing in git
|
||||||
|
# would have created it.
|
||||||
|
do_ensure_base() {
|
||||||
|
if [ -d "$upper_base" ]; then
|
||||||
|
printf 'base: %s exists\n' "$upper_base"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
install -d -m 0755 "$upper_base" 2>/dev/null \
|
||||||
|
|| die "refused: cannot create $upper_base — it lives outside the checkout, run this as root"
|
||||||
|
printf 'base: created %s\n' "$upper_base"
|
||||||
|
}
|
||||||
|
|
||||||
do_plan() {
|
do_plan() {
|
||||||
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
||||||
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
||||||
@@ -234,6 +250,7 @@ do_status() {
|
|||||||
case "$cmd" in
|
case "$cmd" in
|
||||||
plan) do_plan ;;
|
plan) do_plan ;;
|
||||||
verify) do_verify ;;
|
verify) do_verify ;;
|
||||||
|
ensure-base) do_ensure_base ;;
|
||||||
install) do_install ;;
|
install) do_install ;;
|
||||||
reap) do_reap ;;
|
reap) do_reap ;;
|
||||||
status) do_status ;;
|
status) do_status ;;
|
||||||
|
|||||||
@@ -104,6 +104,12 @@ for path in "${CHANGED[@]}"; do
|
|||||||
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
|
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
|
||||||
add_test tests/media_overlay.bats
|
add_test tests/media_overlay.bats
|
||||||
;;
|
;;
|
||||||
|
# The Audiobookshelf bootstrapper (account + library + scan over the API)
|
||||||
|
# and the deployer hook that runs it after the health gate.
|
||||||
|
scripts/audiobookshelf-init.sh|tests/audiobookshelf_init.bats)
|
||||||
|
add_test tests/audiobookshelf_init.bats
|
||||||
|
add_test tests/deployment_flow.bats
|
||||||
|
;;
|
||||||
shared/homepage/*)
|
shared/homepage/*)
|
||||||
add_test tests/homepage_caddy.bats
|
add_test tests/homepage_caddy.bats
|
||||||
add_test tests/homepage_config.bats
|
add_test tests/homepage_config.bats
|
||||||
@@ -334,7 +340,7 @@ for path in "${CHANGED[@]}"; do
|
|||||||
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
||||||
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
||||||
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
||||||
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*)
|
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*|dankserver/audiobookshelf/*)
|
||||||
add_test tests/dankserver_quadlet.bats
|
add_test tests/dankserver_quadlet.bats
|
||||||
add_test tests/deployment_flow.bats
|
add_test tests/deployment_flow.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ This repository tests its configuration at defined seams. The tests run with `ba
|
|||||||
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
|
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
|
||||||
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
|
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
|
||||||
| E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] |
|
| E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] |
|
||||||
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
|
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{audiobookshelf_init,anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
|
||||||
|
|
||||||
## Focused pre-commit testing
|
## Focused pre-commit testing
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,338 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
description: "Audiobookshelf on Dankserver: quadlet shape, array bind, SSD state, API bootstrap, authentik SSO, public vhost, laws."
|
||||||
|
tags:
|
||||||
|
- dankserver
|
||||||
|
- audiobooks
|
||||||
|
- quadlet
|
||||||
|
timestamp: 2026-11-03
|
||||||
|
---
|
||||||
|
|
||||||
|
# Audiobookshelf
|
||||||
|
|
||||||
|
Audiobookshelf (ABS) serves the [[Entropy Volume]] audiobook library to browsers
|
||||||
|
and phones: it scans `/mnt/entropy/Audiobooks`, keeps its own SQLite library with
|
||||||
|
per-listener playback positions, and streams the audio. It is **not** a media
|
||||||
|
server — it shares nothing with [[Plex]] or [[Jellyfin]] and owns no files on the
|
||||||
|
array.
|
||||||
|
|
||||||
|
**Host:** Dankserver — **Unit:** `audiobookshelf.service` (rootless Quadlet, user manager) —
|
||||||
|
**Doors:** LAN `http://192.168.0.235:8000/audiobookshelf` (plain HTTP, nothing in front) and public `https://audiobooks.franksdank.cloud/audiobookshelf/` — Caddy straight to the container, **exempt from the guard chain** because the app must answer non-browser clients ([[Caddy]] *Unguard a vhost*). The public name is the address every client carries: Pi-hole answers it with `192.168.0.143` on the LAN, Porkbun answers it with the WAN address outside, and Caddy serves a real certificate either way.
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|-------|
|
||||||
|
| Unit file | `dankserver/audiobookshelf/quadlet/audiobookshelf.container` (the authority for every number here) |
|
||||||
|
| Image | `ghcr.io/advplyr/audiobookshelf` pinned by **amd64 manifest digest** — see Laws |
|
||||||
|
| Port | `Environment=PORT=8000` + `PublishPort=8000:8000` — web UI and API on one port |
|
||||||
|
| Base path | `Environment=ROUTER_BASE_PATH=/audiobookshelf` — every URL below carries this prefix, and the shipped SPA cannot boot without it (Laws) |
|
||||||
|
| Network | `pasta` (standalone; publishes nothing to other containers) |
|
||||||
|
| Audio bind | `/mnt/entropy/Audiobooks:/audiobooks:ro` — read-only, by design |
|
||||||
|
| State | `/home/dank/Containers/audiobookshelf/config` → `/config`, `…/metadata` → `/metadata` (both NVMe) |
|
||||||
|
| Mount gate | `ExecStartPre=…/wait-for-mount.sh /mnt/entropy` |
|
||||||
|
| Health | `wget` against `/healthcheck` inside the container, 30 s / 3 retries |
|
||||||
|
| I/O share | `IOWeight=10` — a scan loses the array to every other tenant |
|
||||||
|
| Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` |
|
||||||
|
| Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` |
|
||||||
|
| Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` |
|
||||||
|
| Single sign-on | authentik provider `audiobookshelf`; the local `frank` account stays enabled as break-glass — see *Single sign-on* |
|
||||||
|
| Library layout | one book per folder — ABS groups by the first directory holding a media file (see Laws); every move is journalled at `/mnt/entropy/.media-campaign/media-relayout-audiobooks.jsonl` |
|
||||||
|
|
||||||
|
## Bootstrap
|
||||||
|
|
||||||
|
`scripts/audiobookshelf-init.sh` — run by the deployer after the health gate, so a
|
||||||
|
deployed ABS is a *usable* ABS: `/healthcheck` answers 200 on a server with no
|
||||||
|
user, no library and no scan, and nothing else in the deploy notices the
|
||||||
|
difference. The script is idempotent; re-running it costs one incremental scan.
|
||||||
|
|
||||||
|
| Step | Door | Printed |
|
||||||
|
|------|------|---------|
|
||||||
|
| credential | `~/.config/secrets/audiobookshelf` (mode 600, generated once, host-side) | `credential: created …` |
|
||||||
|
| account | `POST /init` with `{"newRoot":{"username","password"}}` | `account: created (frank)` / `already registered` |
|
||||||
|
| session | `POST /login` → bearer token | `token: obtained (root)` |
|
||||||
|
| library | `POST /api/libraries`, folder `/audiobooks`, provider `audiobook` | `library: created Audiobooks (<id>)` / `exists` |
|
||||||
|
| scan | `POST /api/libraries/<id>/scan` | `scan: accepted` |
|
||||||
|
| proof | `GET /api/libraries/<id>/items?limit=1` | `items: <total>` |
|
||||||
|
|
||||||
|
Sign in at `http://192.168.0.235:8000/audiobookshelf` as `frank`; the password is
|
||||||
|
`awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' ~/.config/secrets/audiobookshelf` on
|
||||||
|
Dankserver. Change it in **Settings → Users → frank → Password**; the secret file
|
||||||
|
then holds a stale value and only matters again if the `config` bind is ever
|
||||||
|
destroyed (a fresh DB makes the script create a new owner from it).
|
||||||
|
|
||||||
|
A full scan is slow and runs in the background; watch the `items:` total climb,
|
||||||
|
never `/healthcheck`. It also reads files that are genuinely broken and logs
|
||||||
|
`[AudioFileScanner] Invalid data found when processing input` per file — that is
|
||||||
|
the file, not the bind: the same line count grows while the total climbs.
|
||||||
|
|
||||||
|
## Phone app
|
||||||
|
|
||||||
|
The official **Audiobookshelf** app connects with Server Address
|
||||||
|
`https://audiobooks.franksdank.cloud/audiobookshelf` — the base path belongs in the address —
|
||||||
|
username `frank`, and the same password as the web UI, or the SSO button. One address works
|
||||||
|
at home and away: on the LAN Pi-hole answers that name with `192.168.0.143` and Caddy dials
|
||||||
|
the container; off-LAN the same name resolves to the WAN address and takes the same route
|
||||||
|
(inbound 443 is forwarded at the router — fetched from a datacenter node 2026-11-03). The raw
|
||||||
|
LAN address still answers and is the break-glass door whenever Caddy is down. `/ping`
|
||||||
|
(`{"success":true}`) and `/healthcheck` (`OK`) answer at the ROOT whatever `ROUTER_BASE_PATH`
|
||||||
|
says, so the app's connectivity probe works before it resolves the base path at all, and
|
||||||
|
`download: true` on the account is what makes offline listening work.
|
||||||
|
|
||||||
|
**A home device that bypasses Pi-hole cannot reach it.** The G36 does not hairpin: resolving
|
||||||
|
the name to the WAN address from inside the LAN times out (measured 2026-11-03). Only a
|
||||||
|
client pinned to an external resolver hits this — everything using the router's DNS gets the
|
||||||
|
LAN answer.
|
||||||
|
|
||||||
|
**The login token is at `user.token`, not top level.** `POST /login` answers
|
||||||
|
`{ user, serverSettings, ereaderDevices, Source, userDefaultLibraryId }` and the bearer sits
|
||||||
|
inside the user object. A client that reads `response.token` gets the empty string, sends
|
||||||
|
`Authorization: Bearer ` with nothing after it, and every `/api/*` call then answers `401`
|
||||||
|
as though the password were wrong.
|
||||||
|
|
||||||
|
## Listeners (accounts)
|
||||||
|
|
||||||
|
The door is public now, so the account is the only gate between a listener and the library.
|
||||||
|
Two ways in, both per listener:
|
||||||
|
|
||||||
|
- **Local account** — **Settings → Users → Add User**: type `user`, tick only the libraries
|
||||||
|
they may hear, and leave `download` off unless they need offline listening. Hand them the
|
||||||
|
server address above and their own password.
|
||||||
|
- **Household SSO** — create the authentik account, add it to `audiobookshelf_users`, and the
|
||||||
|
first **Sign in with Authentik** auto-registers the ABS account (`authOpenIDAutoRegister`);
|
||||||
|
a signer in neither group is denied on purpose, after a successful authentication.
|
||||||
|
|
||||||
|
Nothing rate-limits `/audiobookshelf/api/login` — this Caddy build carries no `rate_limit`
|
||||||
|
module and ABS 2.37.2 ships no lockout — so the form is a guessing target exactly like
|
||||||
|
`jelly.`, `plex.` and the *arr logins already are. The mitigation is credential hygiene:
|
||||||
|
long unique passwords per listener, never a reuse of the root password.
|
||||||
|
|
||||||
|
## Single sign-on (authentik)
|
||||||
|
|
||||||
|
Both doors take the household identity: the web UI puts **Sign in with Authentik** beside
|
||||||
|
the local form, and the phone app offers SSO on its server-address screen. The local
|
||||||
|
`frank` account stays enabled on purpose — `authActiveAuthMethods` is `local` **and**
|
||||||
|
`openid`, and `local` is how you get in when the IdP is down.
|
||||||
|
|
||||||
|
| Piece | Owner |
|
||||||
|
|-------|-------|
|
||||||
|
| Provider `audiobookshelf`, the two groups, the scope mapping | `scripts/authentik-audiobookshelf-sso.sh` — re-run it after an authentik rebuild, because this state lives in the IdP's database and a blueprint re-import reverts it silently |
|
||||||
|
| Client secret | `ABS_OIDC_CLIENT_SECRET` in `~/.config/secrets/audiobookshelf` on Dankserver (mode 600; never in the repo, a URL or an argv) |
|
||||||
|
| The ABS half: issuer, the four IdP URLs, claim, auto-register | `scripts/audiobookshelf-init.sh`, armed automatically when that secret exists in the credential file |
|
||||||
|
| Role at first SSO sign-in | the `abs_groups` claim: `audiobookshelf_admins` → `admin`, `audiobookshelf_users` → `user`, in neither → denied (fail closed) |
|
||||||
|
| Admin group members | `akadmin` (the operator's own authentik admin account — sign in as it and the claim grants site admin) and `qa-e2e-admin` (the identity the deploy gate signs in as, so the script re-adds it every run; drop it and the gate fails, not a phone) |
|
||||||
|
| Guard state of the door | none since 2026-11-03: proxy provider `audiobooks-proxy` and application "Audiobookshelf Guard" were deleted and the embedded outpost un-claimed them, because that chain answers a native client with challenge HTML ([[Caddy]] *Unguard a vhost* for the order). The SSO relying party `audiobookshelf` above is a different object and stays |
|
||||||
|
| Redirect URIs on the RP | `<base>/auth/openid/callback` **and** `<base>/auth/openid/mobile-redirect` for **both** origins (LAN address and public name): the app sends the mobile URI belonging to the door it was configured with, and STRICT matching answers an unregistered one with "Redirect URI Error" |
|
||||||
|
|
||||||
|
The IdP saying yes is not the same as being allowed in: with `authOpenIDAutoRegister` on,
|
||||||
|
a first sign-in creates the account, and the claim alone decides the type. A user in
|
||||||
|
neither group is refused after a successful authentication — that is the mapping working,
|
||||||
|
not a broken login.
|
||||||
|
|
||||||
|
Both halves — authentik's provider row and ABS's settings — are asserted end to end by the Playwright gate named in *Verification*: a sign-in as `qa-e2e-admin` that must land inside the app with site-admin rights.
|
||||||
|
|
||||||
|
## Laws
|
||||||
|
|
||||||
|
- **Speak the image's env names, not the README's.** The server reads `PORT`,
|
||||||
|
`CONFIG_PATH`, `METADATA_PATH` and `ROUTER_BASE_PATH`; the `ABS_*` family printed
|
||||||
|
by the server's own README is the npm package's naming and is **inert** in this
|
||||||
|
image. Measured 2026-11-02 on the first deploy: with `ABS_PORT=8000` the server
|
||||||
|
listened on `:80` under base path `/audiobookshelf`, so `systemctl` said
|
||||||
|
`active (running)` while every probe of `:8000` was refused and the deploy gate
|
||||||
|
failed at 180 s. Two seconds with the pinned digest settles it:
|
||||||
|
`podman inspect --format '{{.Config.Env}}' <image>`.
|
||||||
|
- **The URL carries `/audiobookshelf`, and that is not a style choice.** The SPA in
|
||||||
|
`/app/client/dist/index.html` is built with `routerBasePath:"/audiobookshelf"`, so it asks
|
||||||
|
for its assets under that prefix whatever the server is mounted at. Mounted at `/`, every
|
||||||
|
script request falls through to the SPA fallback, comes back `text/html`, Chrome refuses to
|
||||||
|
execute it, and the page never leaves "Loading…" — while `/healthcheck`, the API and the
|
||||||
|
phone app stay perfectly healthy, which is how this hid behind an API-only verification
|
||||||
|
(measured 2026-11-02). Prove a UI claim in a browser, never from `/health`.
|
||||||
|
- **The OIDC redirect subfolder is a value you send, not a default you inherit.** The 2.17.4
|
||||||
|
upgrade only fills `authOpenIDSubfolderForRedirectURLs` on a database that ALREADY had OIDC
|
||||||
|
enabled, so a DB created before SSO leaves it unset and the server builds
|
||||||
|
`http://host/undefined/auth/openid/callback`. authentik answers the authorize request with
|
||||||
|
"Redirect URI Error" and nothing on the ABS side looks wrong (measured 2026-11-02). The
|
||||||
|
provider's registered URI must equal `<origin><subfolder>/auth/openid/callback`, so each
|
||||||
|
script derives the value from one variable of its own (`ABS_URL`, `ABS_BASE`) and
|
||||||
|
`tests/audiobookshelf_init.bats` pins what reaches the wire.
|
||||||
|
- **An ORM-created provider has no grant types.** authentik 2026.8 defaults `grant_types` to
|
||||||
|
`[]`, and `/application/o/authorize/` then refuses the code flow with oauthlib's generic
|
||||||
|
"The request is otherwise malformed" — the wire names nothing, the server log line that names
|
||||||
|
it is `Invalid grant_type for provider`. Worse, the redirect-URI check runs *first*, so a
|
||||||
|
provider carrying both faults shows only "Redirect URI Error" until the URI is fixed, and the
|
||||||
|
second wall appears after the fix ships (measured 2026-11-02: the browser reached authentik,
|
||||||
|
got `code`+`state`, and ABS still bounced to `/login?error=undefined`). The script sets
|
||||||
|
`authorization_code` + `refresh_token`; a blueprint re-import reverts the row silently.
|
||||||
|
- **The guarded door has to tell the truth about its scheme.** ABS validates every OIDC
|
||||||
|
`callback` against the origin it believes it is served from — `getRequestOrigin()` is
|
||||||
|
`x-forwarded-proto://Host` (`server/utils/requestUtils.js`, read in the pinned digest
|
||||||
|
2026-11-03) — so behind a TLS edge that reaches the container over plain http it computes
|
||||||
|
`http://audiobooks.franksdank.cloud` and answers its own https callback with a 400 "Invalid callback
|
||||||
|
URL - must be same-origin" before the IdP is ever contacted. The Caddy `:8017` route pins
|
||||||
|
`header_up X-Forwarded-Proto https` for exactly that reason ([[Caddy]] *Guard a new vhost*);
|
||||||
|
the LAN door needs nothing, because there browser and server agree on `http`.
|
||||||
|
- **The claim mapping is not the whole `property_mappings` list.** A provider holding only the
|
||||||
|
custom mapping grants a token scoped shorter than the scopes ABS requests
|
||||||
|
(`openid profile email abs_groups`): the code is issued and exchanged normally, and ABS's
|
||||||
|
`/userinfo` call then dies `insufficient_scope (The request requires higher privileges than
|
||||||
|
provided by the access token)` — a fault that reads like an ABS bug and is an IdP row. Attach
|
||||||
|
the built-in OpenID `openid`/`profile`/`email` mappings beside the claim mapping, which is
|
||||||
|
what every working RP on this IdP carries (measured 2026-11-02).
|
||||||
|
- **With `authOpenIDMatchExistingBy` unset, an SSO identity never lands on an existing ABS
|
||||||
|
account, and a name collision is a hard denial.** In that mode `findUserFromOpenIdUserInfo`
|
||||||
|
matches on `sub` only, so every first sign-in auto-registers a NEW user; if the IdP username
|
||||||
|
equals an ABS username — the local root `frank` is the one here — the create hits the unique
|
||||||
|
constraint and the callback dies `Unauthorized`. So "create a `frank` user in authentik" does
|
||||||
|
not reach the root account: either set `authOpenIDMatchExistingBy` to `username` (links the
|
||||||
|
IdP account to the existing one, and `setUserGroup` refuses to downgrade a root — root plus an
|
||||||
|
`admin` claim is allowed, root plus `user` is denied) or pick a non-colliding IdP name. Read
|
||||||
|
off `server/models/User.js` and `server/auth/OidcAuthStrategy.js` in the pinned digest
|
||||||
|
(2026-11-02).
|
||||||
|
- **The audio bind stays `:ro`.** ABS writes nothing that belongs beside the
|
||||||
|
audio: its library, covers, progress and sidecar metadata all land in
|
||||||
|
`/metadata` on the SSD. A writable bind would put stray writes on the spindle
|
||||||
|
and hand the array to a service with no reason to hold it. If per-book
|
||||||
|
`.opf`/`.nfo` sidecars on disk are ever wanted, the rollback is one line —
|
||||||
|
point the audio bind at the [[Media Metadata View]] the media servers already
|
||||||
|
use — but that view's upper layer exists for those two servers' sidecars, so a
|
||||||
|
third writer needs a reason first ([[Media Metadata View]]).
|
||||||
|
- **Do not bind the raw array without the mount gate.** In a rootless unit
|
||||||
|
`ConditionPathIsMountPoint=` is fail-*dead* and `RequiresMountsFor=` is inert,
|
||||||
|
so a missing array does not stop the start: podman invents an empty directory,
|
||||||
|
ABS indexes it, and an empty library becomes recorded truth. `wait-for-mount.sh`
|
||||||
|
is the only gate that works here — same law the media servers obey
|
||||||
|
([[Quadlet Lifecycle]]).
|
||||||
|
- **The `HealthCmd` must use `wget`.** The image is Alpine and ships `wget`, not
|
||||||
|
`curl`; a `curl` health command reads a perfectly healthy server as unhealthy
|
||||||
|
and `Restart=always` turns that into a loop.
|
||||||
|
- **The image pin is the amd64 manifest digest, not the index digest.**
|
||||||
|
`podman inspect` reports the platform manifest (`RepoDigests[0]`) for what was
|
||||||
|
pulled, which differs from the manifest-list digest `latest` points at. Pin the
|
||||||
|
one this host runs. `latest` is newer than the newest version tag: upstream
|
||||||
|
stopped publishing version tags after `2.19.0` while `latest` kept moving, so
|
||||||
|
"highest version tag" is stale here — read the index, don't guess from tags.
|
||||||
|
- **`IOWeight=10` is inert without the host drop-ins.** `dankserver/host/systemd/`
|
||||||
|
carries `user.slice.d/io-accounting.conf` and the `user@1000` delegate, and the
|
||||||
|
deployer installs both; without them systemd silently drops the weight
|
||||||
|
([[Quadlet Lifecycle]]).
|
||||||
|
- **No `systemctl --user enable`.** Generated Quadlet units cannot be enabled on
|
||||||
|
this host; `WantedBy=default.target` in the unit file is what carries the
|
||||||
|
service across a reboot ([[Quadlet Lifecycle]]).
|
||||||
|
- **One door for every client, and it sits outside the guard chain on purpose.**
|
||||||
|
`https://audiobooks.franksdank.cloud/audiobookshelf` is what browsers and apps use;
|
||||||
|
`http://192.168.0.235:8000/audiobookshelf` is the raw LAN socket that same Caddy dials,
|
||||||
|
so it doubles as break-glass when the proxy is down. Behind the chain (until
|
||||||
|
2026-11-03) the vhost answered the phone app's `/ping` and `/socket.io/` handshake with
|
||||||
|
Anubis' challenge HTML — Anubis demands a JavaScript proof and the outpost needs a
|
||||||
|
session cookie, and a native client solves neither — so the app evaluated
|
||||||
|
`'isInit' in e.data` on a non-object, reported "Failed to contact server", and the
|
||||||
|
container's request log never recorded the attempt. Podcast RSS and OPDS readers died
|
||||||
|
the same death. ABS's own auth is the gate, as it is for [[Plex]] and [[Jellyfin]]
|
||||||
|
([[Caddy]] *Unguard a vhost*).
|
||||||
|
- **There is no `/setup` route; the creation door is `POST /init`.** `/status`
|
||||||
|
(no `/api` prefix) reports `isInit` = "a root user exists", `POST /init` with
|
||||||
|
`{"newRoot":{"username","password"}}` creates it, `POST /login` returns the
|
||||||
|
bearer every `/api/*` route wants. `POST /setup` answers `Unauthorized` — that
|
||||||
|
is a missing route wearing an auth message. Read off `dist-server/server/Server.js`
|
||||||
|
inside the pinned digest rather than from a forum.
|
||||||
|
- **A 200 may carry plain text.** `res.sendStatus(200)` answers the body `OK`, and
|
||||||
|
`/init` and the scan route use it; a client that parses every success as JSON
|
||||||
|
crashes on the one call that matters (`tests/audiobookshelf_init.bats` pins it).
|
||||||
|
- **The item count is not a field on the library DTO, and paging never ends
|
||||||
|
itself.** The door is `GET /api/libraries/<id>/items?limit=1` → `total`; a poller
|
||||||
|
that guesses a plausible name (`numItems`) prints `0` forever beside a scan that
|
||||||
|
is working. Paging is on `page` (0-based) plus `limit`, and **`offset` is
|
||||||
|
ignored**: `limit=500&offset=500` returns the same 500 ids as `offset=0`, while
|
||||||
|
`page=1` shares none of them. A census that pages on `offset` therefore reads
|
||||||
|
`ceil(total / limit) × limit` rows containing only `limit` distinct items — the
|
||||||
|
exact origin of the "exactly 1,500" and "exactly 4,000" counts two runs
|
||||||
|
reported for one library. Page on `page`, count *unique ids*, stop on `total`,
|
||||||
|
and take the number after the pass settles (`total` moves during a scan).
|
||||||
|
- **A fresh scan is the only thing that notices new audio.** `inotify` does not
|
||||||
|
cross a rootless podman bind — measured for [[Jellyfin]], whose page owns that
|
||||||
|
law — so the in-container watcher sees nothing a host process files onto the
|
||||||
|
array, and the deployer's re-run of the bootstrapper is the cheap scan trigger.
|
||||||
|
- **One stray media file at a collection root turns the whole collection into one
|
||||||
|
book.** Every scan re-walks the tree from disk (`scanFolder` → `recurseFiles`),
|
||||||
|
and `utils/scandir.js:groupFileItemsIntoLibraryItemDirs` assigns each file to the
|
||||||
|
FIRST directory along its path that already has a group: the first file seen at a
|
||||||
|
root owns every deeper file, and the folders below it never become items. One
|
||||||
|
949 MB `.cbr` sitting at `_unsorted`'s root collapsed 652 book folders into a
|
||||||
|
single item of 5,693 tracks, titled after whichever file supplied metadata.
|
||||||
|
"Hundreds of books are missing" is a layout fault, not a scan fault.
|
||||||
|
- **"Media file" means audio OR ebook, so an audio-only census cannot see the file
|
||||||
|
that is hiding the library.** With `audiobooksOnly=false` (the deployed shape)
|
||||||
|
the predicate is `SupportedAudioTypes` ∪ `SupportedEbookTypes`: 20 audio
|
||||||
|
extensions (including `mp4`, `mpg`, `webm`) plus `epub pdf mobi azw3 cbr cbz`. A
|
||||||
|
census reporting "that root holds no loose audio" can be correct and still
|
||||||
|
useless — the file doing the shadowing here was a comic archive.
|
||||||
|
- **Deleting the covering item does not un-shadow anything.** Grouping is recomputed
|
||||||
|
from disk, so the next scan rebuilds the item from the same stray file (measured:
|
||||||
|
`DELETE`, then scan → `1 Added`, the item back at the same path). The fix is the
|
||||||
|
MOVE: give every stray root file a folder of its own (one file per folder is
|
||||||
|
ABS's own single-book layout; formats of one book share a folder), through the
|
||||||
|
journal-guarded mover, then delete the stale item so the old path is not kept,
|
||||||
|
then scan.
|
||||||
|
- **`?force=1` is neither a discovery arm nor a repair arm.** Both arms of
|
||||||
|
`POST /libraries/:id/scan` walk the disk, so finding new folders needs neither
|
||||||
|
and the deployer's cheap trigger stays on the plain arm. Force additionally
|
||||||
|
re-scans every item that already exists (measured 1:33 against 0:04 on ~4,600
|
||||||
|
items) and repaired nothing on a library whose residuals were file-level: item
|
||||||
|
count, path set and media-less count identical before and after.
|
||||||
|
- **`DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()`.** In
|
||||||
|
`LibraryItemController.delete`, `hardDelete = req.query.hard == 1` and then
|
||||||
|
`fs.remove(libraryItemPath)`. Pointed at a covering item, that path is the whole
|
||||||
|
swallowed subtree — `hard=1` there deletes every book beneath it off the array.
|
||||||
|
Assert the item's `path` first, and never pass `hard`.
|
||||||
|
- **`Invalid data found when processing input` names the file, not the bind.** The
|
||||||
|
scanner logs it per file and skips that file; a book whose files ALL fail never
|
||||||
|
becomes an item at all, so the disk-vs-DB folder census shows it as a book with
|
||||||
|
no item. That is corrupt media, not a lost book. Never clear a suspect file with
|
||||||
|
a lenient probe: `ffprobe -show_entries format=duration` happily returns three
|
||||||
|
hours for the WAV bytes wearing a `.mp3` name that the scanner refuses. Read the
|
||||||
|
log's file paths instead, and `stat -c %s` first — 0-byte files are the common
|
||||||
|
case, and a truncated download reports `moov atom not found`.
|
||||||
|
- **A `Disc 01` / `CD 1` subfolder is part of a book, not a book.** The extension
|
||||||
|
census counts those folders; ABS deliberately does not — its grouping attaches
|
||||||
|
their files to the ancestor item (measured: 110 such folders under one item
|
||||||
|
carrying all 192 tracks). Read the basename before believing a disk-vs-DB gap.
|
||||||
|
- **`.cbr`/`.cbz` group like media and never play.** They sit in the grouping
|
||||||
|
predicate, so they hide books, but the comic parser throws `ERAR_BAD_ARCHIVE`
|
||||||
|
(`parseComicMetadata` → `CbrComicBookExtractor`) and the item lands with no
|
||||||
|
media. A media-less row beside those lines is the file, not the scanner.
|
||||||
|
- **A scan is finished when a NEW scan uuid completes.** Match
|
||||||
|
`Library scan "<uuid>" completed in ...`. `podman logs --since 30m` is a sliding
|
||||||
|
window: the set of visible completion lines changes on its own as old lines age
|
||||||
|
out, so a poller keyed on that hash reads a mid-scan number and calls it done.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
`./tests/run.sh` covers this unit through `tests/dankserver_quadlet.bats`
|
||||||
|
(boot persistence, the SSD state roots, the `:ro` array bind, the mount gate, the `wget`
|
||||||
|
health check, the base path the shipped SPA needs), `tests/audiobookshelf_init.bats` (the
|
||||||
|
bootstrapper against a stub of the API above: create-once, idempotent re-run, no secret in a
|
||||||
|
URL, refusal by URL when the server does not answer, the deployer calling it only after the
|
||||||
|
health gate, and the SSO arm sending the mounted base path as the redirect subfolder);
|
||||||
|
`tests/media_overlay.bats` (only [[Plex]] and [[Jellyfin]] may reference the metadata view);
|
||||||
|
and the live Playwright gate `tests/e2e/specs/23-audiobookshelf-sso.spec.ts`, run by
|
||||||
|
`tests/e2e/run.sh` at the end of every `deploy.sh` — the `admin` project signs in as
|
||||||
|
`qa-e2e-admin` against `http://192.168.0.235:8000/audiobookshelf` and asserts the two things
|
||||||
|
no stub can see: every `.js` the app loads arrives as JavaScript (a bundle built for the wrong
|
||||||
|
base path arrives as `text/html`, which boots nothing), and `/api/me` answers 200 with
|
||||||
|
`type: "admin"` — true only while grant types, the OpenID scope mappings and the group claim
|
||||||
|
all hold together.
|
||||||
|
|
||||||
|
The public door is pinned by the guard audit `tests/e2e/specs/14-guard.spec.ts`, which now
|
||||||
|
carries an **exemption** assertion where it used to carry a guarded one: an anonymous raw
|
||||||
|
client must get the app's own `{"success":true}` on `/ping`, a machine-format socket
|
||||||
|
handshake, and the app's own `401` on `/audiobookshelf/api/libraries` — never challenge
|
||||||
|
HTML. `tests/anubis.bats` pins the same shape in the Caddyfile: the vhost never proxies to
|
||||||
|
`anubis:8923`, and the post-auth listener carries no `@audiobooks` route.
|
||||||
|
|
||||||
|
Five book folders hold media no scanner can read, so they stay out of the library:
|
||||||
|
three 2024 Nebula nominees in the `_unsorted` box (two `.m4b` with no `moov` atom,
|
||||||
|
one 36-file `.mp3` set that is not MPEG audio at all), `Anthony Heald/Bounty
|
||||||
|
Hunter Wars - Book 1` and `Star Wars Full Cast Audio Drama/Dark Empire III` — the
|
||||||
|
latter two also exist in readable form under their author folders. Fix by
|
||||||
|
re-acquiring the files; the scanner names every one of them in its log.
|
||||||
@@ -66,3 +66,24 @@ exactly describes the year this bug survived.
|
|||||||
volume absent they start `active (healthy)` against empty directories, which is
|
volume absent they start `active (healthy)` against empty directories, which is
|
||||||
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
||||||
[[Storage Policy]] for what may be written here.
|
[[Storage Policy]] for what may be written here.
|
||||||
|
|
||||||
|
## Write doors
|
||||||
|
|
||||||
|
Something renamed movie folders on this array with no journal in any of the
|
||||||
|
`/mnt/entropy/*.jsonl` plans, no bash history, no cron, no timer, and no job
|
||||||
|
artifact. The census of things that can mutate it:
|
||||||
|
|
||||||
|
- **A local process** as `dank` or root. `scripts/media-apply.py` is the only
|
||||||
|
one that journals; anything else (a hand `mv`, another session's script) is
|
||||||
|
silent.
|
||||||
|
- **SFTP by a human.** GNOME Files on a workstation mounts the array through
|
||||||
|
`gvfsd-sftp`, which runs `sftp-server` on this host as `dank`. Renames and
|
||||||
|
moves are direct `rename()` calls: no shell, no history, no journal. A folder
|
||||||
|
rename bumps **only that directory's ctime** — its mtime and every child's
|
||||||
|
ctime stay at their old values, so the files look untouched while their paths
|
||||||
|
are gone. Diagnose it with `pgrep -af gvfsd-sftp` on the workstation and
|
||||||
|
`ss -tnp` for the peer.
|
||||||
|
- **Copyparty is not a door**: the bind is `ro` at the mount level, so no HTTP
|
||||||
|
or WebDAV write reaches the array regardless of account flags ([[Copyparty]]).
|
||||||
|
- **No SMB or NFS server exists here** — the `*arr` stack on FrankBook-Pro
|
||||||
|
cannot reach the array at all, so it cannot rename, import, or rescan it.
|
||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Jellyfin
|
title: Jellyfin
|
||||||
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
|
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
|
||||||
tags: [jellyfin, dankserver, media-server]
|
tags: [jellyfin, dankserver, media-server]
|
||||||
timestamp: 2026-10-30T00:00:00Z
|
timestamp: 2026-11-02T11:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Jellyfin
|
# Jellyfin
|
||||||
@@ -18,7 +18,7 @@ clients.
|
|||||||
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||||
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
||||||
|
|
||||||
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
|
## Facts (the repo quadlet — deployed as the live unit on 2026-10-31)
|
||||||
|
|
||||||
| Item | Value |
|
| Item | Value |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
||||||
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
||||||
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
||||||
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
|
| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**, live since the 2026-10-31 cutover: sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched |
|
||||||
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
||||||
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
||||||
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
||||||
@@ -37,26 +37,143 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
||||||
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
||||||
/var/lib/media-meta/view -> /data (git: the overlay view)
|
/var/lib/media-meta/view -> /data (git: the overlay view)
|
||||||
/mnt/entropy -> /data (live today: the array, read-only)
|
/mnt/entropy -> /data (the overlay's lower layer; also the rollback bind)
|
||||||
```
|
```
|
||||||
|
|
||||||
Library folders point at `/data` paths: stable across container recreation and
|
Library folders point at `/data` paths: stable across container recreation and
|
||||||
identical to the raw array layout below. Which side of that path the container
|
identical to the raw array layout below. Which side of that path the container
|
||||||
is on is the open question on this host — the installed unit binds the read-only
|
sees is the [[Media Metadata View]] overlay, live since the 2026-10-31 cutover:
|
||||||
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
|
`/var/lib/media-meta/view:/data`, read-write.
|
||||||
|
|
||||||
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
||||||
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
|
reason the view exists: `movie.nfo`, `tvshow.nfo` and `episode.nfo` written once
|
||||||
written once on the NVMe upper layer are files [[Plex]] can read too. While
|
on the NVMe upper layer are files [[Plex]] can read too — and since the
|
||||||
`/data` is bound read-only every item update logs the error below and no sidecar
|
2026-10-31 cutover that is what happens: Jellyfin is the fleet's only metadata
|
||||||
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
|
writer, and every sidecar it saves is an SSD byte the array never sees.
|
||||||
that raises the view.
|
**Artwork is not one of those sidecars.** No image saver is bound for any
|
||||||
|
library, so posters and fanart go to the internal store
|
||||||
|
(`config/metadata/library/<xx>/<item-id>/poster.jpg`), which [[Plex]] never
|
||||||
|
reads — movie art has never existed beside the media on either layer.
|
||||||
|
|
||||||
|
**Library coverage is the first thing to check when a title is missing.** The
|
||||||
|
four libraries are `movies → /data/Movies`, `tvshows → /data/TVShows`,
|
||||||
|
`music → /data/Music`, `boxsets → /config/data/collections`
|
||||||
|
(`GET /Library/VirtualFolders`). The review pile — `/data/NeedsProcessing`, the
|
||||||
|
former `media-park`, drained into it 2026-11-02 — is in **none** of them, so
|
||||||
|
everything filed there is invisible to Jellyfin by configuration,
|
||||||
|
not by fault: a recursive refresh answers 204, burns CPU for minutes and
|
||||||
|
indexes nothing, because it never looks there. Check coverage before re-scanning.
|
||||||
|
|
||||||
|
**Which read proves an item is indexed: the folder view, never the recursive type
|
||||||
|
list.** `GET /Items?IncludeItemTypes=Movie&Recursive=true` answered 2,101 items on
|
||||||
|
2026-11-02 while `GET /Items?ParentId=<Movies ItemId>` listed 2,548 children — every
|
||||||
|
folder on disk — and the eight folders the recursive read "lost" (three `Gundam 00
|
||||||
|
Special Edition`, five `Sailor Moon Memorial`) each answer as `Type: Movie` through
|
||||||
|
`SearchTerm` and sit in the folder listing. The two reads are not interchangeable, so
|
||||||
|
the recursive list is a wrong instrument for "did the refresh index what I moved": its
|
||||||
|
shortfall is grouping, not missing media, and acting on it starts a scan that fixes
|
||||||
|
nothing. Verify per item — `ParentId` listing or `SearchTerm` — and use the recursive
|
||||||
|
count only as a trend line.
|
||||||
|
|
||||||
**Failure signature:** `System.IO.IOException: Read-only file system :
|
**Failure signature:** `System.IO.IOException: Read-only file system :
|
||||||
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
||||||
instead of the view. On Dankserver that is today's expected state — the view has
|
instead of the view — always a fault on this host since the 2026-10-31 cutover.
|
||||||
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
|
`scripts/media-view.sh status` says which: the quadlet drifted back to the raw
|
||||||
same line is a fault: the quadlet drifted or the mount died.
|
array, or the mount died.
|
||||||
|
|
||||||
|
**`EnableInternetProviders` false turns a library into a silent file index.**
|
||||||
|
Every refresh answers `204`, stores nothing and logs nothing — no provider ids,
|
||||||
|
no plot, no rating, no art — while the item count reads like a healthy scan. The
|
||||||
|
switch lives in `config/root/default/<Library>/options.xml`, Jellyfin 12's name
|
||||||
|
for the per-library option file: grepping for the old `library.xml` concludes "no
|
||||||
|
per-library options exist", and the database holds none, so the trail dies there.
|
||||||
|
Two more traps in the same file: `TypeOptions/ImageFetchers` is the *enabled* set,
|
||||||
|
so an empty `<ImageFetchers />` switches art off while `ImageFetcherOrder` stays
|
||||||
|
fully populated — order is not enablement — and the populated `<MetadataFetchers>`
|
||||||
|
built next to it makes the library look healthy.
|
||||||
|
|
||||||
|
**Live posture, measured 2026-11-02** from `GET /Library/VirtualFolders`, cross-checked
|
||||||
|
against each library's `config/root/default/<Library>/options.xml`. The switches are NOT
|
||||||
|
top-level DTO keys: the folder carries `Name`, `CollectionType`, `Locations`, `ItemId`,
|
||||||
|
`PrimaryImageItemId`, `RefreshStatus` and `LibraryOptions`, and every per-library option
|
||||||
|
sits inside `LibraryOptions`. Reading `EnableInternetProviders` off the folder itself
|
||||||
|
answers `None` for all four — the camelCase trap one level deeper, and it looks exactly
|
||||||
|
like an unconfigured library.
|
||||||
|
|
||||||
|
| Library | `CollectionType` | `EnableInternetProviders` | `EnableRealtimeMonitor` |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `Movies` | movies | true (fixed 2026-11-02) | **false** |
|
||||||
|
| `Shows` | tvshows | true (fixed 2026-11-02) | true |
|
||||||
|
| `Music` | music | **false** | true |
|
||||||
|
| `Collections` | boxsets | **false** | false |
|
||||||
|
|
||||||
|
`Music` and `Collections` are silent file indexes in the sense above: a refresh answers
|
||||||
|
`204`, stores nothing, logs nothing. Any metadata or artwork work on either one
|
||||||
|
starts by flipping the switch, never by firing a scan. `Shows` sat in that set
|
||||||
|
until 2026-11-02; the door is `POST /Library/VirtualFolders/LibraryOptions` with
|
||||||
|
the library's own `ItemId` (below), and `scripts/jellyfin-providers.sh verify` is
|
||||||
|
the one command that says which libraries are still in it.
|
||||||
|
|
||||||
|
**The per-library option door on 12.x is `POST /Library/VirtualFolders/LibraryOptions`**
|
||||||
|
— `GET` there answers `405`, which is how the route announces itself. The 10.x doors
|
||||||
|
are gone: `/Library/VirtualFolders/Update` and `/Library/VirtualFolders/Refresh`
|
||||||
|
answer `404` on this build, so every recipe naming them fails outright.
|
||||||
|
|
||||||
|
**A library record can be a phantom.** A per-library option capture taken 2026-11-01
|
||||||
|
carried seven entries — `Shows2`, `Shows3`, `Music2` among them, each with
|
||||||
|
`Locations: []` and a live-looking `ItemId` — while `GET /Library/VirtualFolders` the
|
||||||
|
same day listed four. A numeric suffix on a library name is what a deleted and
|
||||||
|
recreated library leaves behind. The live set is the entries whose `Locations` are
|
||||||
|
non-empty; count those and never the records.
|
||||||
|
|
||||||
|
**`RemoteImages` is the call that separates a provider fault from a library
|
||||||
|
fault.** `GET /Items/{id}/RemoteImages?Type=Primary` returns the provider's real
|
||||||
|
poster list even with every automatic pass switched off — plugin, key and CDN all
|
||||||
|
proven by that one response — so a refresh that stores nothing beside it is a
|
||||||
|
library-switch problem, never a provider problem. It is also the delivery door:
|
||||||
|
`POST /Items/{id}/RemoteImages/Download` saves the chosen image to the internal
|
||||||
|
store on a build where the automatic image pass will not run at all — it is how
|
||||||
|
the Movies library got its art: 2,515 of 2,564 films carry a primary now, and the
|
||||||
|
rest have none in TMDb (shorts like `Astartes`, files whose names match nothing).
|
||||||
|
|
||||||
|
**The Download door's exact wire shape:** the list response carries the address in
|
||||||
|
`Url` — there is no `ImageUrl` field — and the POST takes camelCase query params,
|
||||||
|
`?type=Primary&imageUrl=<Url>`, answering `204`; the stored tag then appears in
|
||||||
|
`ImageTags`. PascalCase params, or an `imageUrl` left empty, answer `500`, which
|
||||||
|
reads as a server fault and is a field-name fault.
|
||||||
|
|
||||||
|
**`RemoteImages` answers empty for an item with no `ProviderIds`** — no provider id,
|
||||||
|
nothing to look up. Art coverage is capped by identification, not by the image door:
|
||||||
|
the items a campaign returns nothing for are the unidentified ones, and no image
|
||||||
|
retry will change that.
|
||||||
|
|
||||||
|
**A `Download` pass reports zero only when it is pinned to one provider.** A TMDb-pinned
|
||||||
|
campaign over `Shows` classified every item in 20 s, downloaded nothing, and concluded that
|
||||||
|
identification was the only lever left. The same door queried with no `Providers=` filter
|
||||||
|
answered for 16 seasons that same day — every offer from `TheTVDB`, which the pinned form
|
||||||
|
never asks — and all 16 are dressed now. A pinned answer proves one provider is empty; the
|
||||||
|
unpinned answer is the provider census, and a TV art loop that pins is simply the wrong tool.
|
||||||
|
Post-fill the artless remainder measures 86 items (80 seasons, 6 series): 61 carry no provider
|
||||||
|
id at all — the unnamed long tail (`3a BBW - Series (2010-11)`, `History Of Africa With Zeinab
|
||||||
|
Badawi`, `(Warhammer TV)`-shaped directories), where the lever is folder names or an identify
|
||||||
|
pass — and 25 hold an id and draw no offer from any provider. No image retry moves either
|
||||||
|
group, which is the half of the old conclusion that survives.
|
||||||
|
|
||||||
|
**Series and season art is fetched; episode stills are not, deliberately.** Every
|
||||||
|
episode query is a fresh provider round-trip — nothing caches per episode — measured
|
||||||
|
at ~11 s per item beside ~2 s per series on the same host, with one hit in five in a
|
||||||
|
sample. Across 42k episodes that is days of provider traffic to dress a fifth of them,
|
||||||
|
and clients display the series and season art anyway. If episode stills are ever
|
||||||
|
ordered, the loop is the same door and its log makes the pass resumable.
|
||||||
|
|
||||||
|
**API shape on 12.x:** `/Items/{id}` needs `UserId` when authenticated by API key — without
|
||||||
|
it the answer is `400 Error processing request`, which reads exactly like an authentication
|
||||||
|
failure, and it bites hardest as a *verification* read directly after a write that succeeded.
|
||||||
|
The `/Items` list route, `?Ids=<id>` included, needs no `UserId`, so verify writes there.
|
||||||
|
`MetadataRefreshMode`
|
||||||
|
accepts `Default` (and `Realtime`) only; `Full` and `ValidationFailure` are
|
||||||
|
rejected outright, and what forces the providers to re-run is
|
||||||
|
`ReplaceAllMetadata=true`.
|
||||||
|
|
||||||
## Ports & access
|
## Ports & access
|
||||||
|
|
||||||
@@ -164,9 +281,17 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
|||||||
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
||||||
`community_supported: false`), so it needs **no PIN** — the optional
|
`community_supported: false`), so it needs **no PIN** — the optional
|
||||||
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
||||||
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
|
- **TVDB is installed but is not a metadata agent**: TMDb still identifies
|
||||||
identifies everything. Enabling TVDB as a provider would re-fetch show
|
everything. Enabling TVDB as a provider would re-fetch show metadata across
|
||||||
metadata across the library, so it is a deliberate, separate change.
|
the library, so it is a deliberate, separate change.
|
||||||
|
- **A plugin owns its own config file, under its own root, and omits null
|
||||||
|
settings** — so the element an installer wants to fill may simply not exist,
|
||||||
|
and its root is `<PluginConfiguration>`, not `<Configuration>`. TVDB's file had
|
||||||
|
no `<ApiKey>` element at all, which is why `install` refused with
|
||||||
|
`no place to write <ApiKey>` from October until the anchor fix: the secret sat
|
||||||
|
valid in `provider-keys` the whole time, and `verify`'s `MISSING key` was this
|
||||||
|
fault, not a missing key. The writer now anchors on the file's own root and
|
||||||
|
still refuses by name when a file has none.
|
||||||
- **The deployer places the keys before it restarts jellyfin**
|
- **The deployer places the keys before it restarts jellyfin**
|
||||||
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
||||||
failing when `provider-keys` is absent, so a host without it still deploys.
|
failing when `provider-keys` is absent, so a host without it still deploys.
|
||||||
@@ -212,17 +337,51 @@ receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answe
|
|||||||
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
||||||
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
||||||
later `204` is accepted and then discarded — the status code stops meaning
|
later `204` is accepted and then discarded — the status code stops meaning
|
||||||
anything. The fix is `systemctl --user restart jellyfin.service` (never
|
anything. The fix is `systemctl --user restart jellyfin.service`, one POST after
|
||||||
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
|
|
||||||
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
||||||
response code.
|
response code.
|
||||||
|
|
||||||
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
|
**The door that scans is `POST /Library/Refresh` (204), and `ScheduledTasks` is
|
||||||
is the incremental pass, and inotify does not cross a rootless podman bind, so
|
the progress read.** A task that has never completed answers
|
||||||
folders created on the host stay invisible to it. The move is
|
`LastTaskResult=None, EndTimeUtc=None`, and that pair is what separates "never
|
||||||
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
|
ran" from "finished a moment ago" — the difference between a door that is dead
|
||||||
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
|
and one that merely answered `204`.
|
||||||
record's `RefreshStatus` is the progress read.
|
|
||||||
|
**The settle read is `GET /ScheduledTasks`: the row whose `Name` is exactly
|
||||||
|
`Scan Media Library`, and its `State` (`Idle`|`Running`).** `/Library/ScanStatus`
|
||||||
|
is a 404 on 12.4, so a watcher built on the old endpoint reports nothing and a
|
||||||
|
scanning server looks an idle one. Call it settled only after a `Running`
|
||||||
|
sighting followed by two consecutive `Idle`s: sub-tasks leave gaps that a single
|
||||||
|
poll reads as finished, and a census taken in one reports the pre-scan library.
|
||||||
|
|
||||||
|
**The per-library `POST /Items/{id}/Refresh` queues nothing, and the provider
|
||||||
|
switch is not the reason.** Measured twice on 2026-11-02 against `Shows`: before
|
||||||
|
the flip and again after it, the call answered `204`, the journal wrote zero lines
|
||||||
|
over the following ten minutes, and `Scan Media Library` kept its never-run
|
||||||
|
signature; a service restart changes nothing. `POST /Library/Refresh` does run —
|
||||||
|
`204`, then `Running` inside 15 s at 33 % CPU — but it walks every library, so on
|
||||||
|
this build a library-scoped metadata pass has no working door and the global one
|
||||||
|
is the only lever.
|
||||||
|
|
||||||
|
**That endpoint takes its parameters from one source.** The documented query form
|
||||||
|
(empty body) and the JSON body form (no query) each parse; supplying both at once
|
||||||
|
answers `400` with an empty body whatever the values, and `ValidationFailure` is
|
||||||
|
rejected in the body while the query form accepts it.
|
||||||
|
|
||||||
|
**`RefreshStatus` on the `GET /Library/VirtualFolders` record is not a progress
|
||||||
|
read.** It stayed `Idle` through a scan that was verifiably running at 75 %. The
|
||||||
|
record's useful field is `ItemId`, and the live library set is still the entries
|
||||||
|
whose `Locations` are non-empty.
|
||||||
|
|
||||||
|
`inotify` does not cross a rootless podman bind, so folders created on the host
|
||||||
|
stay invisible until a pass like this runs; every pass that writes into a library
|
||||||
|
ends by firing one.
|
||||||
|
|
||||||
|
**`Movies` has no watcher at all.** `EnableRealtimeMonitor` is per library (table in
|
||||||
|
*Data layout*) and it is false for `Movies` and `Collections`. Nothing notices a
|
||||||
|
folder created under `/data/Movies`: it stays invisible until a recursive refresh
|
||||||
|
names it. So every pass that writes into a library ends by firing one, and "Jellyfin
|
||||||
|
does not see the file I just moved in" is the designed answer, not a fault to chase.
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ timestamp: 2026-10-31T00:00:00Z
|
|||||||
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
||||||
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
||||||
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
||||||
is the NARRATOR, both matched out of the files' own tags. Today one book is split
|
is the NARRATOR, both matched out of the files' own tags. [[Audiobookshelf]]
|
||||||
|
streams the same tree, read-only, and owns nothing in it. Today one book is split
|
||||||
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
||||||
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
||||||
re-files the tree so the property both servers' book scanners assume is true —
|
re-files the tree so the property both servers' book scanners assume is true —
|
||||||
@@ -36,8 +37,11 @@ python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
|
|||||||
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
||||||
# as in step 5 of the movie Procedure — from inside the container, never over
|
# as in step 5 of the movie Procedure — from inside the container, never over
|
||||||
# the host port.
|
# the host port.
|
||||||
|
# Audiobookshelf stores absolute paths, so the same refresh has to reach its
|
||||||
|
# library too ([[Audiobookshelf]]) — otherwise it keeps rows for folders that
|
||||||
|
# no longer exist.
|
||||||
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
||||||
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
--journal /mnt/entropy/.media-campaign/media-relayout-audiobooks.jsonl --apply
|
||||||
```
|
```
|
||||||
|
|
||||||
The group is the identity Plex asserts in its own database — **(narrator, book,
|
The group is the identity Plex asserts in its own database — **(narrator, book,
|
||||||
@@ -64,7 +68,7 @@ no chapter row names, and a loose file in a folder holding two books all park un
|
|||||||
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
|
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
|
||||||
`Various Artists` is not a placeholder — that is a multi-cast read.
|
`Various Artists` is not a placeholder — that is a multi-cast read.
|
||||||
|
|
||||||
The reviewed plan lives at `/mnt/entropy/media-relayout-audiobooks-plan.tsv`; it
|
The reviewed plan lives at `/mnt/entropy/.media-campaign/media-relayout-audiobooks-plan.tsv`; it
|
||||||
applies on its own journal, behind the music writer, and then refreshes the Plex
|
applies on its own journal, behind the music writer, and then refreshes the Plex
|
||||||
section. Verify it with `--db-kind plex` — Jellyfin holds no rows here, so a
|
section. Verify it with `--db-kind plex` — Jellyfin holds no rows here, so a
|
||||||
Jellyfin pass proves nothing either way.
|
Jellyfin pass proves nothing either way.
|
||||||
|
|||||||
@@ -0,0 +1,254 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Media Duplicate Prune
|
||||||
|
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
|
||||||
|
tags: [dankserver, media-server, storage, procedure]
|
||||||
|
timestamp: 2026-11-02T01:00:00Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Media Duplicate Prune
|
||||||
|
|
||||||
|
The movie library holds the same film twice in two different ways — the same file
|
||||||
|
downloaded again, or the same film grabbed a second time at another quality — and
|
||||||
|
[[Entropy Volume]] is permanently full, so both are worth finding. They need
|
||||||
|
different rules, and neither is decided by name similarity alone.
|
||||||
|
|
||||||
|
A series duplicates differently again: one show spread over several folders, so
|
||||||
|
neither the file nor the folder is the unit — see *A series is measured per
|
||||||
|
episode*.
|
||||||
|
|
||||||
|
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
|
||||||
|
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||||
|
|
||||||
|
## What counts
|
||||||
|
|
||||||
|
| Class | Test | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
|
||||||
|
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
|
||||||
|
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
|
||||||
|
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
|
||||||
|
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |
|
||||||
|
|
||||||
|
## The screen
|
||||||
|
|
||||||
|
The array cannot afford a full-content read, so identity is established in steps
|
||||||
|
and each step only narrows what the next one touches:
|
||||||
|
|
||||||
|
1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
|
||||||
|
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
|
||||||
|
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
|
||||||
|
evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
|
||||||
|
the better part of an hour; the sparse window cost 50 s, a file that merely
|
||||||
|
shares its head and tail cannot survive it, and the destination is a
|
||||||
|
recoverable hold rather than a deletion.
|
||||||
|
4. Durations come from `scripts/media-census.py probe` — the relayout's own
|
||||||
|
census. Do not hand-roll a second probe: it re-seeks the array for an answer
|
||||||
|
the repo tool already produces, and a probe is the one census step that costs
|
||||||
|
hours instead of seconds.
|
||||||
|
|
||||||
|
## Which copy survives
|
||||||
|
|
||||||
|
In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
|
||||||
|
folder (an extra is not a copy of the film, and extra names collide across films);
|
||||||
|
then the shallowest path; then the file whose name equals its `Title (year)`
|
||||||
|
folder; then the largest. For the same-work class the largest is the answer, and
|
||||||
|
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
|
||||||
|
outlive `Film.mkv` for no reason but a space sorting before a dot.
|
||||||
|
|
||||||
|
## A series is measured per episode
|
||||||
|
|
||||||
|
`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
|
||||||
|
executor's own format; it never moves anything itself.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||||
|
--hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
|
||||||
|
--meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
|
||||||
|
--ids /mnt/entropy/.media-campaign/tv-tvdb-ids.tsv \
|
||||||
|
--out /mnt/entropy/.media-campaign/tv-dupes-plan.tsv --json /tmp/tv-groups.json
|
||||||
|
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
|
||||||
|
name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
|
||||||
|
and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
|
||||||
|
match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
|
||||||
|
`BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
|
||||||
|
stripped to the bare top-level folder name — 616 folders, none carrying two
|
||||||
|
ids, written to `/mnt/entropy/.media-campaign/tv-tvdb-ids.tsv`. Two folders with disjoint ids
|
||||||
|
are two shows however alike they read — *Avatar: The Last Airbender* (1999)
|
||||||
|
beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
|
||||||
|
had also titled *Millennium* — and a pair whose identity cannot be proven is
|
||||||
|
reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
|
||||||
|
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
|
||||||
|
fill an untagged folder and are **dropped** when they contradict its own
|
||||||
|
`[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
|
||||||
|
provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
|
||||||
|
[tvdbid-73838]`; unioned, those folders are one show and the special's bigger
|
||||||
|
files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
|
||||||
|
the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
|
||||||
|
staged file rests on a borrowed id.
|
||||||
|
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
|
||||||
|
2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
|
||||||
|
Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
|
||||||
|
season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
|
||||||
|
the pass would have staged about 340 GB of unique episodes; judged per episode it
|
||||||
|
staged 54 GiB of true copies.
|
||||||
|
- **A file is droppable only when every episode it carries is held elsewhere in the
|
||||||
|
same show at >= its size** (Frank's rule, applied per episode). A byte-identical
|
||||||
|
pair ties and the tie breaks by path so exactly one copy stages; a file holding
|
||||||
|
one unique episode stays whatever else it carries.
|
||||||
|
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
|
||||||
|
poster or a whole season, so key-less names are never staged and are counted in
|
||||||
|
the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
|
||||||
|
Where a packager named files by id, Jellyfin's own table is the witness: a
|
||||||
|
`path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
|
||||||
|
and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
|
||||||
|
rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
|
||||||
|
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
|
||||||
|
`pragma table_info` before trusting the query, because an empty witness silently
|
||||||
|
falls back to name parsing.
|
||||||
|
Seven clips in the library are still named only by number, and where Jellyfin
|
||||||
|
was asked its own `Name` for them is the number too, so no database holds a
|
||||||
|
better title — that residue is naming, not a task.
|
||||||
|
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
|
||||||
|
when both halves are held elsewhere, and it wins the survivor seat for both when
|
||||||
|
it is the largest carrier: the byte law compares files, and a name both servers
|
||||||
|
parse as a range stays indexed as two episodes.
|
||||||
|
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
|
||||||
|
artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
|
||||||
|
survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
|
||||||
|
copies were bare rips, and counting a hold's non-media files is the census that
|
||||||
|
tells you which case you are in.
|
||||||
|
- **The audit gates the plan file.** Every survivor is re-statted and every
|
||||||
|
destination re-checked before `--out` is written; on failure the report prints
|
||||||
|
`AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
|
||||||
|
- **Two folders that look duplicated and share zero episode keys are complementary,
|
||||||
|
not duplicated.** That is how the two pairs the census still refuses to merge were
|
||||||
|
settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
|
||||||
|
beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
|
||||||
|
Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
|
||||||
|
episode keys: 0` in both cases, so a tag would have freed nothing and cost a
|
||||||
|
rename with library consequences. Re-running the whole census with the id map over
|
||||||
|
all 753 top-level folders (606 already tagged) staged **0 files**: the library is
|
||||||
|
duplicate-free to the limit of provable identity, and what remains is naming.
|
||||||
|
|
||||||
|
## Guards
|
||||||
|
|
||||||
|
Each of these is a file that would have been destroyed without it:
|
||||||
|
|
||||||
|
- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
|
||||||
|
the finished file while qB still counts the download incomplete; moving it
|
||||||
|
breaks the re-check and re-downloads the film. Report it, leave it. Both
|
||||||
|
leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
|
||||||
|
not a prune decision.
|
||||||
|
- **Require a year, and keep digits in the title.** Stripping digits merged
|
||||||
|
*Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
|
||||||
|
happened to share a name merged.
|
||||||
|
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
|
||||||
|
is excluded: half a multi-disc release is not a copy of the other half.
|
||||||
|
- Media containers only. Grouping every extension buries the report in subtitle
|
||||||
|
and cover pairs — and a duration guard that appears to fire on 720 groups is
|
||||||
|
counting subtitles, not protecting films.
|
||||||
|
|
||||||
|
## The hold
|
||||||
|
|
||||||
|
Nothing is deleted. Every doomed copy moves into a dated hold with the same
|
||||||
|
executor the relayout uses, so any file comes back:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/dup-prune-similar-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl # dry
|
||||||
|
scripts/media-apply.py apply --plan … --root /mnt/entropy --journal … --apply
|
||||||
|
scripts/media-apply.py verify --plan … --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl
|
||||||
|
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||||
|
scripts/media-apply.py rollback --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
|
||||||
|
every hold parent), and the planner re-checks the two things that go stale between
|
||||||
|
analysis and apply: a file that has moved since the census, and a group whose
|
||||||
|
keeper is missing — either one drops the whole group rather than half-applying it.
|
||||||
|
|
||||||
|
- **A hold on the same volume frees nothing.** The purge is a separate, approved
|
||||||
|
act and the only thing that moves the free-space number; `df` does not budge
|
||||||
|
when 1 TiB moves sideways.
|
||||||
|
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
|
||||||
|
ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
|
||||||
|
era, not this pass — read the mtimes before blaming the prune.
|
||||||
|
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
|
||||||
|
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
|
||||||
|
- The 2026-11-02 TV pass holds 220 files (54 GiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
|
||||||
|
journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
|
||||||
|
same day's TVShows detritus pass holds 109 files (552 KiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
|
||||||
|
`tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges
|
||||||
|
pending Frank's word, beside the 2026-11-01 hold above. All four journals and
|
||||||
|
plans live at the `entropy` root, not `/tmp`.
|
||||||
|
- **Re-verify each survivor by size, not by the recorded path, before purging.**
|
||||||
|
Detection artifacts and plans key on paths, and a folder renamed by an
|
||||||
|
interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look
|
||||||
|
deleted while it sits untouched under the new name. The 2026-11-01 audit hit
|
||||||
|
three such groups: every keeper was present, same bytes, different folder. A
|
||||||
|
purge decided from the plan's paths alone would have reported survivors gone.
|
||||||
|
|
||||||
|
## After the pass
|
||||||
|
|
||||||
|
Both servers still hold rows for files that are now in the hold, and neither
|
||||||
|
notices: the media bind is the [[Media Metadata View]], and array changes do not
|
||||||
|
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
|
||||||
|
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
|
||||||
|
are owned by [[Jellyfin]] and [[Plex]].
|
||||||
|
|
||||||
|
The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user
|
||||||
|
stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a
|
||||||
|
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
|
||||||
|
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
|
||||||
|
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
|
||||||
|
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
|
||||||
|
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
|
||||||
|
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.
|
||||||
|
|
||||||
|
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
|
||||||
|
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
|
||||||
|
from folders whose whole subtree held no media, so it creates no stale rows and
|
||||||
|
needs no scan. The guards that hold a folder back are the finding: external
|
||||||
|
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
|
||||||
|
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
|
||||||
|
extension the census does not recognise is a reason to stop, not to sweep.
|
||||||
|
|
||||||
|
A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
|
||||||
|
file is a manual match — a human decision about what the folder is — and detritus
|
||||||
|
rules do not overrule one.
|
||||||
|
|
||||||
|
`scripts/media-verify.py` reads differently for a prune than for a relayout:
|
||||||
|
`landed: 0` is the **correct** end state (no server may index the hold), so the
|
||||||
|
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
|
||||||
|
from the database are the pass succeeding.
|
||||||
|
|
||||||
|
One residue is normal, so do not chase it: a section pass rewrites the feature rows
|
||||||
|
first, and rows for clips in a film's `other/` or `extras/` subfolder can still name
|
||||||
|
a held path after the section's `refreshing` flag is back to `0` — and a second
|
||||||
|
section pass does not clear them either, so do not run one hoping for `0`. Beside
|
||||||
|
`REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch
|
||||||
|
the cache, the database, or the hold.
|
||||||
|
|
||||||
|
## Failure signatures
|
||||||
|
|
||||||
|
| Symptom | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
|
||||||
|
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
|
||||||
|
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
|
||||||
|
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |
|
||||||
|
|
||||||
|
## Links
|
||||||
|
|
||||||
|
- [[Media Relayout]] — the census tools, the executor, the journal format.
|
||||||
|
- [[Media Metadata View]] — why neither server sees the change without a scan.
|
||||||
|
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
|
||||||
|
- [[Entropy Volume]] — the array that is always full.
|
||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Media Metadata View
|
title: Media Metadata View
|
||||||
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||||
tags: [dankserver, media-server, jellyfin, plex, storage]
|
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||||
timestamp: 2026-10-30T00:00:00Z
|
timestamp: 2026-11-02T01:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Media Metadata View
|
# Media Metadata View
|
||||||
@@ -20,18 +20,40 @@ The mount point is not a volume and holds nothing while unmounted. It sits insid
|
|||||||
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
||||||
next to the array.
|
next to the array.
|
||||||
|
|
||||||
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
|
**Deployed on Dankserver — 2026-10-31, both servers.** `plex` binds
|
||||||
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
|
`/var/lib/media-meta/view:/mnt/entropy:ro` and `jellyfin` binds
|
||||||
`/var/lib/media-meta` does not exist, and the installed quadlets under
|
`/var/lib/media-meta/view:/data:rw`; `findmnt` reports the overlay with
|
||||||
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
|
`lowerdir=/mnt/entropy upperdir=/var/lib/media-meta/upper`, and each container
|
||||||
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
|
was proven to resolve a real media file through it — `podman exec <svc> ls -la
|
||||||
is the shape git describes, not the shape the servers run. The cutover is an
|
<its data path>` is the whole proof, because both servers keep the paths their
|
||||||
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
|
databases recorded. Two service-scoped deploys, each gated by
|
||||||
`ensure_media_view` (`install` then `verify`) before it restarts the container,
|
`media-view.sh verify` and the Playwright suite.
|
||||||
so it must not be fired through the middle of a library scan. Until that deploy
|
|
||||||
lands there is no upper layer, so no relayout can leave ghost metadata and
|
Re-verified 2026-11-01: `mount` shows the overlay on
|
||||||
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
|
`/var/lib/media-meta/view` (`lowerdir=/mnt/entropy`,
|
||||||
view really is a mountpoint.
|
`upperdir=/var/lib/media-meta/upper`), and the installed quadlets under
|
||||||
|
`~/.config/containers/systemd/` carry the view binds:
|
||||||
|
`Volume=/var/lib/media-meta/view:/data` for [[Jellyfin]],
|
||||||
|
`Volume=/var/lib/media-meta/view:/mnt/entropy:ro` for [[Plex]]. Prove it in that
|
||||||
|
order: `scripts/media-view.sh verify` (the deploy gate), then
|
||||||
|
`grep -n Volume ~/.config/containers/systemd/{jellyfin,plex}.container`.
|
||||||
|
|
||||||
|
Moving a bind means restarting the server, and **a restart discards that
|
||||||
|
server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
|
||||||
|
`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
|
||||||
|
it. Renames made on the array while the view was unmounted are invisible to both
|
||||||
|
watchers too, so the additive index has to be asked for by hand; the door is an
|
||||||
|
API key, and the row in the failure table below says what to check when it
|
||||||
|
refuses.
|
||||||
|
|
||||||
|
The cutover itself is an ordinary deploy: `deploy.sh dankserver --service
|
||||||
|
jellyfin|plex` runs `ensure_media_view` (`install` then `verify`) before it
|
||||||
|
restarts the container, so it must not be fired through the middle of a library
|
||||||
|
scan. Two consequences are now live rather than theoretical: inotify events for
|
||||||
|
array writes do not cross the merge, so any relayout or prune needs a manual
|
||||||
|
library refresh; and a move can leave ghost sidecars above, so `reap` runs after
|
||||||
|
a relayout — a pass that moves only media files with no sidecars next to them
|
||||||
|
leaves the upper layer untouched, and `reap` correctly finds nothing.
|
||||||
|
|
||||||
## Shape
|
## Shape
|
||||||
|
|
||||||
@@ -112,6 +134,19 @@ the defaults are the host values above.
|
|||||||
upper layer*, not for changes on the array underneath. After any relayout or
|
upper layer*, not for changes on the array underneath. After any relayout or
|
||||||
externally-written metadata, trigger a library refresh — see
|
externally-written metadata, trigger a library refresh — see
|
||||||
[[Media Relayout]] for the refresh procedure.
|
[[Media Relayout]] for the refresh procedure.
|
||||||
|
- **`stat` through the view can outlive the file.** Measured 2026-11-02 while a TV
|
||||||
|
dedupe pass moved folders on the raw path: the merged tree still answered `stat`
|
||||||
|
at the old coordinates — link count `0`, and `ls` of the same parent no longer
|
||||||
|
listed the name. Overlayfs keeps a dentry for a lower file that was moved
|
||||||
|
underneath it, so a script that tests existence with `[ -e ]` against the view can
|
||||||
|
believe a file its own `readdir` denies. Prove existence by listing the parent.
|
||||||
|
The media servers walk by `readdir`, so ghosts never reach the library — this bites
|
||||||
|
a tool, never a user, which is why no server symptom accompanies it.
|
||||||
|
- **Tools that move or measure media walk the raw array, never the view** — every
|
||||||
|
relayout, prune and detritus pass, and every journal it writes. The view is for
|
||||||
|
Jellyfin's writes and Plex's reads only. One deliberate exception: `tvshow.nfo`
|
||||||
|
sidecars live in the upper layer, so a tool reading show identity points at the
|
||||||
|
view (`media-tvdupes.py --meta /var/lib/media-meta/view/TVShows`).
|
||||||
|
|
||||||
## Failure signatures
|
## Failure signatures
|
||||||
|
|
||||||
@@ -122,7 +157,11 @@ the defaults are the host values above.
|
|||||||
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
||||||
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
||||||
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
||||||
|
| `install` prints `refused: upper base /var/lib/media-meta does not exist (put it on the SSD first)` | A fresh host. `install` deliberately never creates the base — a mistyped `--upper-base` must not build a metadata tree on the wrong filesystem — so `scripts/media-view.sh ensure-base` makes the documented one, which the deployer calls first. |
|
||||||
|
| A media server's scan record reads `"Status": "Aborted"` | Its container restarted (a cutover, an image bump, an OOM) and the walk was discarded; systemd's `Restart=always` restarts the *server*, never the scan. Trigger a refresh — see [[Media Relayout]]. |
|
||||||
|
| The Jellyfin API answers 401 to every token header | Almost always the header **form**, not the key: 12.4 authenticates only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for every key, valid or not — and one collapsed backslash in the curl config produces the same 401. `[[Jellyfin]]` owns the block that works. Before concluding anything, prove the key against the server's own table: `select count(*) from ApiKeys where AccessToken = ?` on `jellyfin.db` opened `mode=ro&immutable=1`. |
|
||||||
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
||||||
|
| Stale dentry through the view: `stat` succeeds with link count `0` while the parent's `ls` omits the name | The lower file was moved on the raw path after the view cached its dentry. Nothing is lost; existence through the view is proved by the parent listing. |
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.
|
|||||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
|
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
|
||||||
# executor, journal and rollback as [[Media Relayout]] shows, on its own journal.
|
# executor, journal and rollback as [[Media Relayout]] shows, on its own journal.
|
||||||
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
|
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
|
||||||
--journal /mnt/entropy/media-relayout-music.jsonl --apply
|
--journal /mnt/entropy/.media-campaign/media-relayout-music.jsonl --apply
|
||||||
```
|
```
|
||||||
|
|
||||||
The group is what the tags assert, not where the files sit: **(album-artist tag,
|
The group is what the tags assert, not where the files sit: **(album-artist tag,
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Media Relayout
|
title: Media Relayout
|
||||||
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||||
timestamp: 2026-10-30T00:00:00Z
|
timestamp: 2026-11-02T11:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Media Relayout
|
# Media Relayout
|
||||||
@@ -71,12 +71,23 @@ that would be filed wrong, and the plan is not run until it is understood.
|
|||||||
```bash
|
```bash
|
||||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
|
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
|
||||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
|
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
|
||||||
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
|
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/.media-campaign/media-relayout.jsonl
|
||||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
||||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||||
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
scripts/media-apply.py rollback --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Every campaign artifact — plan, journal, exceptions file, run log — lives in
|
||||||
|
`/mnt/entropy/.media-campaign/`.** The array root carries libraries and nothing
|
||||||
|
else: a root strewn with `*.jsonl` and `*-plan.tsv` gets browsed as content, and
|
||||||
|
the dot-directory is ignored by both media servers. Relocated 2026-11-02, 48 files,
|
||||||
|
undo with `media-apply.py rollback --journal /mnt/entropy/.media-campaign/media-campaign-relocate.jsonl --root /mnt/entropy --apply`.
|
||||||
|
The holds and trash (`.AGENTTRASH`, `.HOLD-<date>-*`) stay at the root on purpose:
|
||||||
|
the purge machinery is given those paths as `--hold`, and a hold that moves out
|
||||||
|
from under the tool that made it is a hold nobody can find. Name `--journal`
|
||||||
|
explicitly on every run — the tool's default is `media-relayout.jsonl` beside
|
||||||
|
`--root`, so a bare run recreates exactly the root-level log this page forbids.
|
||||||
|
|
||||||
An apply also leaves the release folders it emptied standing where they were —
|
An apply also leaves the release folders it emptied standing where they were —
|
||||||
hundreds of them at the library root — which is what `sweep` removes, dry-run
|
hundreds of them at the library root — which is what `sweep` removes, dry-run
|
||||||
first, from the same journal.
|
first, from the same journal.
|
||||||
@@ -110,7 +121,7 @@ a read beside a running scan can fail as "database disk image is malformed".
|
|||||||
`immutable` stays only the fallback for a database whose `-shm` cannot be read.
|
`immutable` stays only the fallback for a database whose `-shm` cannot be read.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
|
scripts/media-verify.py --journal /mnt/entropy/.media-campaign/media-relayout-music.jsonl \
|
||||||
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
|
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -138,6 +149,8 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
|||||||
no `Accept` and client-identifier header (see the scan block above for the form
|
no `Accept` and client-identifier header (see the scan block above for the form
|
||||||
that works).
|
that works).
|
||||||
|
|
||||||
|
A duplicate prune is the same census and the same proof, with its own keep rules — films by title and duration, a series per episode: [[Media Duplicate Prune]].
|
||||||
|
|
||||||
## TV in the movie library
|
## TV in the movie library
|
||||||
|
|
||||||
A box set that arrives in the movie share stays there until someone moves it:
|
A box set that arrives in the movie share stays there until someone moves it:
|
||||||
@@ -148,11 +161,11 @@ by series, and writes a plan in the executor's own format:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
|
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
|
||||||
--out /mnt/entropy/media-tv-plan.tsv
|
--out /mnt/entropy/.media-campaign/media-tv-plan.tsv
|
||||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
|
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-tv-plan.tsv --root /mnt/entropy
|
||||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
|
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-tv-plan.tsv --root /mnt/entropy \
|
||||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
--journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl --apply
|
||||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
|
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl \
|
||||||
--root /mnt/entropy --apply
|
--root /mnt/entropy --apply
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -184,162 +197,10 @@ show. A name whose marker is `Part 2` is a film and stays put.
|
|||||||
|
|
||||||
## The TV library's own relayout
|
## The TV library's own relayout
|
||||||
|
|
||||||
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
`/mnt/entropy/TVShows` is its own arm: a different authority (tvdb), its own
|
||||||
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
refusals, and its own test for whether a folder really holds two shows —
|
||||||
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
[[Media TV Relayout]]. It shares this page's executor, the one-journal-per-arm
|
||||||
|
law, the sweep and the rollback.
|
||||||
<Series> (<year>) [tvdbid-NNNN]/Season NN/
|
|
||||||
<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
|
||||||
|
|
||||||
It never guesses: every series name, episode number and title it writes traces to
|
|
||||||
an authority, and what has none is parked for a human to read.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
|
||||||
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
|
||||||
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
|
||||||
--out /tmp/tv_titles.tsv
|
|
||||||
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
|
||||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
|
||||||
--out /tmp/tv_enum.tsv
|
|
||||||
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
|
||||||
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
|
||||||
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
|
||||||
# the last one — a duration belongs to the file, not to its name: re-key the
|
|
||||||
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
|
||||||
# seeking the array again. It is also the only column the film arm reads, and
|
|
||||||
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
|
||||||
# all of them covered by a re-key that costs seconds, not an hour).
|
|
||||||
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
|
||||||
# which travels as the `api_key` query parameter. Only a v4 request token is
|
|
||||||
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
|
||||||
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
|
||||||
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
|
||||||
# cache file, so the plan is reproducible offline afterwards: run it once to
|
|
||||||
# fill the cache, and quote the second run.
|
|
||||||
# Paths are only strings to a dry run, so this step can run on the workstation
|
|
||||||
# against scp'd TSVs — which is where the two provider secrets live.
|
|
||||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
|
||||||
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
|
||||||
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
|
|
||||||
--park /mnt/entropy/media-park/TVShows \
|
|
||||||
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
|
|
||||||
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
|
|
||||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
|
||||||
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
|
||||||
# applied pass is its unwind record, so a second pass that reuses the name
|
|
||||||
# destroys the only map back.
|
|
||||||
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
|
|
||||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
|
|
||||||
```
|
|
||||||
|
|
||||||
Evidence, strongest first — the order the planner believes things:
|
|
||||||
|
|
||||||
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`); a digit
|
|
||||||
run longer than three at that position is NOT a number — `S03E2019` and
|
|
||||||
`S04E47370355` are one box set's airdate fragment and pack index — so such a file
|
|
||||||
is renumbered by tvdb's title or parks, and is never truncated down to fit;
|
|
||||||
- the **`Season NN` folder it sits in and the number its name leads with inside
|
|
||||||
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
|
||||||
- the **provider row**: which episode the file IS, and the numbering of a file
|
|
||||||
whose own name says nothing;
|
|
||||||
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
|
||||||
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
|
||||||
episode title, because a name the server read off a folder is not a show name.
|
|
||||||
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
|
|
||||||
Plex's scanner both read, so a show whose title is ambiguous resolves by number
|
|
||||||
instead of by a human guessing which folder is which. The id rides the folder
|
|
||||||
only — a file name gets 180 characters and the episode title needs the room.
|
|
||||||
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
|
||||||
series' canonical extra folder;
|
|
||||||
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
|
||||||
when the movie provider's title is a run of words the FILE itself states and its
|
|
||||||
year agrees with the file's: the query is only what `title_of` salvaged from a
|
|
||||||
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
|
||||||
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
|
||||||
parks.
|
|
||||||
|
|
||||||
Four refusals carry the design: what the planner will NOT do is the point of it.
|
|
||||||
|
|
||||||
- **Folders resolve a series; they never invent one.** A folder's words may
|
|
||||||
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
|
||||||
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
|
||||||
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
|
||||||
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
|
||||||
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
|
||||||
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
|
||||||
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
|
||||||
the file's own leading number, win.
|
|
||||||
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
|
|
||||||
specials by airdate, so a row can arrive saying `season 2002`; that is not a
|
|
||||||
season and it never becomes a folder. The one authority on such a row is tvdb's
|
|
||||||
episode list read backwards: the title the row carries names the (season,
|
|
||||||
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
|
|
||||||
the file parks with the reason it always carried. `Season 00` IS a legal
|
|
||||||
season — specials live there — so a row that really says 0 files there instead
|
|
||||||
of parking.
|
|
||||||
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
|
|
||||||
numbered by its release group books `S05E01` in a season tvdb does not hold, or
|
|
||||||
books more episodes there than tvdb lists: the slot its number points at is a
|
|
||||||
different episode, and dressing it in that episode's title is a confident lie.
|
|
||||||
The file still renames to the standard shape; the title does not ride, and the
|
|
||||||
series is listed under *seasons whose numbering tvdb contradicts* in the
|
|
||||||
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
|
|
||||||
partial pack is the normal shape. Neither reading holds for a list tvdb
|
|
||||||
cut off at its own page limit: the default episode query answers at most
|
|
||||||
500 rows and states no total, and One Piece, Crayon Shin-chan and
|
|
||||||
Detective Conan each answer exactly 500 while holding over a thousand —
|
|
||||||
a page bounds nothing above itself, so it contradicts nothing.
|
|
||||||
|
|
||||||
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
|
||||||
after the folder it scanned, so a library filed by release folders ends up with
|
|
||||||
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
|
||||||
copying those names verbatim launders folder junk into the contract. The planner
|
|
||||||
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
|
||||||
identity is also what turns a box set split across five release folders into ONE
|
|
||||||
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
|
||||||
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
|
||||||
identity* in the exceptions — that list, not a rename, is the review.
|
|
||||||
|
|
||||||
**The review pile lives outside the library.** Files parked inside
|
|
||||||
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
|
||||||
show's rows then looked like evidence. So a row whose series is the hidden
|
|
||||||
top-level folder the file itself sits in is dropped — and the folder is matched
|
|
||||||
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
|
||||||
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
|
||||||
visible series folder that real files get renamed into. `--park` outside the root
|
|
||||||
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
|
||||||
top-level folder a file's own words may MATCH a registered show, never invent one.
|
|
||||||
|
|
||||||
**The operator's id map is the door for a name tvdb cannot be searched by.**
|
|
||||||
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
|
|
||||||
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
|
|
||||||
tvdb titles a record in the show's own language and the English name rides in
|
|
||||||
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
|
|
||||||
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
|
|
||||||
2023 live action, not to the 1999 record (81797) the pack here is. The map
|
|
||||||
outranks a provider id — a box set's provider id is the server's own guess at a
|
|
||||||
folder name — and it merges the release folders that are one show. Everything it
|
|
||||||
touched is written to the exceptions: every folder it named, every row that
|
|
||||||
matched no folder, every disagreement with a provider id, every id tvdb never
|
|
||||||
answered. A line whose id is not a number stops the plan, because a typo renames
|
|
||||||
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
|
|
||||||
it; a row for a folder the search would resolve hides a live lookup behind a
|
|
||||||
number nobody re-checks.
|
|
||||||
|
|
||||||
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
|
|
||||||
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
|
|
||||||
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
|
|
||||||
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
|
|
||||||
release junk is stripped from the folder's words), `NEAR` (tvdb answered
|
|
||||||
something else — the best hit is printed for a human and nothing applies it),
|
|
||||||
`NONE`. Acceptance never moved across the passes that built this: a hit counts
|
|
||||||
only when tvdb's name equals the query with every non-alphanumeric dropped from
|
|
||||||
both sides, and the record's year still has to fall inside any year span the
|
|
||||||
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
|
|
||||||
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
|
|
||||||
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
|
|
||||||
and the sheet is never fed to the planner.
|
|
||||||
|
|
||||||
## The music and audiobook arms
|
## The music and audiobook arms
|
||||||
|
|
||||||
@@ -401,9 +262,12 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
|||||||
- **The journal lives BESIDE the library root, never inside it** (default:
|
- **The journal lives BESIDE the library root, never inside it** (default:
|
||||||
`media-relayout.jsonl` next to the root): a log inside the library is a file
|
`media-relayout.jsonl` next to the root): a log inside the library is a file
|
||||||
Jellyfin indexes.
|
Jellyfin indexes.
|
||||||
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
|
- **Refusals are all-or-nothing.** An occupied target, a vanished source, a
|
||||||
duplicate destination stops the run *before the first rename*, so the operator
|
duplicate destination, or a plan row whose verb is not `mkdir`, `keep` or
|
||||||
fixes one thing and re-runs instead of untangling a half-applied plan.
|
`move`, stops the run *before the first rename*, so the operator fixes one
|
||||||
|
thing and re-runs instead of untangling a half-applied plan. An unread verb is
|
||||||
|
refused by line number: silently dropping one turned a pass meant to rename
|
||||||
|
two series folders into `applied: 0 moved` with exit 0 (2026-11-02).
|
||||||
`--force` relaxes only the opinion guards; it never permits overwriting.
|
`--force` relaxes only the opinion guards; it never permits overwriting.
|
||||||
- **A sweep deletes only what the journal proves the run emptied.** A folder is
|
- **A sweep deletes only what the journal proves the run emptied.** A folder is
|
||||||
eligible when a journalled file moved out of it or out of something under it,
|
eligible when a journalled file moved out of it or out of something under it,
|
||||||
@@ -430,6 +294,14 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
|||||||
`--root` set to the share that holds BOTH libraries: the executor refuses any
|
`--root` set to the share that holds BOTH libraries: the executor refuses any
|
||||||
path outside `--root`, and a journal shared with the movie relayout would
|
path outside `--root`, and a journal shared with the movie relayout would
|
||||||
undo both runs when only one of them is wanted.
|
undo both runs when only one of them is wanted.
|
||||||
|
- **A hold folder outside `--root` forces `--force` on BOTH `apply` and
|
||||||
|
`sweep`.** `sweep` refuses the entire journal when any journalled path sits
|
||||||
|
outside the root (`this journal belongs to another library`) — the same guard
|
||||||
|
the plan hits, and it exists to stop one pass unwinding another's files.
|
||||||
|
`--force` relaxes it and stays safe there: `sweep_targets` marks only
|
||||||
|
ancestors under `--root`, so a hold folder beyond the root can never be
|
||||||
|
selected; it only blocks everything that is. The cleaner shape is the law
|
||||||
|
above — put `--root` on the share that holds both.
|
||||||
|
|
||||||
- **Sidecar metadata does not follow a rename, because it lives on another
|
- **Sidecar metadata does not follow a rename, because it lives on another
|
||||||
filesystem.** The media servers write `.nfo` and artwork through the
|
filesystem.** The media servers write `.nfo` and artwork through the
|
||||||
@@ -438,6 +310,9 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
|||||||
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
|
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
|
||||||
that both servers browse and report as content. Whether this host mounts the
|
that both servers browse and report as content. Whether this host mounts the
|
||||||
view at all is stated on that page; `media-apply.py` stays silent when it does not.
|
view at all is stated on that page; `media-apply.py` stays silent when it does not.
|
||||||
|
A rename of folders the server never identified strands nothing: Jellyfin
|
||||||
|
writes a sidecar only for a series it recognised, so `reap` answering
|
||||||
|
`no ghosts` after such a pass is the expected result, not a broken tool.
|
||||||
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
|
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
|
||||||
whiteout in the upper layer, which hides the array's copy without removing it;
|
whiteout in the upper layer, which hides the array's copy without removing it;
|
||||||
the file returns when the view is unmounted. Deletions in this pipeline happen
|
the file returns when the view is unmounted. Deletions in this pipeline happen
|
||||||
|
|||||||
@@ -0,0 +1,241 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Media TV Relayout
|
||||||
|
description: "Re-file TVShows from scene packs into the shape Jellyfin reads: authority order, refusals, the two-series split test."
|
||||||
|
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||||
|
timestamp: 2026-11-02T11:00:00Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Media TV Relayout
|
||||||
|
|
||||||
|
The `/mnt/entropy/TVShows` share arrived as scene packs — release folders whose
|
||||||
|
names are the release, not the show. This arm re-files it into the shape
|
||||||
|
[[Jellyfin]] reads, with tvdb as the name authority. The executor, the journal,
|
||||||
|
the sweep and the rollback belong to [[Media Relayout]]; this page owns the TV
|
||||||
|
authority order, the refusals, and the test that decides a split.
|
||||||
|
|
||||||
|
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
||||||
|
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
||||||
|
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
||||||
|
|
||||||
|
<Series> (<year>) [tvdbid-NNNN]/Season NN/
|
||||||
|
<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
||||||
|
|
||||||
|
It never guesses: every series name, episode number and title it writes traces to
|
||||||
|
an authority, and what has none is parked for a human to read.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
||||||
|
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
||||||
|
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
||||||
|
--out /tmp/tv_titles.tsv
|
||||||
|
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
||||||
|
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
||||||
|
--out /tmp/tv_enum.tsv
|
||||||
|
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
||||||
|
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
||||||
|
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
||||||
|
# the last one — a duration belongs to the file, not to its name: re-key the
|
||||||
|
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
||||||
|
# seeking the array again. It is also the only column the film arm reads, and
|
||||||
|
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
||||||
|
# all of them covered by a re-key that costs seconds, not an hour).
|
||||||
|
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
||||||
|
# which travels as the `api_key` query parameter. Only a v4 request token is
|
||||||
|
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
||||||
|
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
||||||
|
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
||||||
|
# cache file, so the plan is reproducible offline afterwards: run it once to
|
||||||
|
# fill the cache, and quote the second run.
|
||||||
|
# Paths are only strings to a dry run, so this step can run on the workstation
|
||||||
|
# against scp'd TSVs — which is where the two provider secrets live.
|
||||||
|
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
||||||
|
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
||||||
|
--tmdb-cache /mnt/entropy/.media-campaign/media-tv-names.json --tmdb --tvdb \
|
||||||
|
--park /mnt/entropy/NeedsProcessing/TVShows \
|
||||||
|
--out /mnt/entropy/.media-campaign/media-relayout-shows2-plan.tsv \
|
||||||
|
--exceptions /mnt/entropy/.media-campaign/media-relayout-shows2-exceptions.txt
|
||||||
|
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
||||||
|
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
||||||
|
# applied pass is its unwind record, so a second pass that reuses the name
|
||||||
|
# destroys the only map back.
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-relayout-shows2-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-shows2.jsonl --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Evidence, strongest first — the order the planner believes things:
|
||||||
|
|
||||||
|
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`); a digit
|
||||||
|
run longer than three at that position is NOT a number — `S03E2019` and
|
||||||
|
`S04E47370355` are one box set's airdate fragment and pack index — so such a file
|
||||||
|
is renumbered by tvdb's title or parks, and is never truncated down to fit;
|
||||||
|
- the **`Season NN` folder it sits in and the number its name leads with inside
|
||||||
|
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
||||||
|
- the **provider row**: which episode the file IS, and the numbering of a file
|
||||||
|
whose own name says nothing;
|
||||||
|
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
||||||
|
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
||||||
|
episode title, because a name the server read off a folder is not a show name.
|
||||||
|
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
|
||||||
|
Plex's scanner both read, so a show whose title is ambiguous resolves by number
|
||||||
|
instead of by a human guessing which folder is which. The id rides the folder
|
||||||
|
only — a file name gets 180 characters and the episode title needs the room.
|
||||||
|
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
||||||
|
series' canonical extra folder;
|
||||||
|
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
||||||
|
when the movie provider's title is a run of words the FILE itself states and its
|
||||||
|
year agrees with the file's: the query is only what `title_of` salvaged from a
|
||||||
|
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
||||||
|
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
||||||
|
parks.
|
||||||
|
|
||||||
|
Four refusals carry the design: what the planner will NOT do is the point of it.
|
||||||
|
|
||||||
|
- **Folders resolve a series; they never invent one.** A folder's words may
|
||||||
|
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
||||||
|
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
||||||
|
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
||||||
|
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
||||||
|
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
||||||
|
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
||||||
|
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
||||||
|
the file's own leading number, win.
|
||||||
|
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
|
||||||
|
specials by airdate, so a row can arrive saying `season 2002`; that is not a
|
||||||
|
season and it never becomes a folder. The one authority on such a row is tvdb's
|
||||||
|
episode list read backwards: the title the row carries names the (season,
|
||||||
|
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
|
||||||
|
the file parks with the reason it always carried. `Season 00` IS a legal
|
||||||
|
season — specials live there — so a row that really says 0 files there instead
|
||||||
|
of parking.
|
||||||
|
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
|
||||||
|
numbered by its release group books `S05E01` in a season tvdb does not hold, or
|
||||||
|
books more episodes there than tvdb lists: the slot its number points at is a
|
||||||
|
different episode, and dressing it in that episode's title is a confident lie.
|
||||||
|
The file still renames to the standard shape; the title does not ride, and the
|
||||||
|
series is listed under *seasons whose numbering tvdb contradicts* in the
|
||||||
|
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
|
||||||
|
partial pack is the normal shape. Neither reading holds for a list tvdb
|
||||||
|
cut off at its own page limit: the default episode query answers at most
|
||||||
|
500 rows and states no total, and One Piece, Crayon Shin-chan and
|
||||||
|
Detective Conan each answer exactly 500 while holding over a thousand —
|
||||||
|
a page bounds nothing above itself, so it contradicts nothing.
|
||||||
|
|
||||||
|
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
||||||
|
after the folder it scanned, so a library filed by release folders ends up with
|
||||||
|
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
||||||
|
copying those names verbatim launders folder junk into the contract. The planner
|
||||||
|
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
||||||
|
identity is also what turns a box set split across five release folders into ONE
|
||||||
|
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
||||||
|
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
||||||
|
identity* in the exceptions — that list, not a rename, is the review.
|
||||||
|
|
||||||
|
**The review pile lives outside the library.** Files parked inside
|
||||||
|
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
||||||
|
show's rows then looked like evidence. So a row whose series is the hidden
|
||||||
|
top-level folder the file itself sits in is dropped — and the folder is matched
|
||||||
|
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
||||||
|
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
||||||
|
visible series folder that real files get renamed into. `--park` outside the root
|
||||||
|
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
||||||
|
top-level folder a file's own words may MATCH a registered show, never invent one.
|
||||||
|
|
||||||
|
**The operator's id map is the door for a name tvdb cannot be searched by.**
|
||||||
|
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
|
||||||
|
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
|
||||||
|
tvdb titles a record in the show's own language and the English name rides in
|
||||||
|
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
|
||||||
|
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
|
||||||
|
2023 live action, not to the 1999 record (81797) the pack here is. The map
|
||||||
|
outranks a provider id — a box set's provider id is the server's own guess at a
|
||||||
|
folder name — and it merges the release folders that are one show. Everything it
|
||||||
|
touched is written to the exceptions: every folder it named, every row that
|
||||||
|
matched no folder, every disagreement with a provider id, every id tvdb never
|
||||||
|
answered. A line whose id is not a number stops the plan, because a typo renames
|
||||||
|
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
|
||||||
|
it; a row for a folder the search would resolve hides a live lookup behind a
|
||||||
|
number nobody re-checks.
|
||||||
|
|
||||||
|
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
|
||||||
|
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
|
||||||
|
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
|
||||||
|
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
|
||||||
|
release junk is stripped from the folder's words), `NEAR` (tvdb answered
|
||||||
|
something else — the best hit is printed for a human and nothing applies it),
|
||||||
|
`NONE`. Acceptance never moved across the passes that built this: a hit counts
|
||||||
|
only when tvdb's name equals the query with every non-alphanumeric dropped from
|
||||||
|
both sides, and the record's year still has to fall inside any year span the
|
||||||
|
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
|
||||||
|
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
|
||||||
|
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
|
||||||
|
and the sheet is never fed to the planner.
|
||||||
|
|
||||||
|
**A parked folder merges into the row that owns the show.** The folder that never
|
||||||
|
got an id usually holds episodes the identified folder for that show lacks, and
|
||||||
|
the server shows the audience neither: the parked copy is invisible and the owner
|
||||||
|
reads as a short series. One `media-apply.py` plan per show, built from the
|
||||||
|
server's own rows:
|
||||||
|
|
||||||
|
- the **owner is the folder whose `Series` row carries the tvdb id**; the parked
|
||||||
|
side is the folder with the same words and no id (or the same id under a second
|
||||||
|
row — see the law below);
|
||||||
|
- the **comparison is episode identity, not filename**: which (season, episode)
|
||||||
|
the provider row gives each file, read straight out of the server. Release
|
||||||
|
folders renumber — one park carried `S01E01-11` of a show tvdb books as seasons
|
||||||
|
2 and 3, and every episode runs the same eleven minutes, so neither the marker
|
||||||
|
nor the duration says what one query into the rows says;
|
||||||
|
- **a season the owner has no file in moves whole**: `mkdir` the `Season NN`,
|
||||||
|
then one row per file with its name unchanged — the folder shape is the
|
||||||
|
contract, the file name is not;
|
||||||
|
- **a subtitle pack is re-keyed, never just moved**: a sidecar attaches only when
|
||||||
|
its stem equals the video's, so each `.srt` has to land on the OWNER's stem with
|
||||||
|
the language suffix kept (`… - The Cat and the Claw (1).en.srt`). Moved under
|
||||||
|
its release-group name it stays a folder of 88 subtitles no player ever offers;
|
||||||
|
- **the same episode on both sides is the operator's call**, never a plan row:
|
||||||
|
which copy survives is a quality judgment — the parked copy is usually the
|
||||||
|
bigger one, and a re-cut is not a worse one — and the parked file often carries
|
||||||
|
no title to align by. Those pairs are listed and left alone.
|
||||||
|
|
||||||
|
The two sides speak different path dialects, since the server records the
|
||||||
|
container coordinate ([[Jellyfin]]) and the journal speaks the array; a join that
|
||||||
|
forgets the translation returns zero rows and looks like an empty library.
|
||||||
|
|
||||||
|
## Laws of this arm
|
||||||
|
|
||||||
|
- **A folder that names two series is not evidence of two series — the files
|
||||||
|
answer.** `ffprobe -show_entries format_tag=title` names the series each file
|
||||||
|
belongs to. Measured 2026-11-02 on `COW and CHICKEN … Complete Animated TV
|
||||||
|
Series`, a bundle whose name pairs *Cow and Chicken* with *I Am Weasel*: 103
|
||||||
|
of 104 files carry a `Cow and Chicken - SxxEyy - <title>` tag and none carries
|
||||||
|
an *I Am Weasel* one. The 7-minute duration class is a half-episode segment,
|
||||||
|
not a second show, and a file holding both segments needs a remux, which this
|
||||||
|
executor will never do. Split only when the tags (or a provider episode list)
|
||||||
|
name both series per file; everything else is a rename.
|
||||||
|
- **When the numbering is right and only the name is wrong, rename the folder
|
||||||
|
and nothing else.** Tags whose `Sxx` matches the `Season NN` folder holding
|
||||||
|
them, per-season counts equal, and no duplicate `SxxEyy` key anywhere prove
|
||||||
|
the layout is already correct. Renumbering files under that evidence buys
|
||||||
|
nothing and can silently mislabel an episode.
|
||||||
|
- **One tvdb id can carry two `Series` rows, so the id-less folder list proves
|
||||||
|
nothing about a merge.** A parked folder whose words parse to a registered show
|
||||||
|
gets identified on its own and keeps its own folder: the count of id-less
|
||||||
|
folders drops, and the library now offers the same show twice. Prove a merge by
|
||||||
|
the episode rows standing under the owner folder, never by that list.
|
||||||
|
- **Move library files only while `Scan Media Library` is Idle.** A rename that
|
||||||
|
lands under a running scan is a row the server read a moment ago pointing at a
|
||||||
|
path that no longer exists. `ScheduledTasks` is the read ([[Jellyfin]]); wait
|
||||||
|
for a `Running` sighting and then two consecutive `Idle` polls, because a gap
|
||||||
|
between sub-tasks reads as finished.
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
[[Media Relayout]] — the executor, the journal, the sweep and the rollback this
|
||||||
|
arm shares. [[Jellyfin]] — the reader of the resulting shape. [[Plex]] — the
|
||||||
|
second reader. [[Media Metadata View]] — where a rename's sidecars strand, or
|
||||||
|
do not.
|
||||||
|
|
||||||
|
## Citations
|
||||||
|
|
||||||
|
- Jellyfin Shows documentation — series, season and episode folder/file naming:
|
||||||
|
<https://jellyfin.org/docs/general/server/media/shows>
|
||||||
@@ -27,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||||
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — live since the 2026-10-31 cutover, so the PMS database's recorded paths keep resolving with zero re-scan |
|
||||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||||
@@ -40,7 +40,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
The migration moved zero bytes: the container bind-mounts the native data
|
The migration moved zero bytes: the container bind-mounts the native data
|
||||||
root where it already lived, and the media archive re-mounts at
|
root where it already lived, and the media archive re-mounts at
|
||||||
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||||
(the array today; the overlay view in git has never been deployed), so every
|
(the array until the 2026-10-31 view cutover, the overlay since), so every
|
||||||
absolute library path in the
|
absolute library path in the
|
||||||
PMS database keeps resolving. The one metadata change: rootless podman
|
PMS database keeps resolving. The one metadata change: rootless podman
|
||||||
cannot map the real host uid 997, so the fleet model applies — container
|
cannot map the real host uid 997, so the fleet model applies — container
|
||||||
@@ -86,7 +86,22 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
|||||||
4. **Cold container boot is slow.** PMS pre-binds 32400 and empty-replies
|
4. **Cold container boot is slow.** PMS pre-binds 32400 and empty-replies
|
||||||
while the HTTP layer finishes loading; the first boot of a years-old
|
while the HTTP layer finishes loading; the first boot of a years-old
|
||||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||||
600 s before failing; later boots are much faster (warm page cache).
|
600 s before failing; later boots are much faster (warm page cache). A
|
||||||
|
`systemctl --user start` issued after an `unmask` is a job in the manager,
|
||||||
|
not in the shell that issued it: it lands minutes later and survives that
|
||||||
|
client dying, so a wrapper script that looks hung is usually only waiting —
|
||||||
|
read `is-active` and `podman ps` before declaring a fault, and give every
|
||||||
|
step of such a script its own `timeout`.
|
||||||
|
5. **`32400` answers TLS only to remote clients.** PMS requires secure
|
||||||
|
connections, so `curl http://192.168.0.235:32400/identity` from any other
|
||||||
|
machine closes without a single byte (curl exit 52) — that reads as a dead
|
||||||
|
server beside a healthy one, and it is how a post-cutover health check gets
|
||||||
|
misread. Probe with `curl -k https://192.168.0.235:32400/identity`. Two
|
||||||
|
consequences: `plex.franksdank.cloud` works **only** because its `transport
|
||||||
|
http` block carries `tls_insecure_skip_verify` (that option makes Caddy speak
|
||||||
|
TLS to PMS's self-signed cert — delete it and the vhost 502s), and the scan
|
||||||
|
block below rides `http://127.0.0.1` **inside** the container, where insecure
|
||||||
|
connections are allowed.
|
||||||
|
|
||||||
## Triggering a scan
|
## Triggering a scan
|
||||||
|
|
||||||
@@ -121,10 +136,13 @@ shred -u /tmp/px.tok
|
|||||||
```
|
```
|
||||||
|
|
||||||
Every section answers `204` — or `200` with a Directory body when
|
Every section answers `204` — or `200` with a Directory body when
|
||||||
`Accept: application/json` is sent. Inside the quoted `printf` format a
|
`Accept: application/json` is sent. A refresh fired while PMS is still warming
|
||||||
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
|
answers `000` instead (the empty reply of gotcha 4, arriving mid-request): retry
|
||||||
one glued line and one stray backslash is a config file curl rejects, and the
|
until it answers and never restart the service over it. Inside the quoted
|
||||||
failure prints as `option -K: is unknown`, never as a bad config line.
|
`printf` format a backslash-newline is LITERAL, not a continuation, so `mk`
|
||||||
|
writes two `printf`s: one glued line and one stray backslash is a config file
|
||||||
|
curl rejects, and the failure prints as `option -K: is unknown`, never as a bad
|
||||||
|
config line.
|
||||||
|
|
||||||
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
||||||
replays the run's journal against the database — the only read that
|
replays the run's journal against the database — the only read that
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ description: "Outpost, proxy-provider, OAuth2, LDAP and flow configuration for t
|
|||||||
tags:
|
tags:
|
||||||
- infrastructure
|
- infrastructure
|
||||||
- authentik
|
- authentik
|
||||||
timestamp: 2026-10-19
|
timestamp: 2026-11-03
|
||||||
---
|
---
|
||||||
|
|
||||||
# Authentik Outpost Reference
|
# Authentik Outpost Reference
|
||||||
@@ -56,7 +56,7 @@ correctly — the outpost matches on hostname.
|
|||||||
|
|
||||||
## Provisioning rules for proxy providers
|
## Provisioning rules for proxy providers
|
||||||
|
|
||||||
Two provisioning rules, each verified live by an outage:
|
Three provisioning rules, each verified live by an outage or a dead login:
|
||||||
|
|
||||||
1. **All three flows are required.** A ProxyProvider created via raw API/ORM
|
1. **All three flows are required.** A ProxyProvider created via raw API/ORM
|
||||||
with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
|
with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
|
||||||
@@ -80,6 +80,16 @@ Two provisioning rules, each verified live by an outage:
|
|||||||
the app. No error is logged anywhere — the SSO "works" and the app never
|
the app. No error is logged anywhere — the SSO "works" and the app never
|
||||||
appears. Create the Application with `provider_id` set (the UI does this
|
appears. Create the Application with `provider_id` set (the UI does this
|
||||||
automatically; raw API/ORM drives must do it explicitly).
|
automatically; raw API/ORM drives must do it explicitly).
|
||||||
|
3. **A raw ORM create skips the OAuth defaults.** The REST serializer calls
|
||||||
|
`ProxyProvider.set_oauth_defaults()`; `ProxyProvider.objects.create()` does not. The row
|
||||||
|
then looks fully configured — external_host, internal_host, application bound, outpost
|
||||||
|
claimed — while `redirect_uris` is empty (those URIs are *derived* from `external_host` by
|
||||||
|
`_get_callback_url`) and `grant_types` is `[]`. Both kill the guard before any credential,
|
||||||
|
in this order: authentik renders its own **"Redirect URI Error"** page, and once the URIs
|
||||||
|
are filled by hand the next wall is a bodiless **HTTP 400** out of `/application/o/authorize/`
|
||||||
|
(both measured on the Audiobookshelf guard provider, 2026-11-03). Census and repair in one idempotent pass, all
|
||||||
|
providers at once: `scripts/authentik-proxy-provider-defaults.sh`; `--check` prints the
|
||||||
|
census and exits 1 on drift, and a provider that already matches is left alone.
|
||||||
|
|
||||||
## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
|
## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ description: "Caddy is the single reverse proxy for both hosts."
|
|||||||
tags:
|
tags:
|
||||||
- infrastructure
|
- infrastructure
|
||||||
- caddy
|
- caddy
|
||||||
timestamp: 2026-08-29
|
timestamp: 2026-11-03
|
||||||
---
|
---
|
||||||
|
|
||||||
# Caddy
|
# Caddy
|
||||||
@@ -90,7 +90,111 @@ pihole.franksdank.cloud {
|
|||||||
|
|
||||||
After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.
|
After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.
|
||||||
|
|
||||||
Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile:14-24,283-423`). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), and the media/*arr vhosts. Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik (`dankclaw/Caddyfile:122-132`). See [[Homepage]] and [[Authentik]].
|
Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`, `notebook.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile`: the header comment and the two internal listeners). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), the media/*arr vhosts, and `audiobooks.` — Audiobookshelf enforces its own auth (local accounts plus authentik OIDC) and must answer native app clients, podcast RSS and OPDS, which the chain structurally cannot deliver: it answers a client that solves no JavaScript proof-of-work with challenge HTML wherever that client expects JSON or a socket payload ([[Audiobookshelf]]). Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik. See [[Homepage]] and [[Authentik]].
|
||||||
|
|
||||||
|
## Guard a new vhost
|
||||||
|
|
||||||
|
Order is the whole procedure: the IdP row has to exist **before** the route flips, or
|
||||||
|
authenticated traffic reaches the outpost with no matching provider and the user gets
|
||||||
|
authentik's error page instead of the app.
|
||||||
|
|
||||||
|
1. **Provider first.** A proxy provider plus an application, claimed by the embedded
|
||||||
|
outpost. No blueprint owns this state — it lives only in the IdP's database, which is
|
||||||
|
why it is easy to forget:
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
|
||||||
|
from authentik.providers.proxy.models import ProxyProvider
|
||||||
|
from authentik.core.models import Application
|
||||||
|
from authentik.outposts.models import Outpost
|
||||||
|
NAME, EXT = "example", "https://example.franksdank.cloud/"
|
||||||
|
pp = ProxyProvider.objects.filter(name=f"{NAME}-proxy").first()
|
||||||
|
if pp is None:
|
||||||
|
pp = ProxyProvider.objects.create(name=f"{NAME}-proxy", external_host=EXT,
|
||||||
|
internal_host="http://caddy:8017")
|
||||||
|
if not Application.objects.filter(provider=pp).exists():
|
||||||
|
Application.objects.create(name=f"{NAME.title()} Guard", slug=f"{NAME}-guard",
|
||||||
|
provider=pp)
|
||||||
|
o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
|
||||||
|
o.providers.add(pp) # an unclaimed provider is invisible to the outpost
|
||||||
|
print(pp.name, pp.external_host, o.providers.count())
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
`external_host` is the exact vhost URL **with the trailing slash**: the embedded
|
||||||
|
outpost routes on the ORIGINAL `Host` header alone, so anything else matches zero
|
||||||
|
providers. Never print a provider's fields wholesale — a proxy provider carries a
|
||||||
|
client secret, and it lands in whatever transcript the command runs in.
|
||||||
|
|
||||||
|
`ProxyProvider.objects.create()` is not the API: the serializer's
|
||||||
|
`set_oauth_defaults()` never runs, so the row lands with `redirect_uris` and
|
||||||
|
`grant_types` empty, and the guard fails *before* any credential — authentik's
|
||||||
|
"Redirect URI Error" page, then a bodiless HTTP 400 once the URIs are filled
|
||||||
|
by hand. Run `scripts/authentik-proxy-provider-defaults.sh` after the snippet
|
||||||
|
(idempotent, covers every provider; `--check` is the census); law and failure
|
||||||
|
signatures in [[Authentik Outpost Reference]].
|
||||||
|
2. **Then the route.** The public site block proxies the whole request to `anubis:8923`
|
||||||
|
with `header_up X-Real-Ip {remote_host}`, and the post-auth `http://:8017` listener
|
||||||
|
gains a `@<name> host <vhost>` handler pointing at the app — copy `romm`/`files`,
|
||||||
|
which are the same shape including a Dankserver LAN-port upstream.
|
||||||
|
An app that derives its own origin from the request needs the scheme pinned:
|
||||||
|
Audiobookshelf checks every OIDC callback against `x-forwarded-proto://Host`; this
|
||||||
|
listener is fed over plain http so Caddy's own `X-Forwarded-Proto` is `http`, and the
|
||||||
|
app answers a 400 "Invalid callback URL - must be same-origin" for its own https
|
||||||
|
callback (measured 2026-11-03, while this vhost was still guarded — it is exempt
|
||||||
|
now, and the same pin sits on its edge block instead).
|
||||||
|
The edge for a guarded name is TLS-only, so `header_up X-Forwarded-Proto
|
||||||
|
https` is the truthful value ([[Audiobookshelf]]).
|
||||||
|
3. **Then the name and the audit.** A `dankclaw/pihole/addn-hosts` line (the `pihole`
|
||||||
|
seam fails the build if a Caddyfile vhost has no record), a Porkbun A record if the
|
||||||
|
name must resolve off-LAN ([[LAN-DNS]]), and the subdomain added to `GUARDED` in
|
||||||
|
`tests/e2e/specs/14-guard.spec.ts` with an app-specific leak marker — that spec is
|
||||||
|
the deploy gate's proof that the guard, not the app, answers an anonymous client.
|
||||||
|
|
||||||
|
## Unguard a vhost
|
||||||
|
|
||||||
|
The same three artifacts, in the opposite order. Getting it wrong takes the app
|
||||||
|
down for browsers while looking healthy:
|
||||||
|
|
||||||
|
1. **Route first.** Move the public site block off `anubis:8923` to a direct
|
||||||
|
`reverse_proxy` (keeping any origin-pinning `header_up` the app needs), delete
|
||||||
|
the `@<name>` handler from the `http://:8017` listener, and deploy. Traffic
|
||||||
|
stops entering the auth listener at `:8016`, so the IdP row goes unused — and
|
||||||
|
nothing breaks.
|
||||||
|
2. **Then the IdP rows**, once the route is verified:
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
|
||||||
|
from authentik.providers.proxy.models import ProxyProvider
|
||||||
|
from authentik.outposts.models import Outpost
|
||||||
|
pp = ProxyProvider.objects.filter(name="audiobooks-proxy").first()
|
||||||
|
if pp:
|
||||||
|
o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
|
||||||
|
o.providers.remove(pp)
|
||||||
|
app = pp.application # the "<Name> Guard" application row
|
||||||
|
if app:
|
||||||
|
app.delete()
|
||||||
|
pp.delete()
|
||||||
|
print("providers now claimed:", Outpost.objects.get(
|
||||||
|
managed="goauthentik.io/outposts/embedded").providers.count())
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
Deleting the provider **before** the route flips is the wrong order: the
|
||||||
|
outpost then matches zero providers for a Host that still arrives, and every
|
||||||
|
browser gets authentik's error page instead of the app.
|
||||||
|
3. **Then the audit lists.** Drop the subdomain from `GUARDED` in
|
||||||
|
`tests/e2e/specs/14-guard.spec.ts` (and its leak marker), and replace it
|
||||||
|
there with an exemption assertion — the deploy gate must keep proving the
|
||||||
|
exemption is deliberate, the way `plex` does: app bytes to an anonymous
|
||||||
|
client, zero challenge HTML, and the app's own `401` on an unauthenticated
|
||||||
|
API call rather than the gate's `200`.
|
||||||
|
|
||||||
|
What the exemption costs, said plainly: the app's login endpoint becomes
|
||||||
|
internet-visible with no rate limiter in front of it unless the edge carries one
|
||||||
|
(this Caddy build has no `rate_limit` module) or the app ships a lockout (ABS
|
||||||
|
2.37.2 does not). That is the same exposure `jelly.`, `plex.` and the *arr
|
||||||
|
vhosts already live with; the mitigation is the credentials.
|
||||||
|
|
||||||
## Operations
|
## Operations
|
||||||
|
|
||||||
|
|||||||
@@ -107,6 +107,12 @@ job.sh watch # refreshing status table
|
|||||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
||||||
every failure disappeared once the script was a file.
|
every failure disappeared once the script was a file.
|
||||||
|
- **A script fed over stdin owns that stdin.** `ssh host 'bash -s' <<'EOS'` hands
|
||||||
|
the script itself to stdin, so any child that reads stdin — `podman exec -i`,
|
||||||
|
`ssh -t`, a bare `read` — swallows everything after it and the remainder
|
||||||
|
vanishes without a word. It reads exactly like a hang: the commands before the
|
||||||
|
child print, the ones after never existed. Drop `-i`, or give every child
|
||||||
|
`< /dev/null`.
|
||||||
|
|
||||||
## Verify before trusting it (3am law)
|
## Verify before trusting it (3am law)
|
||||||
|
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ is a locator: one row, one clause, no numbers.
|
|||||||
|
|
||||||
| Page | Owns |
|
| Page | Owns |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
| [[Audiobookshelf]] | Audiobookshelf on Dankserver: quadlet shape, array bind, SSD state, API bootstrap, authentik SSO, public vhost, laws. |
|
||||||
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
|
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
|
||||||
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
|
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
|
||||||
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
||||||
@@ -52,9 +53,11 @@ is a locator: one row, one clause, no numbers.
|
|||||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||||
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
||||||
|
| [[Media Duplicate Prune]] | Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge. |
|
||||||
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||||
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
||||||
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||||
|
| [[Media TV Relayout]] | Re-file TVShows from scene packs into the shape Jellyfin reads: authority order, refusals, the two-series split test. |
|
||||||
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
|
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
|
||||||
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
|
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
|
||||||
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
|
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96"><polygon points="48,6 86,27 86,69 48,90 10,69 10,27" fill="#1c1300" stroke="#3fc1c9" stroke-width="3"/><text x="48" y="60" font-family="monospace" font-size="30" font-weight="bold" fill="#3fc1c9" text-anchor="middle">AB</text></svg>
|
||||||
|
After Width: | Height: | Size: 293 B |
@@ -511,6 +511,19 @@
|
|||||||
"showLink": true,
|
"showLink": true,
|
||||||
"art": "/img/apps/mono-tt.svg"
|
"art": "/img/apps/mono-tt.svg"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "Audiobookshelf",
|
||||||
|
"category": "admin",
|
||||||
|
"section": "Media Library Apps",
|
||||||
|
"description": "Audiobook server for the entropy library: its own SQLite library, its own login, per-listener positions.",
|
||||||
|
"url": "http://audiobooks.franksdank.cloud/audiobookshelf/",
|
||||||
|
"docs": "",
|
||||||
|
"probe": "192.168.0.235:8000",
|
||||||
|
"method": "http",
|
||||||
|
"wip": false,
|
||||||
|
"showLink": true,
|
||||||
|
"art": "/img/apps/mono-ab.svg"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "Postgres",
|
"name": "Postgres",
|
||||||
"category": "admin",
|
"category": "admin",
|
||||||
|
|||||||
+11
-2
@@ -75,8 +75,10 @@ PY
|
|||||||
}
|
}
|
||||||
|
|
||||||
@test "exempt vhosts never route through anubis (auth, plex, own-auth apps)" {
|
@test "exempt vhosts never route through anubis (auth, plex, own-auth apps)" {
|
||||||
# The IdP login flow and git smart-HTTP clients cannot solve challenges;
|
# The IdP login flow, git smart-HTTP clients, and any non-browser client (the
|
||||||
# these vhosts bypass the chain by construction.
|
# Audiobookshelf phone app, podcast RSS readers) cannot solve a challenge, so
|
||||||
|
# services that enforce their own complete auth bypass the chain by
|
||||||
|
# construction — the gate would answer their API calls with HTML.
|
||||||
local block
|
local block
|
||||||
block=$(awk '/^auth\.franksdank\.cloud/,/^}/' "$CADDY")
|
block=$(awk '/^auth\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||||
echo "$block" | grep -q 'reverse_proxy authentik-server:9000'
|
echo "$block" | grep -q 'reverse_proxy authentik-server:9000'
|
||||||
@@ -84,4 +86,11 @@ PY
|
|||||||
block=$(awk '/^plex\.franksdank\.cloud/,/^}/' "$CADDY")
|
block=$(awk '/^plex\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||||
echo "$block" | grep -q '192\.168\.0\.235:32400'
|
echo "$block" | grep -q '192\.168\.0\.235:32400'
|
||||||
! echo "$block" | grep -q anubis
|
! echo "$block" | grep -q anubis
|
||||||
|
block=$(awk '/^audiobooks\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||||
|
echo "$block" | grep -q '192\.168\.0\.235:8000'
|
||||||
|
! echo "$block" | grep -q anubis
|
||||||
|
# The exemption is total: the post-auth listener must not still carry an
|
||||||
|
# audiobooks route. A leftover handle there is a second owner for one host,
|
||||||
|
# and its X-Forwarded-Proto pin would silently disagree with the edge.
|
||||||
|
! grep -q '@audiobooks host audiobooks.franksdank.cloud' "$CADDY"
|
||||||
}
|
}
|
||||||
Executable
+276
@@ -0,0 +1,276 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/audiobookshelf-init.sh — the Audiobookshelf bootstrapper.
|
||||||
|
#
|
||||||
|
# A freshly deployed Audiobookshelf answers /healthcheck 200 while it has no
|
||||||
|
# user, no library and no scan, so "healthy" proves nothing about usefulness.
|
||||||
|
# This seam pins what the bootstrapper must make true, against a stub that
|
||||||
|
# speaks the same API as the pinned image (POST /init + POST /login at the
|
||||||
|
# root, bearer-authenticated /api/*, library count only on /items):
|
||||||
|
# 1. an unregistered server gains the account, the library and a scan, and the
|
||||||
|
# credential lands mode 600 with both keys;
|
||||||
|
# 2. a second run creates nothing new (exactly one /init, one library) and
|
||||||
|
# still fires the incremental scan;
|
||||||
|
# 3. neither the bearer token nor the password ever rides a URL — a secret in
|
||||||
|
# a URL or an argv is world-readable in /proc/<pid>/cmdline, which is how
|
||||||
|
# an HF token leaked into a transcript on 2026-09-16;
|
||||||
|
# 4. a server that cannot be reached is refused by URL with exit 1, never
|
||||||
|
# reported as success (a green deploy over a dead API is a lie);
|
||||||
|
# 5. ABS_SCAN outside 0/1 is a usage error raised before anything is touched;
|
||||||
|
# 6. the deployer runs the bootstrapper only after the health gate answered.
|
||||||
|
# 7. the SSO arm sends the base path: ABS builds its redirect_uri as
|
||||||
|
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback, and the
|
||||||
|
# value must be the path the app is mounted under — never the literal
|
||||||
|
# "undefined" a fresh database leaves there.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
INIT="$REPO_ROOT/scripts/audiobookshelf-init.sh"
|
||||||
|
TMP="$(mktemp -d)"
|
||||||
|
PORT=18937
|
||||||
|
cat > "$TMP/stub.py" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
|
||||||
|
port, logf = int(sys.argv[1]), sys.argv[2]
|
||||||
|
state = {"init": False, "user": None, "pw": None, "libs": [], "oidc": None}
|
||||||
|
|
||||||
|
|
||||||
|
def log(line):
|
||||||
|
with open(logf, "a") as f:
|
||||||
|
f.write(line + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def norm(p):
|
||||||
|
# ABS mounts every route under ROUTER_BASE_PATH. The stub accepts the API at the
|
||||||
|
# root or under that prefix, so a test can still assert which one the client used.
|
||||||
|
return p[len("/audiobookshelf"):] if p.startswith("/audiobookshelf") else p
|
||||||
|
|
||||||
|
|
||||||
|
class H(BaseHTTPRequestHandler):
|
||||||
|
def log_message(self, *a):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def send(self, code, body, as_json=False):
|
||||||
|
data = json.dumps(body).encode() if as_json else str(body).encode()
|
||||||
|
self.send_response(code)
|
||||||
|
self.send_header("Content-Type", "application/json" if as_json else "text/plain")
|
||||||
|
self.send_header("Content-Length", str(len(data)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(data)
|
||||||
|
|
||||||
|
def body(self):
|
||||||
|
n = int(self.headers.get("Content-Length") or 0)
|
||||||
|
return json.loads(self.rfile.read(n) or b"{}")
|
||||||
|
|
||||||
|
def authed(self):
|
||||||
|
return self.headers.get("Authorization") == "Bearer stub-token"
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
path = norm(self.path.split("?")[0])
|
||||||
|
log("GET %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||||
|
if path == "/status":
|
||||||
|
methods = ["local", "openid"] if state["oidc"] else ["local"]
|
||||||
|
return self.send(200, {"app": "audiobookshelf", "isInit": state["init"],
|
||||||
|
"authMethods": methods}, True)
|
||||||
|
if not self.authed():
|
||||||
|
return self.send(401, "Unauthorized")
|
||||||
|
if path == "/api/libraries":
|
||||||
|
return self.send(200, {"libraries": state["libs"]}, True)
|
||||||
|
if path == "/api/auth-settings":
|
||||||
|
return self.send(200, state["oidc"] or {}, True)
|
||||||
|
if path.endswith("/items"):
|
||||||
|
return self.send(200, {"total": 42, "results": []}, True)
|
||||||
|
return self.send(404, "nope")
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
path = norm(self.path.split("?")[0])
|
||||||
|
payload = self.body()
|
||||||
|
log("POST %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||||
|
if path == "/init":
|
||||||
|
if state["init"]:
|
||||||
|
return self.send(500, "already initialized")
|
||||||
|
root = payload.get("newRoot", {})
|
||||||
|
state["init"] = True
|
||||||
|
state["user"] = root.get("username")
|
||||||
|
state["pw"] = root.get("password")
|
||||||
|
log("INIT %s" % root.get("username"))
|
||||||
|
return self.send(200, "OK")
|
||||||
|
if path == "/login":
|
||||||
|
if not state["init"]:
|
||||||
|
return self.send(401, "Unauthorized")
|
||||||
|
if (payload.get("username"), payload.get("password")) != (state["user"], state["pw"]):
|
||||||
|
return self.send(401, "Unauthorized")
|
||||||
|
return self.send(200, {"user": {"username": state["user"], "type": "root",
|
||||||
|
"id": "u1", "token": "stub-token"}}, True)
|
||||||
|
if not self.authed():
|
||||||
|
return self.send(401, "Unauthorized")
|
||||||
|
if path == "/api/libraries":
|
||||||
|
lib = {"id": "lib-1", "name": payload.get("name"), "folders": payload.get("folders", [])}
|
||||||
|
state["libs"].append(lib)
|
||||||
|
log("LIBRARY %s %s" % (lib["name"], lib["folders"]))
|
||||||
|
return self.send(200, lib, True)
|
||||||
|
if path.endswith("/scan"):
|
||||||
|
log("SCAN")
|
||||||
|
return self.send(200, "OK")
|
||||||
|
return self.send(404, "nope")
|
||||||
|
|
||||||
|
def do_PATCH(self):
|
||||||
|
path = norm(self.path.split("?")[0])
|
||||||
|
payload = self.body()
|
||||||
|
log("PATCH %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||||
|
if not self.authed():
|
||||||
|
return self.send(401, "Unauthorized")
|
||||||
|
if path == "/api/auth-settings":
|
||||||
|
# Stored and echoed verbatim: the test asserts the wire, not the stub's
|
||||||
|
# opinion about which keys are patchable.
|
||||||
|
state["oidc"] = payload
|
||||||
|
for k in sorted(payload):
|
||||||
|
log("OIDCFG %s=%r" % (k, payload[k]))
|
||||||
|
return self.send(200, payload, True)
|
||||||
|
return self.send(404, "nope")
|
||||||
|
|
||||||
|
|
||||||
|
ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()
|
||||||
|
PY
|
||||||
|
python3 "$TMP/stub.py" "$PORT" "$TMP/requests.log" > "$TMP/stub.out" 2>&1 &
|
||||||
|
STUB_PID=$!
|
||||||
|
for _ in $(seq 1 50); do
|
||||||
|
curl -s -o /dev/null "http://127.0.0.1:$PORT/status" && break
|
||||||
|
sleep 0.1
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
teardown() {
|
||||||
|
[ -n "${STUB_PID:-}" ] && kill "$STUB_PID" 2>/dev/null || true
|
||||||
|
rm -rf "$TMP"
|
||||||
|
}
|
||||||
|
|
||||||
|
boot() {
|
||||||
|
env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" \
|
||||||
|
ABS_ADMIN_USER=frank ABS_LIBRARY=Audiobooks ABS_FOLDER=/audiobooks \
|
||||||
|
ABS_SCAN="${SCAN:-1}" bash "$INIT"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an unregistered server gains the account, the library and the scan" {
|
||||||
|
run boot
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qx "credential: created $TMP/cred"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^account: created (frank)"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^token: obtained (root)"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^library: created Audiobooks (lib-1)"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^items: 42"
|
||||||
|
# The state roots of this service hold an account credential, so the file
|
||||||
|
# that carries it must never be group- or world-readable.
|
||||||
|
[ "$(stat -c %a "$TMP/cred")" = "600" ]
|
||||||
|
grep -qx 'ABS_ADMIN_USER=frank' "$TMP/cred"
|
||||||
|
grep -Eq '^ABS_ADMIN_PASSWORD=[0-9a-f]{24}$' "$TMP/cred"
|
||||||
|
# The folder the library indexes is the bind inside the container, not the
|
||||||
|
# host path — a library pointed at /mnt/entropy would answer 0 items and
|
||||||
|
# look like an empty audiobook collection.
|
||||||
|
grep -q "LIBRARY Audiobooks \[{'fullPath': '/audiobooks'}\]" "$TMP/requests.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a second run creates nothing new and still fires the incremental scan" {
|
||||||
|
boot > /dev/null
|
||||||
|
[ -e "$TMP/requests.log" ]
|
||||||
|
run boot
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^account: already registered"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^library: exists Audiobooks (lib-1)"
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
|
||||||
|
# No second credential, no second account, no duplicate library: the deployer
|
||||||
|
# may run this on every re-deploy of the service.
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -q "credential: created"
|
||||||
|
[ "$(grep -c '^INIT ' "$TMP/requests.log")" -eq 1 ]
|
||||||
|
[ "$(grep -c '^LIBRARY ' "$TMP/requests.log")" -eq 1 ]
|
||||||
|
[ "$(grep -c '^SCAN$' "$TMP/requests.log")" -eq 2 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the bearer token and the password never ride a URL" {
|
||||||
|
boot
|
||||||
|
# The stub logs the request line (method + path). A secret showing up there
|
||||||
|
# means it travelled as a query parameter — visible in access logs, shell
|
||||||
|
# history and /proc/<pid>/cmdline.
|
||||||
|
! awk '{print $2}' "$TMP/requests.log" | grep -q 'stub-token'
|
||||||
|
pw=$(awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' "$TMP/cred")
|
||||||
|
[ -n "$pw" ]
|
||||||
|
! awk '{print $2}' "$TMP/requests.log" | grep -q -- "$pw"
|
||||||
|
# and the authenticated routes only answered once the header was present
|
||||||
|
grep -q '^GET /api/libraries auth=yes$' "$TMP/requests.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a server that does not answer is refused by URL, not reported as success" {
|
||||||
|
run env ABS_URL="http://127.0.0.1:1" ABS_SECRET="$TMP/cred" bash "$INIT"
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[[ "$output" == *refused:* ]]
|
||||||
|
[[ "$output" == *"127.0.0.1:1"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "ABS_SCAN outside 0/1 is a usage error raised before anything is touched" {
|
||||||
|
run env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" ABS_SCAN=2 \
|
||||||
|
bash "$INIT"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *ABS_SCAN* ]]
|
||||||
|
[ ! -e "$TMP/cred" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the deployer bootstraps only after the health gate has answered" {
|
||||||
|
body=$(awk '/^deploy_audiobookshelf\(\)/,/^}/' "$REPO_ROOT/scripts/deploy-remote.sh")
|
||||||
|
[ -n "$body" ]
|
||||||
|
printf '%s\n' "$body" | grep -q \
|
||||||
|
'wait_for_health audiobookshelf http://127\.0\.0\.1:8000/healthcheck'
|
||||||
|
printf '%s\n' "$body" | grep -q 'scripts/audiobookshelf-init\.sh'
|
||||||
|
printf '%s\n' "$body" | awk '
|
||||||
|
/wait_for_health audiobookshelf/ { health = NR }
|
||||||
|
/audiobookshelf-init\.sh/ { init = NR }
|
||||||
|
END { exit !(health && init && init > health) }'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the SSO arm sends the base path as the redirect subfolder" {
|
||||||
|
# What authentik has registered must equal what ABS builds:
|
||||||
|
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback. A database
|
||||||
|
# created before SSO leaves that key unset and ABS then sends the literal
|
||||||
|
# "/undefined/auth/openid/callback" — measured 2026-11-02, where the live login died
|
||||||
|
# on authentik's "Redirect URI Error" page. So the value has to reach the wire, and
|
||||||
|
# it has to be the path the app is served under.
|
||||||
|
umask 077
|
||||||
|
cat > "$TMP/cred" <<'CRED'
|
||||||
|
ABS_ADMIN_USER=frank
|
||||||
|
ABS_ADMIN_PASSWORD=0123456789abcdef0123456789
|
||||||
|
ABS_OIDC_CLIENT_SECRET=0123456789abcdef0123456789abcdef0123456789abcdef0123456789
|
||||||
|
CRED
|
||||||
|
run env ABS_URL="http://127.0.0.1:$PORT/audiobookshelf" ABS_SECRET="$TMP/cred" \
|
||||||
|
ABS_SCAN=0 bash "$INIT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q "^oidc: armed"
|
||||||
|
grep -q "^OIDCFG authOpenIDSubfolderForRedirectURLs='/audiobookshelf'$" "$TMP/requests.log"
|
||||||
|
# every route the bootstrapper calls rides under the same prefix it armed with
|
||||||
|
grep -q "^POST /audiobookshelf/init auth=no$" "$TMP/requests.log"
|
||||||
|
grep -q "^GET /audiobookshelf/api/auth-settings auth=yes$" "$TMP/requests.log"
|
||||||
|
|
||||||
|
# A root-mounted deployment sends the EMPTY subfolder (the value the 2.17.4 upgrade
|
||||||
|
# uses), not whatever the last URL happened to carry.
|
||||||
|
run env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" ABS_SCAN=0 bash "$INIT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(grep "^OIDCFG authOpenIDSubfolderForRedirectURLs=" "$TMP/requests.log" | tail -1)" \
|
||||||
|
= "OIDCFG authOpenIDSubfolderForRedirectURLs=''" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the app and the IdP provider script build the same redirect path" {
|
||||||
|
# The provider row is written by scripts/authentik-audiobookshelf-sso.sh from
|
||||||
|
# ABS_BASE; the app builds its redirect_uri from the base path the deployer hands
|
||||||
|
# audiobookshelf-init.sh, which comes from the unit. Drift between the two is a
|
||||||
|
# redirect_uri mismatch — the login dies at the IdP before any credential is issued.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
mounted=$(awk -F= '/^Environment=ROUTER_BASE_PATH=/{print $3}' "$unit")
|
||||||
|
[ -n "$mounted" ]
|
||||||
|
provider=$(grep -m1 '^BASE=' "$REPO_ROOT/scripts/authentik-audiobookshelf-sso.sh" \
|
||||||
|
| grep -oE 'https?://[^"}]+')
|
||||||
|
bootstrapper=$(grep -m1 '^ABS_URL=' "$INIT" | grep -oE 'https?://[^"}]+')
|
||||||
|
# Origins legitimately differ (LAN address vs loopback); the PATH must not.
|
||||||
|
[ "${provider#http://192.168.0.235:8000}/auth/openid/callback" \
|
||||||
|
= "${bootstrapper#http://127.0.0.1:8000}/auth/openid/callback" ]
|
||||||
|
[ "${bootstrapper#http://127.0.0.1:8000}" = "$mounted" ]
|
||||||
|
}
|
||||||
@@ -127,3 +127,68 @@ load helpers
|
|||||||
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
|
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
|
||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf unit exists, is boot-persistent and self-healing" {
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
[ -f "$unit" ] || return 1
|
||||||
|
grep -q '^WantedBy=default.target$' "$unit" || return 1
|
||||||
|
grep -q '^Restart=always$' "$unit" || return 1
|
||||||
|
grep -q '^RestartSec=' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf keeps every byte it writes on the SSD" {
|
||||||
|
# The point of the unit: its library database, accounts, progress and cached
|
||||||
|
# covers are all server state, and server state belongs on the NVMe. A bind
|
||||||
|
# of /mnt/entropy that is not :ro would silently put writes on the spindle.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^Volume=/home/dank/Containers/audiobookshelf/config:/config$' "$unit" || return 1
|
||||||
|
grep -q '^Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata$' "$unit" || return 1
|
||||||
|
grep -q '^Volume=/mnt/entropy/Audiobooks:/audiobooks:ro$' "$unit" || return 1
|
||||||
|
# The overlay view is reserved for the two media servers that write sidecars
|
||||||
|
# (tests/media_overlay.bats pins that set); a reader must not join it.
|
||||||
|
! grep -q 'media-meta/view' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf refuses to start against an unmounted array" {
|
||||||
|
# Without the gate podman invents an empty bind source and the first scan
|
||||||
|
# records an empty library as truth.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^ExecStartPre=.*/wait-for-mount.sh /mnt/entropy$' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf health check names a binary its image actually ships" {
|
||||||
|
# The image is Alpine: wget yes, curl no. A curl HealthCmd reads the
|
||||||
|
# container as unhealthy while it is serving perfectly.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^HealthCmd=wget ' "$unit" || return 1
|
||||||
|
! grep -q '^HealthCmd=curl' "$unit" || return 1
|
||||||
|
grep -q '^PublishPort=8000:8000$' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the audiobookshelf deployer creates both state roots before it starts" {
|
||||||
|
body="$(sed -n '/^deploy_audiobookshelf()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
mkdir_line="$(printf '%s\n' "$body" | grep -n 'mkdir -p' | head -1 | cut -d: -f1)"
|
||||||
|
start_line="$(printf '%s\n' "$body" | grep -n 'systemctl --user restart' | head -1 | cut -d: -f1)"
|
||||||
|
[ -n "$mkdir_line" ] && [ -n "$start_line" ] || return 1
|
||||||
|
[ "$mkdir_line" -lt "$start_line" ] || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'audiobookshelf/config' || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'audiobookshelf/metadata' || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'wait_for_health audiobookshelf' || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf sets its port and base path with the names the image reads" {
|
||||||
|
# Measured 2026-11-02 on the first deploy: ABS_PORT / ABS_CONFIG_DIR are the npm
|
||||||
|
# package's names and this image ignores them, so the server came up on :80
|
||||||
|
# under base path /audiobookshelf and the deploy gate timed out beside a unit
|
||||||
|
# that was active (running). PORT and ROUTER_BASE_PATH are what the image's own
|
||||||
|
# ENV block and the server source actually use.
|
||||||
|
# The base path is not a free choice either: /app/client/dist/index.html inside the
|
||||||
|
# pinned image carries routerBasePath:"/audiobookshelf", so a server mounted at / hands
|
||||||
|
# out a page whose every script request falls through to the SPA fallback and comes back
|
||||||
|
# text/html — the UI never leaves "Loading…" while the API stays healthy (measured
|
||||||
|
# 2026-11-02, which is exactly how it hid behind an API-only verification).
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^Environment=PORT=8000$' "$unit" || return 1
|
||||||
|
grep -q '^Environment=ROUTER_BASE_PATH=/audiobookshelf$' "$unit" || return 1
|
||||||
|
! grep -qE '^Environment=ABS_' "$unit" || return 1
|
||||||
|
}
|
||||||
@@ -28,15 +28,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
|||||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the retired ik chat template plans nothing, and its archive is still here" {
|
@test "the retired ik chat template plans nothing" {
|
||||||
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
||||||
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
||||||
# llama.cpp frank-zealot runtime serves the template the GGUF itself
|
# llama.cpp frank-zealot runtime serves the template the GGUF itself
|
||||||
# carries, which is the one with reasoning_effort, preserve_thinking and
|
# carries, which is the one with reasoning_effort, preserve_thinking and
|
||||||
# the tool-call grammar. Editing the archive must deploy nothing -- and the
|
# the tool-call grammar. The planner's rule is a PATH rule, so an edit at
|
||||||
# archive must still be present, so this cannot pass by the file having
|
# that path deploys nothing no matter where the bytes currently sit: the seam
|
||||||
# vanished.
|
# deliberately does not assert the archive exists. Trash is a purgeable hold
|
||||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp" ]
|
# (untracked in #111, and the deletion policy says the purger deletes it), so
|
||||||
|
# pinning it fails every fresh checkout and proves nothing about planning.
|
||||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
[ -z "$output" ]
|
[ -z "$output" ]
|
||||||
|
|||||||
@@ -42,7 +42,10 @@ export default defineConfig({
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "admin",
|
name: "admin",
|
||||||
testMatch: /11-.*\.spec\.ts/,
|
// 23- joins 2026-11-02 (Audiobookshelf SSO): it needs BOTH halves of the admin
|
||||||
|
// fixture — a live authentik session to ride the OIDC round trip, and the
|
||||||
|
// qa-e2e-admin group membership that grants site admin in the app.
|
||||||
|
testMatch: /(11|23)-.*\.spec\.ts/,
|
||||||
use: { storageState: ".auth/qa-e2e-admin.json" },
|
use: { storageState: ".auth/qa-e2e-admin.json" },
|
||||||
dependencies: ["setup"],
|
dependencies: ["setup"],
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -6,9 +6,10 @@ import { expect, test } from "@playwright/test";
|
|||||||
// navigation would skip. These tests drive the browser as an UNAUTHENTICATED
|
// navigation would skip. These tests drive the browser as an UNAUTHENTICATED
|
||||||
// user (the guest project has no storage state) and navigate directly to each
|
// user (the guest project has no storage state) and navigate directly to each
|
||||||
// guarded vhost's root path: the guard must present authentik's flow UI in
|
// guarded vhost's root path: the guard must present authentik's flow UI in
|
||||||
// place and must never leak app content. plex is the documented exemption —
|
// place and must never leak app content. plex and audiobooks are the documented
|
||||||
// PMS enforces its own auth — so it must be reachable unauthenticated with
|
// exemptions — PMS and Audiobookshelf enforce their own auth — so they must be
|
||||||
// zero challenge pages, proving the exemption is deliberate, not accidental.
|
// reachable unauthenticated with zero challenge pages, proving the exemption is
|
||||||
|
// deliberate, not accidental.
|
||||||
|
|
||||||
const BASE = "https://www.franksdank.cloud";
|
const BASE = "https://www.franksdank.cloud";
|
||||||
|
|
||||||
@@ -17,7 +18,9 @@ const BASE = "https://www.franksdank.cloud";
|
|||||||
// added here or the guard-chain audit is incomplete.
|
// added here or the guard-chain audit is incomplete.
|
||||||
const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
|
const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
|
||||||
|
|
||||||
// Markers that prove app content leaked past the guard to an anonymous user.
|
// Markers that prove app content leaked past the guard to an anonymous user. Each
|
||||||
|
// marker must be a string the APP renders that the gate itself could never print:
|
||||||
|
// authentik's gate never renders an app's UI text, because it IS the IdP.
|
||||||
const LEAK_MARKERS: Record<string, string[]> = {
|
const LEAK_MARKERS: Record<string, string[]> = {
|
||||||
"search.franksdank.cloud": ["SearXNG preferences"],
|
"search.franksdank.cloud": ["SearXNG preferences"],
|
||||||
"romm.franksdank.cloud": ["RomM"],
|
"romm.franksdank.cloud": ["RomM"],
|
||||||
@@ -98,3 +101,54 @@ test("plex is exempt from the guard chain and serves PMS content unauthenticated
|
|||||||
const resp = await page.request.get("https://plex.franksdank.cloud/");
|
const resp = await page.request.get("https://plex.franksdank.cloud/");
|
||||||
expect(resp.status(), "PMS must answer directly, not error").toBeLessThan(500);
|
expect(resp.status(), "PMS must answer directly, not error").toBeLessThan(500);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Audiobookshelf is exempt by construction (2026-11-03). The phone app is a
|
||||||
|
// non-browser client and solves no JS proof-of-work, so behind the guard chain it
|
||||||
|
// was answered with challenge HTML wherever the socket `init` payload belongs —
|
||||||
|
// every remote sign-in died as a TypeError on a non-object, and the container's
|
||||||
|
// request log never saw a byte of it. ABS enforces its own auth (local accounts
|
||||||
|
// plus authentik OIDC), exactly like PMS above. Asserted from the CLIENT side: the
|
||||||
|
// endpoints an app must call before it holds any credential are answered by the
|
||||||
|
// app, and the app's own 401 — not a 200 of HTML — is what an anonymous API call gets.
|
||||||
|
test("audiobookshelf answers an unauthenticated app client with app bytes, not gate bytes", async ({
|
||||||
|
request,
|
||||||
|
}) => {
|
||||||
|
const ABS = "https://audiobooks.franksdank.cloud";
|
||||||
|
|
||||||
|
// The probe every Audiobookshelf client sends first. The app answers it at the
|
||||||
|
// ROOT whatever ROUTER_BASE_PATH says, so this is the vhost's own behaviour.
|
||||||
|
const ping = await request.get(`${ABS}/ping`, { timeout: 30_000 });
|
||||||
|
expect(ping.status(), "/ping must be answered by the app").toBe(200);
|
||||||
|
expect(await ping.json(), "/ping must carry the app's own payload").toEqual({ success: true });
|
||||||
|
|
||||||
|
// The socket.io handshake. Engine.IO answers this in a machine format; the gate
|
||||||
|
// answered it in HTML, which is the defect this exemption exists to kill.
|
||||||
|
const sock = await request.get(`${ABS}/audiobookshelf/socket.io/?EIO=4&transport=polling`, {
|
||||||
|
timeout: 30_000,
|
||||||
|
});
|
||||||
|
expect(
|
||||||
|
await sock.text(),
|
||||||
|
"the socket handshake was answered by the gate, not Engine.IO",
|
||||||
|
).not.toContain("Making sure you're not a bot");
|
||||||
|
expect(
|
||||||
|
sock.headers()["content-type"] ?? "",
|
||||||
|
"the socket handshake must not arrive as HTML",
|
||||||
|
).not.toMatch(/text\/html/);
|
||||||
|
|
||||||
|
// An API call with no credential: the app refuses it. A gate would answer 200
|
||||||
|
// with HTML, so the status code alone distinguishes the two owners.
|
||||||
|
const api = await request.get(`${ABS}/audiobookshelf/api/libraries`, { timeout: 30_000 });
|
||||||
|
expect(api.status(), "the app's own auth gate must own unauthenticated API calls").toBe(401);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("audiobookshelf login renders the app's own sign-in to an anonymous browser", async ({ page }) => {
|
||||||
|
test.setTimeout(120_000);
|
||||||
|
await page.goto("https://audiobooks.franksdank.cloud/audiobookshelf/login");
|
||||||
|
expect(new URL(page.url()).hostname).toBe("audiobooks.franksdank.cloud");
|
||||||
|
|
||||||
|
const body = await page.locator("body").innerText();
|
||||||
|
expect(body, "the bot challenge still fronts this vhost").not.toContain("Making sure you're not a bot");
|
||||||
|
expect(await page.locator('input[name="uidField"]').count(), "the IdP gate form must not front the app").toBe(0);
|
||||||
|
// The app answers, and its login screen carries SSO beside the local form.
|
||||||
|
await expect(page.getByText("Sign in with Authentik")).toBeVisible({ timeout: 90_000 });
|
||||||
|
});
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
import { expect, test } from "@playwright/test";
|
||||||
|
|
||||||
|
// Audiobookshelf: the SSO round trip must land a session with a real permission level.
|
||||||
|
//
|
||||||
|
// The service answers on a LAN address with no DNS name and no Caddy vhost, so it is in no
|
||||||
|
// guarded-vhost list and no other spec ever fetched a byte of it — every deploy gate passed
|
||||||
|
// while its login was dead. Two defect classes, both caught live on 2026-11-02:
|
||||||
|
//
|
||||||
|
// 1. The bundle must be BUILT for /audiobookshelf (ROUTER_BASE_PATH in the quadlet). Built
|
||||||
|
// for "/" the container still answers /healthcheck and still serves index.html, so the
|
||||||
|
// deploy health gate is green while the browser sees a blank page: every <script>
|
||||||
|
// resolves against the root, misses the static allowlist, and falls through to the SPA
|
||||||
|
// catch-all that returns index.html as text/html. Hence the MIME assertion below — the
|
||||||
|
// one thing that distinguishes a booted SPA from a served HTML stub.
|
||||||
|
// 2. The authentik provider row must carry grant_types AND the built-in OpenID scope
|
||||||
|
// mappings (provisioned by scripts/authentik-audiobookshelf-sso.sh). A blueprint
|
||||||
|
// re-import reverts hand-made provider rows silently, and the wire errors are generic:
|
||||||
|
// oauthlib's "The request is otherwise malformed" for an empty grant list, then
|
||||||
|
// "insufficient_scope" from ABS's /userinfo call for a short scope. Only the level
|
||||||
|
// assertion below — that the abs_groups claim granted site admin — sees either.
|
||||||
|
const HOST = "192.168.0.235:8000";
|
||||||
|
const ABS = `http://${HOST}/audiobookshelf`;
|
||||||
|
|
||||||
|
test("SSO round trip boots the SPA and grants a real Audiobookshelf session", async ({
|
||||||
|
page,
|
||||||
|
}) => {
|
||||||
|
test.setTimeout(180_000);
|
||||||
|
|
||||||
|
const scripts: { url: string; type: string; status: number }[] = [];
|
||||||
|
page.on("response", (res) => {
|
||||||
|
const u = new URL(res.url());
|
||||||
|
if (u.host === HOST && u.pathname.endsWith(".js")) {
|
||||||
|
scripts.push({ url: u.pathname, type: res.headers()["content-type"] ?? "", status: res.status() });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
await page.goto(`${ABS}/login`, { waitUntil: "domcontentloaded" });
|
||||||
|
const sso = page.getByText("Sign in with Authentik");
|
||||||
|
await expect(sso).toBeVisible();
|
||||||
|
await sso.click();
|
||||||
|
|
||||||
|
// The admin project's storageState carries a live authentik session, so this is a silent
|
||||||
|
// round trip: authorize -> code -> callback -> token. Still on /login means the IdP
|
||||||
|
// refused the authorize request or ABS rejected the code exchange.
|
||||||
|
await expect
|
||||||
|
.poll(() => !new URL(page.url()).pathname.endsWith("/login"), { timeout: 90_000 })
|
||||||
|
.toBe(true);
|
||||||
|
|
||||||
|
await expect
|
||||||
|
.poll(
|
||||||
|
async () => {
|
||||||
|
const u = new URL(page.url());
|
||||||
|
return u.host === HOST && u.pathname.startsWith("/audiobookshelf");
|
||||||
|
},
|
||||||
|
{ timeout: 30_000 },
|
||||||
|
)
|
||||||
|
.toBe(true);
|
||||||
|
|
||||||
|
// Defect 1 gate: nothing the app loaded may be HTML dressed as a script. The render proof
|
||||||
|
// is the SSO button above — it only exists once the bundle has executed, so no DOM-shape
|
||||||
|
// guess is needed (the shell's own mount markup says nothing about whether it booted).
|
||||||
|
// Only responses that carried a body can be judged: the OIDC bounce re-validates the entry
|
||||||
|
// script (Cache-Control: max-age=0 + ETag) and a 304 carries no Content-Type by spec.
|
||||||
|
const bodies = scripts.filter((s) => s.status === 200);
|
||||||
|
expect(bodies.length, "the SPA loaded no JavaScript").toBeGreaterThan(2);
|
||||||
|
for (const s of bodies) {
|
||||||
|
expect(s.type, `${s.url} arrived as ${s.type}`).toContain("javascript");
|
||||||
|
}
|
||||||
|
|
||||||
|
// The token the app landed on is the credential its own API must accept. Give the SPA a
|
||||||
|
// moment to finish the exchange: the URL leaves /login the instant authentik bounces back
|
||||||
|
// through the callback route, which is BEFORE the app has stored what it exchanged.
|
||||||
|
const grabToken = () =>
|
||||||
|
page.evaluate(() => {
|
||||||
|
for (let i = 0; i < localStorage.length; i++) {
|
||||||
|
const v = localStorage.getItem(localStorage.key(i) as string) || "";
|
||||||
|
if (/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(v)) return v;
|
||||||
|
}
|
||||||
|
const c = document.cookie.match(/(?:^|;\s*)abs_token=([^;]+)/);
|
||||||
|
return c ? decodeURIComponent(c[1]) : "";
|
||||||
|
});
|
||||||
|
await expect.poll(grabToken, { timeout: 60_000 }).not.toBe("");
|
||||||
|
const token = await grabToken();
|
||||||
|
|
||||||
|
const me = await page.request.get(`${ABS}/api/me`, {
|
||||||
|
headers: { Authorization: `Bearer ${token}` },
|
||||||
|
});
|
||||||
|
expect(me.status()).toBe(200);
|
||||||
|
const user = await me.json();
|
||||||
|
expect(user.username).toBe("qa-e2e-admin");
|
||||||
|
// Site role, derived ONLY from the abs_groups claim: it says "admin" if and only if the
|
||||||
|
// scope mapping reached ABS with audiobookshelf_admins in the list. A short token scope
|
||||||
|
// (missing OpenID mappings) never gets this far — the exchange dies before it.
|
||||||
|
expect(user.type).toBe("admin");
|
||||||
|
});
|
||||||
@@ -13,9 +13,12 @@
|
|||||||
# 3. a downloaded plugin artifact is accepted only on its pinned sha256, its
|
# 3. a downloaded plugin artifact is accepted only on its pinned sha256, its
|
||||||
# official guid and its pinned version (a swapped or downgraded artifact
|
# official guid and its pinned version (a swapped or downgraded artifact
|
||||||
# is refused before anything lands in plugins/);
|
# is refused before anything lands in plugins/);
|
||||||
# 4. the TMDb key is written into the EXISTING config with every other
|
# 4. a key is written into the EXISTING config with every other setting
|
||||||
# setting preserved — Jellyfin owns that file, and clobbering it would
|
# preserved, under whatever root that file's owner chose — Jellyfin's core
|
||||||
# silently reset country, language and adult settings;
|
# configs say `<Configuration>`, TheTVDB writes `<PluginConfiguration>` and
|
||||||
|
# omits a null setting outright, so the element to fill can be absent and
|
||||||
|
# the anchor has to come from the file; clobbering the file would silently
|
||||||
|
# reset country, language and adult settings;
|
||||||
# 5. a TVDB config the operator already saved through the UI is kept;
|
# 5. a TVDB config the operator already saved through the UI is kept;
|
||||||
# 6. install is idempotent: a second run changes no byte;
|
# 6. install is idempotent: a second run changes no byte;
|
||||||
# 7. whatever is replaced is copied to the trash first (fleet law: never
|
# 7. whatever is replaced is copied to the trash first (fleet law: never
|
||||||
@@ -24,6 +27,9 @@
|
|||||||
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
||||||
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
||||||
# installer to jellyfin.
|
# installer to jellyfin.
|
||||||
|
# 10. verify refuses a library that is a SILENT FILE INDEX — internet providers
|
||||||
|
# switched off, or a type whose ImageFetchers is empty while its
|
||||||
|
# MetadataFetchers list is populated (the 2026-11-01 missing-posters fault).
|
||||||
|
|
||||||
load helpers
|
load helpers
|
||||||
|
|
||||||
@@ -59,6 +65,32 @@ setup() {
|
|||||||
TRASH="$BATS_TEST_TMPDIR/trash"
|
TRASH="$BATS_TEST_TMPDIR/trash"
|
||||||
SEC="$BATS_TEST_TMPDIR/secrets"
|
SEC="$BATS_TEST_TMPDIR/secrets"
|
||||||
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
||||||
|
# The per-library option file Jellyfin 12 owns (it was library.xml). Conforming
|
||||||
|
# by default so the pre-existing verify tests keep testing what they tested.
|
||||||
|
mkdir -p "$JF/root/default/Movies"
|
||||||
|
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<EnableInternetProviders>true</EnableInternetProviders>
|
||||||
|
<MetadataSavers>
|
||||||
|
<string>Nfo</string>
|
||||||
|
</MetadataSavers>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>Movie</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetchers>
|
||||||
|
<ImageFetcherOrder>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetcherOrder>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
||||||
<?xml version="1.0" encoding="utf-8"?>
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
@@ -225,6 +257,38 @@ XML
|
|||||||
grep -qF '<EnableSeasonPathMatching>true</EnableSeasonPathMatching>' "$cfg"
|
grep -qF '<EnableSeasonPathMatching>true</EnableSeasonPathMatching>' "$cfg"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "the key lands in a plugin-written config with no ApiKey element" {
|
||||||
|
# What TheTVDB actually writes on a fresh install: its own root element and
|
||||||
|
# no <ApiKey /> at all, because its serializer drops null settings. The
|
||||||
|
# installer used to anchor on </Configuration> and refuse this file.
|
||||||
|
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<PluginConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<SubscriberPIN />
|
||||||
|
<CacheDurationInHours>1</CacheDurationInHours>
|
||||||
|
<OriginalTitle>true</OriginalTitle>
|
||||||
|
</PluginConfiguration>
|
||||||
|
XML
|
||||||
|
run prov install
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'filled the ApiKey'
|
||||||
|
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
grep -qF "<ApiKey>$TVDB_KEY</ApiKey>" "$cfg"
|
||||||
|
grep -qF '<CacheDurationInHours>1</CacheDurationInHours>' "$cfg"
|
||||||
|
grep -qF '<OriginalTitle>true</OriginalTitle>' "$cfg"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a config file with no root element is refused by name" {
|
||||||
|
# The guard still has to fire when there is genuinely nowhere to write —
|
||||||
|
# a refusal naming the file beats inventing a document shape.
|
||||||
|
printf '<?xml version="1.0" encoding="utf-8"?>\n' \
|
||||||
|
>"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
run prov install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qF 'Jellyfin.Plugin.Tvdb.xml'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'no place to write'
|
||||||
|
}
|
||||||
|
|
||||||
@test "a pin in the TVDB secret lands in the plugin config" {
|
@test "a pin in the TVDB secret lands in the plugin config" {
|
||||||
# A subscriber-shaped key needs the account's PIN beside it; a licensed key
|
# A subscriber-shaped key needs the account's PIN beside it; a licensed key
|
||||||
# does not, so the element appears only when the secret carries one.
|
# does not, so the element appears only when the secret carries one.
|
||||||
@@ -270,6 +334,69 @@ XML
|
|||||||
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "verify refuses a library whose internet providers are switched off" {
|
||||||
|
# The whole library becomes a file index: refreshes answer 204 and store
|
||||||
|
# nothing, and the item count still looks like a healthy scan.
|
||||||
|
prov install >/dev/null
|
||||||
|
sed -i 's#<EnableInternetProviders>true</EnableInternetProviders>#<EnableInternetProviders>false</EnableInternetProviders>#' \
|
||||||
|
"$JF/root/default/Movies/options.xml"
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'SILENT INDEX Movies'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'EnableInternetProviders'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "verify refuses a type with no image fetchers while metadata fetchers are named" {
|
||||||
|
prov install >/dev/null
|
||||||
|
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions>
|
||||||
|
<EnableInternetProviders>true</EnableInternetProviders>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>Movie</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers />
|
||||||
|
<ImageFetcherOrder>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetcherOrder>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'ImageFetchers empty'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an older options.xml that never carried the switch is not refused" {
|
||||||
|
# BoxSets on this host were written before the property existed; absence is
|
||||||
|
# the vendor default, not a switched-off library, so it must not fail.
|
||||||
|
prov install >/dev/null
|
||||||
|
mkdir -p "$JF/root/default/Collections"
|
||||||
|
cat >"$JF/root/default/Collections/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>BoxSet</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetchers>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF '= Collections'
|
||||||
|
}
|
||||||
|
|
||||||
@test "no plugin configuration XML is tracked in this repo" {
|
@test "no plugin configuration XML is tracked in this repo" {
|
||||||
# These files hold provider keys and are owned by the running container;
|
# These files hold provider keys and are owned by the running container;
|
||||||
# tracking them would put credentials in git and guarantee drift.
|
# tracking them would put credentials in git and guarantee drift.
|
||||||
|
|||||||
@@ -10,7 +10,8 @@
|
|||||||
# 4. both media units bind the VIEW and refuse to start without it;
|
# 4. both media units bind the VIEW and refuse to start without it;
|
||||||
# 5. only those two units touch the view — the LLM and library tenants keep the
|
# 5. only those two units touch the view — the LLM and library tenants keep the
|
||||||
# raw volume, so the blast radius is two containers;
|
# raw volume, so the blast radius is two containers;
|
||||||
# 6. the deployer brings the view up BEFORE it restarts either server;
|
# 6. the deployer creates the upper base and brings the view up BEFORE it
|
||||||
|
# restarts either server;
|
||||||
# 7. the planner routes a change to these files to both media services.
|
# 7. the planner routes a change to these files to both media services.
|
||||||
|
|
||||||
load helpers
|
load helpers
|
||||||
@@ -110,6 +111,30 @@ UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
|||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "ensure-base creates a base directory that install refuses to invent" {
|
||||||
|
base="$BATS_TEST_TMPDIR/fresh-base"
|
||||||
|
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -d "$base" ]
|
||||||
|
# The deployer runs this on every media deploy, so a second call must be a
|
||||||
|
# no-op and must say so rather than recreate anything.
|
||||||
|
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "${lines[0]}" == *"exists"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the media deployer creates the upper base before it installs the view" {
|
||||||
|
body="$(sed -n '/^ensure_media_view()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
[ -n "$body" ]
|
||||||
|
base="$(printf '%s\n' "$body" | grep -n 'media-view.sh ensure-base' | head -1 | cut -d: -f1)"
|
||||||
|
inst="$(printf '%s\n' "$body" | grep -n 'media-view.sh install' | head -1 | cut -d: -f1)"
|
||||||
|
# Order is the whole test: install refuses a missing base, so an installer
|
||||||
|
# that runs first dies on a rebuilt host (measured 2026-10-31).
|
||||||
|
[ -n "$base" ]
|
||||||
|
[ -n "$inst" ]
|
||||||
|
[ "$base" -lt "$inst" ]
|
||||||
|
}
|
||||||
|
|
||||||
@test "the planner routes the view files to both media services" {
|
@test "the planner routes the view files to both media services" {
|
||||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
||||||
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
||||||
|
|||||||
@@ -460,6 +460,39 @@ print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "
|
|||||||
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# A hand-written plan row whose verb is not in the executor's vocabulary used
|
||||||
|
# to be dropped in silence: read_plan's if/elif chain had no else, so a plan of
|
||||||
|
# nothing but typos applied as a successful no-op (hit for real on 2026-11-02,
|
||||||
|
# when a pass meant to rename two series folders reported "applied: 0 moved"
|
||||||
|
# and exit 0). An unread row is a plan bug, and a plan bug refuses.
|
||||||
|
@test "a plan row with a verb this tool does not know stops the whole run" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
at "$LIB/Stray/Stray.S01E01.mkv" 4000
|
||||||
|
printf 'rename\t%s\t%s\tfile\n' \
|
||||||
|
"$LIB/Stray/Stray.S01E01.mkv" "$LIB/Stray/Stray - S01E01.mkv" >> "$OUT"
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"rename"* ]]
|
||||||
|
# all-or-nothing: the rows this tool DOES understand did not run either
|
||||||
|
[ -e "$LIB/Stray/Stray.S01E01.mkv" ]
|
||||||
|
[ ! -e "$JOURNAL" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a plan whose every row is an unknown verb refuses instead of reporting success" {
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
at "$LIB/Stray/Stray.S01E01.mkv" 4000
|
||||||
|
{ printf 'action\tsrc\tdst\tkind\n'
|
||||||
|
printf 'rename\t%s\t%s\tfile\n' \
|
||||||
|
"$LIB/Stray/Stray.S01E01.mkv" "$LIB/Stray/Stray - S01E01.mkv"
|
||||||
|
} > "$BATS_TEST_TMPDIR/typo.tsv"
|
||||||
|
run python3 "$APPLY" apply --plan "$BATS_TEST_TMPDIR/typo.tsv" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"unknown action"* ]]
|
||||||
|
[[ "$output" != *"applied"* ]]
|
||||||
|
[ -e "$LIB/Stray/Stray.S01E01.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
@test "a run interrupted between the journal write and the move finishes the job" {
|
@test "a run interrupted between the journal write and the move finishes the job" {
|
||||||
plan
|
plan
|
||||||
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
|||||||
Executable
+215
@@ -0,0 +1,215 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/media-tvdupes.py — the episode-level duplicate census for the TV
|
||||||
|
# library.
|
||||||
|
#
|
||||||
|
# The failure this seam exists to prevent is the one that almost cost real
|
||||||
|
# content: two folders of one show look like copies, and a folder-level "keep
|
||||||
|
# the bigger folder" deletes the half of the series only the smaller folder
|
||||||
|
# holds (the 2026-11-02 census found 10 such complementary pairs). So every
|
||||||
|
# assertion here is about WHICH files land in the plan and WHICH the tool
|
||||||
|
# refuses to touch — never about its internals. Everything runs on a synthetic
|
||||||
|
# library in the test's own tmpdir: no media server, no network, no array.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
TOOL="$REPO_ROOT/scripts/media-tvdupes.py"
|
||||||
|
|
||||||
|
# at PATH SIZE — a file of deterministic bytes.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
# sources FILE — the sources the plan moves, one per line.
|
||||||
|
sources() { awk -F'\t' '$1=="move"{print $2}' "$1"; }
|
||||||
|
|
||||||
|
# destinations FILE — where the plan puts them.
|
||||||
|
destinations() { awk -F'\t' '$1=="move"{print $3}' "$1"; }
|
||||||
|
|
||||||
|
# plan_moves FILE — how many files the plan stages.
|
||||||
|
plan_moves() {
|
||||||
|
[ -f "$1" ] || { echo 0; return; }
|
||||||
|
awk -F'\t' '$1=="move"' "$1" | wc -l
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
LIB="$BATS_TEST_TMPDIR/TVShows"
|
||||||
|
HOLD="$BATS_TEST_TMPDIR/.AGENTTRASH/20991231-tv-duplicates"
|
||||||
|
PLAN="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
mkdir -p "$LIB" "$HOLD"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the smaller copy of an episode is staged and the larger stays put" {
|
||||||
|
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/Doc Who - S01E01 - Pilot.mkv" 4000
|
||||||
|
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/backup/Doc.Who.S01E01.720p.mkv" 1200
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"backup/Doc.Who.S01E01.720p.mkv"* ]]
|
||||||
|
[[ "$output" != *"Pilot.mkv"* ]]
|
||||||
|
run destinations "$PLAN"
|
||||||
|
[[ "$output" == "$HOLD/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an equal-size pair stages exactly one copy" {
|
||||||
|
at "$LIB/Equal (2001) [tvdbid-11]/Season 01/Equal - S01E01.mkv" 2500
|
||||||
|
at "$LIB/Equal (2001) [tvdbid-11]/Rip/Equal - S01E01.mkv" 2500
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file holding one episode nothing else has is never staged" {
|
||||||
|
# The shape that defeats a folder-level verdict: Season 02 exists only here,
|
||||||
|
# so the folder that looks like a leftover is the only copy of an episode.
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 01/King - S01E01 - One.mkv" 3000
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 01/backup/King - S01E01 - One.mkv" 900
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 02/King - S02E01 - Two.mkv" 800
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" != *"S02E01"* ]]
|
||||||
|
[[ "$output" == *"backup/King - S01E01 - One.mkv"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a two-in-one pack competes for every episode in its range" {
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv" 1500
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 1500
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Packs/Range - S01E01-02.mkv" 2400
|
||||||
|
|
||||||
|
# The pack is the biggest carrier of BOTH episodes, so the two rips go.
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Season 01/Range - S01E01.mkv"* ]]
|
||||||
|
[[ "$output" != *"Packs/Range - S01E01-02.mkv"* ]]
|
||||||
|
|
||||||
|
# Make the season file the bigger carrier of E02 and the pack becomes the
|
||||||
|
# only carrier of E01: it stays, however small it is.
|
||||||
|
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv"
|
||||||
|
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv"
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 9000
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two tvdb ids on the same words are never merged" {
|
||||||
|
at "$LIB/Avatar The Last Airbender (2024) [tvdbid-385925]/Season 01/Avatar - S01E01.mkv" 9000
|
||||||
|
at "$LIB/Avatar The Last Airbender (1999) [tvdbid-74852]/Season 01/Avatar - S01E01.mkv" 500
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *CONFLICT* ]]
|
||||||
|
[[ "$output" == *"tvdb [74852] vs [385925]"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a derived id that contradicts the folder's own tag never merges them" {
|
||||||
|
# Measured 2026-11-02: Jellyfin's provider table carried a 2026 special's tvdb
|
||||||
|
# id on the folder tagged [tvdbid-73838]. Unioning the two makes one show of a
|
||||||
|
# 2000 series and a 2026 special, and the special's bigger files dominate the
|
||||||
|
# original's pilot. The tag is the operator's word; the nfo and `--ids` layers
|
||||||
|
# fill an untagged folder and never overrule a tagged one.
|
||||||
|
at "$LIB/Middle (2000) [tvdbid-73838]/Season 01/Middle - S01E01 - Pilot.mkv" 800
|
||||||
|
at "$LIB/Middle (2026) [tvdbid-457779]/Season 01/Middle - S01E01 - Episode 1.mkv" 1100
|
||||||
|
IDS="$BATS_TEST_TMPDIR/ids.tsv"
|
||||||
|
printf '%s\t%s\n' "Middle (2000) [tvdbid-73838]" 457779 > "$IDS"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --ids "$IDS"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"DISAGREE"* ]]
|
||||||
|
[[ "$output" == *"tag tvdb [73838]"* ]]
|
||||||
|
[[ "$output" == *"derived tvdb [457779]"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an untagged drop is identified by the tvshow.nfo in the metadata view" {
|
||||||
|
at "$LIB/Clone Wars/Season 01/Clone Wars - S01E01.mkv" 1200
|
||||||
|
at "$LIB/Clone Wars (2008) [tvdbid-84859]/Season 01/Clone Wars - S01E01.mkv" 4000
|
||||||
|
VIEW="$BATS_TEST_TMPDIR/view"
|
||||||
|
mkdir -p "$VIEW/Clone Wars"
|
||||||
|
printf '%s\n' '<tvshow><title>Clone Wars</title>' \
|
||||||
|
'<uniqueid type="tvdb" default="true">84859</uniqueid></tvshow>' \
|
||||||
|
> "$VIEW/Clone Wars/tvshow.nfo"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --meta "$VIEW"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Clone Wars/Season 01/Clone Wars - S01E01.mkv"* ]]
|
||||||
|
|
||||||
|
# Without the nfo layer nothing identifies the drop, so nothing moves.
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/no-meta.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *UNRESOLVED* ]]
|
||||||
|
[ "$(plan_moves "$BATS_TEST_TMPDIR/no-meta.tsv")" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file named by an id is keyed only when the witness names it" {
|
||||||
|
BIG="$LIB/Edd (1999) [tvdbid-75897]/Season 01/Edd - S01E01 - Down the Line.mkv"
|
||||||
|
SMALL="$LIB/Edd (1999) [tvdbid-75897]/Season 01/104348.mkv"
|
||||||
|
at "$BIG" 2100
|
||||||
|
at "$SMALL" 700
|
||||||
|
WIT="$BATS_TEST_TMPDIR/witness.tsv"
|
||||||
|
printf '%s\t1\t1\n' "$SMALL" > "$WIT"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"unkeyed, never staged: 1 files"* ]]
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --witness "$WIT"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *104348.mkv* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the sidecars of a staged episode are staged with it" {
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.mkv" 3000
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.mkv" 1000
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.srt" 400
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.srt" 450
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Rip/Side - S01E01 - First.srt"* ]]
|
||||||
|
[[ "$output" != *"Season 01/Side - S01E01 - First.srt"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a hold inside the library is refused" {
|
||||||
|
at "$LIB/Hostage (2007) [tvdbid-51]/Season 01/Hostage - S01E01.mkv" 100
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$LIB/.AGENTTRASH/hold" \
|
||||||
|
--out "$PLAN"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"--hold"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a plan whose hold target is occupied is refused, plan and all" {
|
||||||
|
at "$LIB/Gone (2009) [tvdbid-61]/Season 01/Gone - S01E01.mkv" 1000
|
||||||
|
at "$LIB/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
|
||||||
|
# The hold already holds one of the destinations: the executor would refuse
|
||||||
|
# it, so a plan that cannot run must not be handed over at all.
|
||||||
|
mkdir -p "$HOLD/Gone (2009) [tvdbid-61]/Rip"
|
||||||
|
at "$HOLD/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||||
|
rm "$PLAN"
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[[ "$output" == *"hold target occupied"* ]]
|
||||||
|
[ ! -e "$PLAN" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a library with nothing duplicated writes an empty plan" {
|
||||||
|
at "$LIB/Unique (2015) [tvdbid-71]/Season 01/Unique - S01E01.mkv" 1000
|
||||||
|
at "$LIB/Other Show (2016) [tvdbid-72]/Season 01/Other Show - S01E01.mkv" 1000
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"plan: 0 moves"* ]]
|
||||||
|
}
|
||||||
Reference in new issue
Block a user