Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fab17129e2 | ||
|
|
f46bf60939 | ||
|
|
b3eb9c6314 | ||
|
|
a3ddb4c65a | ||
|
|
b5173033b6 | ||
|
|
53e4a777b4 | ||
|
|
52d6da4f49 | ||
|
|
d138788411 | ||
|
|
6c169fc4ad | ||
|
|
e6a85176c4 | ||
|
|
9ff7ce6a12 | ||
|
|
f25be876fb | ||
|
|
bad78079d7 | ||
|
|
fee8121d47 | ||
|
|
1b5c11fb60 | ||
|
|
cae283047f | ||
|
|
51007ac604 | ||
|
|
f7c41966b0 | ||
|
|
25114ca622 | ||
|
|
a07f492b89 | ||
|
|
9961ed62c0 | ||
|
|
70be24e9d5 | ||
|
|
70aa40a24a | ||
|
|
8f2c3bb378 | ||
|
|
146c28da9f | ||
|
|
b82655d30a | ||
|
|
0161ab2aa8 | ||
|
|
085a5d570c | ||
|
|
9ca69d4e7e | ||
|
|
64b06b9426 | ||
|
|
e519f6b554 | ||
|
|
1e0247f29e | ||
|
|
71e7e94b94 | ||
|
|
24d793eb7d | ||
|
|
95375683d5 | ||
|
|
e7d7c6eea2 | ||
|
|
e1bf1d6e80 | ||
|
|
ef306397a5 | ||
|
|
3896dce364 | ||
|
|
3f82c4d5d8 | ||
|
|
5030f83c55 | ||
|
|
b875256500 | ||
|
|
51abc6550d | ||
|
|
64f489023e | ||
|
|
3511e04ae9 | ||
|
|
21b7ad15d9 | ||
|
|
4ad55c28a0 | ||
|
|
c9a8469d04 | ||
|
|
5794b07c32 | ||
|
|
1a47b831ee | ||
|
|
5be66df2bb | ||
|
|
d3aedd1837 | ||
|
|
66766f23ef | ||
|
|
5cf5416471 | ||
|
|
30186d8365 | ||
|
|
3a6f7c7280 | ||
|
|
c7643fb4d0 | ||
|
|
882a9f6808 | ||
|
|
aa57c1a894 | ||
|
|
2a48751283 | ||
|
|
6083903ccb | ||
|
|
ee9eb353fb | ||
|
|
df597bb1b1 | ||
|
|
23ad0ca317 | ||
|
|
71d5804eaf | ||
|
|
5ef141eb17 | ||
|
|
fadbe10420 | ||
|
|
17ea2f4110 | ||
|
|
603513a15e | ||
|
|
5e0592381a | ||
|
|
ef89f77ea7 | ||
|
|
9e2584ec8b | ||
|
|
a5dfc1ee99 | ||
|
|
7e72126a07 | ||
|
|
80a65b013f | ||
|
|
54ec72da17 | ||
|
|
d82b8efdd7 | ||
|
|
27b3c24065 | ||
|
|
7e9bfbc2be | ||
|
|
f5410bd253 | ||
|
|
98522cecba | ||
|
|
3138e782bd | ||
|
|
393b77dfd2 | ||
|
|
cfdd3435bc | ||
|
|
f3a6b4f715 | ||
|
|
5c888ff948 | ||
|
|
03e0d84380 | ||
|
|
a5603d8f01 | ||
|
|
988c4899d4 | ||
|
|
77e9973ff5 | ||
|
|
625263dd48 | ||
|
|
42bc482ed8 | ||
|
|
aea24353ad | ||
|
|
d1493de31a | ||
|
|
93725f42b3 | ||
|
|
f80148c5db | ||
|
|
541067e6b2 | ||
|
|
8f0b1982c2 | ||
|
|
a405c5ed35 | ||
|
|
b232033b11 | ||
|
|
4a33898a6c | ||
|
|
21c98bd485 | ||
|
|
a651c81dfb | ||
|
|
0951d909c9 | ||
|
|
69effb542d | ||
|
|
e9314ea1ee | ||
|
|
ad9563412d | ||
|
|
f71c7ccce4 | ||
|
|
f541566714 | ||
|
|
006bbcef28 | ||
|
|
8889a4ceff | ||
|
|
56dfc6d0b9 | ||
|
|
f1ae1ddb36 | ||
|
|
c21c48e61e | ||
|
|
22cd17d064 | ||
|
|
48ef6d34cd | ||
|
|
317f36b456 | ||
|
|
7a8c6bd2e3 | ||
|
|
3c035ccef3 | ||
|
|
a6ec644ecb | ||
|
|
b7f4b7e148 | ||
|
|
ed1a20a960 | ||
|
|
511d8dca93 | ||
|
|
29aefc3b6f | ||
|
|
2e69215920 | ||
|
|
b93e8481f9 | ||
|
|
51ed5c0b2a | ||
|
|
9f9043c037 | ||
|
|
a25f139281 | ||
|
|
52600b28ab | ||
|
|
7cf388367e | ||
|
|
13830865f6 | ||
|
|
010b9affc7 | ||
|
|
ed5eed6d54 | ||
|
|
14f340370c | ||
|
|
50e0c5a786 | ||
|
|
649bbafa17 | ||
|
|
b7c19592e5 | ||
|
|
6c115c1394 | ||
|
|
6d79709cc0 | ||
|
|
0336f42e2a | ||
|
|
0f4a527337 | ||
|
|
1d92627935 |
No files matched your search
@@ -227,9 +227,28 @@ See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
|
||||
|
||||
## External Exposure Policy (MANDATORY)
|
||||
|
||||
Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
|
||||
Authentik embedded outpost) or explicitly internal-only — no public DNS name,
|
||||
no vhost. New services default to internal-only until guarded.
|
||||
Every Caddy-proxied service is in exactly one of three postures: behind the
|
||||
auth chain (Anubis :8923 → Authentik embedded outpost), **exempt by
|
||||
construction** because it authenticates itself (`auth.`, git smart-HTTP, Gitea,
|
||||
OpenWebUI, Jellyfin, Plex, Audiobookshelf, the *arr stack), or internal-only —
|
||||
served to nobody off-LAN. New services default to internal-only until guarded.
|
||||
|
||||
**Internal-only is a source gate, not a missing DNS record.** Caddy answers
|
||||
every Host it knows, and its per-name certificates are published in Certificate
|
||||
Transparency logs, so a name with no Porkbun A record is still in a stranger's
|
||||
reach (`crt.sh` lists it). A household-only service therefore carries an edge
|
||||
gate — `@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8
|
||||
172.16.0.0/12`, `handle @lan { reverse_proxy … }`, `handle { respond 404 }` —
|
||||
which is what the *arr stack and qBittorrent carry (2026-11-04: Frank's law,
|
||||
"never leaving the home boxes"). A future household-only vhost gets the same
|
||||
block, not merely an absent record.
|
||||
|
||||
The guard chain is a **browser-only device**: Anubis demands a JavaScript
|
||||
proof-of-work and the outpost works from a session cookie, so any non-browser
|
||||
client — a native app, a podcast RSS reader, an OPDS client, `git`, a
|
||||
service-to-service call — is answered with challenge HTML where it expected
|
||||
JSON or a socket payload. A service that must serve such a client cannot sit
|
||||
behind the chain; it has to own its authentication.
|
||||
|
||||
When guarding a new vhost — provider first, then route:
|
||||
|
||||
@@ -248,6 +267,10 @@ When guarding a new vhost — provider first, then route:
|
||||
Exempt categories (document the reason in the vhost comment): the IdP itself
|
||||
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.
|
||||
|
||||
Unguarding a vhost is the mirror image of the procedure above — the route flips
|
||||
first and the IdP rows go last — and `serverAppConfigs-docs/Infrastructure/Caddy.md`
|
||||
*Unguard a vhost* owns the steps.
|
||||
|
||||
## Deletion & Trash Policy (MANDATORY)
|
||||
|
||||
Never `rm`. Move deletions into trash so they stay recoverable:
|
||||
@@ -265,7 +288,10 @@ archive.
|
||||
**Swap the model. Do not instrument it.** A payload change is: edit the trio
|
||||
(env / serve script / unit) → `scripts/deploy.sh <host> --service <svc>` →
|
||||
`/health` answers → one chat turn, one tool call, and one vision probe on an
|
||||
endpoint that serves images. That is the entire procedure.
|
||||
endpoint that serves images → re-key that endpoint's OpenWebUI `model` row
|
||||
(`meta.description` + `params.context.length`, API door in [[OpenWebUI]]). That is
|
||||
the entire procedure — no deployer step touches `webui.db`, and a stale
|
||||
`context.length` makes OpenWebUI advertise a window the endpoint does not grant.
|
||||
|
||||
- **No benchmarking unless Frank orders it.** No pre-swap baseline battery,
|
||||
no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers
|
||||
@@ -476,8 +502,8 @@ disagrees with it is the bug.
|
||||
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
|
||||
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
|
||||
`stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
|
||||
- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
|
||||
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
|
||||
- Shape: TP=2 over `wht6`, `--max-num-seqs 4`, `--max-model-len 200000`,
|
||||
`expert_tiered` placement, 49,152 MiB pinned host pool per rank, `fp8` KV,
|
||||
MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
|
||||
tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
|
||||
by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
|
||||
@@ -735,7 +761,12 @@ linked runbooks.
|
||||
**Ops**
|
||||
|
||||
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
||||
Use `pkill -x <exact-name>`.
|
||||
Use `pkill -x <exact-name>`, or the bracket trick — `pkill -f "serve[-]script.sh"` —
|
||||
when the pattern has to be named: the bracket makes the pattern stop matching the
|
||||
command line that carries it. Quote it: a zsh remote glob-expands an unquoted `[s]`
|
||||
and the kill silently never runs. Re-hit 2026-11-02 —
|
||||
`ssh host 'pkill -f jfshowswatch.py; …'` killed the remote shell mid-command, and
|
||||
ssh exits **255 with zero output**, which reads exactly like a network fault.
|
||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
||||
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
||||
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
||||
@@ -818,6 +849,7 @@ per-attempt page and no per-swap ledger anywhere else.
|
||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||
| Duplicate films and the purge hold (Dankserver) | `serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md` |
|
||||
|
||||
There is no page for a retired endpoint and no change diary. A payload swap
|
||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||
@@ -860,6 +892,6 @@ history.
|
||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
|
||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Deployed on Dankserver 2026-10-31, re-verified 2026-11-01: both quadlets bind the view, not the raw array. A restart for any reason discards that server's in-flight library scan, so re-trigger the refresh — see [[Media Metadata View]].** |
|
||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||
@@ -22,13 +22,16 @@
|
||||
# CHALLENGE everyone else) → Caddy's internal listener http://:8016 (auth
|
||||
# decision; Host headers survive every hop) → authenticated traffic is
|
||||
# forwarded back to http://:8017 and routed straight to the app.
|
||||
# Apps with their own auth (Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) and the
|
||||
# Authentik IdP itself (auth.franksdank.cloud — its login flow must never sit
|
||||
# behind a challenge) are exempt by construction: their vhosts never route
|
||||
# through Anubis. ComfyUI left this exempt set on 2026-08-31 and Grafana left
|
||||
# it on 2026-09-01 (OAuth2 relying-party mode against authentik): both now sit
|
||||
# behind the guard chain; Grafana authenticates users itself via the OAuth2
|
||||
# code flow, so no identity header is forwarded to it.
|
||||
# Apps with their own auth (Gitea, OpenWebUI, Jellyfin, Plex, Audiobookshelf,
|
||||
# *arr, Pi-hole) and the Authentik IdP itself (auth.franksdank.cloud — its login
|
||||
# flow must never sit behind a challenge) are exempt by construction: their
|
||||
# vhosts never route through Anubis. The chain is browser-only by nature — a
|
||||
# native app client, a podcast RSS reader or a git client solves no JavaScript
|
||||
# proof, so anything that must serve one has to own its own authentication.
|
||||
# ComfyUI left this exempt set on 2026-08-31 and Grafana left it on 2026-09-01
|
||||
# (OAuth2 relying-party mode against authentik): both now sit behind the guard
|
||||
# chain; Grafana authenticates users itself via the OAuth2 code flow, so no
|
||||
# identity header is forwarded to it.
|
||||
# The pihole vhost keeps its basicauth edge; the hash placeholder below is
|
||||
# rendered by deploy.sh from CADDY_BASICAUTH_HASH (host .env, git-ignored) into
|
||||
# /home/dank/Containers/caddy/Caddyfile, which the container mounts.
|
||||
@@ -264,20 +267,67 @@ files.franksdank.cloud {
|
||||
}
|
||||
}
|
||||
|
||||
# --- Jellyfin ---
|
||||
# --- Audiobookshelf (exempt by construction: own auth) --------------------
|
||||
# Caddy → app: no Anubis, no outpost. The guard chain answers a native client's
|
||||
# /ping, /api/login and /socket.io/ handshake with challenge HTML, so behind it
|
||||
# the phone app parsed a non-object where the socket `init` payload belongs and
|
||||
# podcast RSS/OPDS feeds broke the same way. ABS authenticates itself (local
|
||||
# accounts plus authentik OIDC), so it joins jelly/plex/*arr outside the chain
|
||||
# (2026-11-03); the IdP rows that guarded this name — proxy provider
|
||||
# `audiobooks-proxy` and application "Audiobookshelf Guard" — are retired with
|
||||
# the change, after the route flips. Accepted exposure: /audiobookshelf/api/login
|
||||
# is internet-visible with no rate limiter on either side (this Caddy build has
|
||||
# no rate_limit module; ABS 2.37.2 ships no lockout) — the same class every other
|
||||
# own-auth vhost here lives with, so shared accounts need long unique passwords.
|
||||
# ROUTER_BASE_PATH=/audiobookshelf, so the bare root is a convenience hop onto it.
|
||||
# X-Forwarded-Proto must say https: ABS derives its OIDC origin from
|
||||
# x-forwarded-proto://Host and this hop is plain http, so without the pin its own
|
||||
# https callback is a 400 "Invalid callback URL - must be same-origin".
|
||||
audiobooks.franksdank.cloud {
|
||||
redir / /audiobookshelf/ permanent
|
||||
reverse_proxy 192.168.0.235:8000 {
|
||||
header_up X-Forwarded-Proto https
|
||||
}
|
||||
}
|
||||
|
||||
# --- Jellyfin (exempt by construction: own auth) --------------------------
|
||||
# No Anubis, no outpost: Jellyfin carries its own complete login (its own
|
||||
# account store, no Authentik link), which is the policy's exempt category —
|
||||
# the same family as plex/ABS/*arr around it. Accepted exposure: the sign-in
|
||||
# form is internet-visible with no rate limiter on either side (this Caddy
|
||||
# build ships no rate_limit module), so accounts need long unique passwords.
|
||||
# The homepage card opened it to the household on 2026-11-03; accounts are
|
||||
# created in Jellyfin's own admin UI, not by SSO.
|
||||
jelly.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.235:8096
|
||||
}
|
||||
|
||||
|
||||
# --- *arr Stack (exempt by construction: own auth) ---
|
||||
|
||||
# --- *arr Stack + qBittorrent: LAN-ONLY at the edge (2026-11-04) -----------
|
||||
# Frank's law: these never leave the home boxes. DNS secrecy is NOT a gate —
|
||||
# Caddy issues one certificate per name and every one is in public CT logs
|
||||
# (crt.sh lists prowlarr/radarr/lidarr/readarr/qbittorrent.franksdank.cloud),
|
||||
# so a stranger can send the Host. The gate is the source address: a LAN client
|
||||
# proxies through, anything off-LAN gets 404. `remote_ip` reads the real client
|
||||
# address because the router DNATs the forward (no proxy hop in between).
|
||||
prowlarr.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.112:9696
|
||||
@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
|
||||
handle @lan {
|
||||
reverse_proxy 192.168.0.112:9696
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
}
|
||||
|
||||
radarr.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.112:7878
|
||||
@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
|
||||
handle @lan {
|
||||
reverse_proxy 192.168.0.112:7878
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
}
|
||||
|
||||
# --- Sonarr: internal-only by construction (retired 2026-09-04) -----------
|
||||
@@ -288,15 +338,33 @@ radarr.franksdank.cloud {
|
||||
# zero-auth UI to the internet via manual DNS — retired.
|
||||
|
||||
lidarr.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.112:8686
|
||||
@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
|
||||
handle @lan {
|
||||
reverse_proxy 192.168.0.112:8686
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
}
|
||||
|
||||
readarr.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.112:8787
|
||||
@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
|
||||
handle @lan {
|
||||
reverse_proxy 192.168.0.112:8787
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
}
|
||||
|
||||
qbittorrent.franksdank.cloud {
|
||||
reverse_proxy 192.168.0.112:8080
|
||||
@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
|
||||
handle @lan {
|
||||
reverse_proxy 192.168.0.112:8080
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
}
|
||||
|
||||
# --- Internal listener: auth decision layer ---------------------------------
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
192.168.0.143 romm.franksdank.cloud
|
||||
192.168.0.143 files.franksdank.cloud
|
||||
192.168.0.143 jelly.franksdank.cloud
|
||||
192.168.0.143 audiobooks.franksdank.cloud
|
||||
192.168.0.143 radarr.franksdank.cloud
|
||||
192.168.0.143 lidarr.franksdank.cloud
|
||||
192.168.0.143 readarr.franksdank.cloud
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# Quadlet unit: Audiobookshelf audiobook server (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/audiobookshelf.container
|
||||
# Unit name: audiobookshelf.service (podman 5.x drops the container- prefix)
|
||||
#
|
||||
# Everything this server writes is its own state — the SQLite library, accounts,
|
||||
# playback progress, cached covers — and it lives under
|
||||
# /home/dank/Containers/audiobookshelf on the NVMe, so no metadata byte ever
|
||||
# lands on the spindle. The audio itself is bound READ-ONLY off the array: the
|
||||
# only sanctioned write path into /mnt/entropy is the overlay view
|
||||
# (scripts/media-view.sh), Audiobookshelf has no need of it, and a read-only bind
|
||||
# makes a stray sidecar write structurally impossible rather than discouraged.
|
||||
#
|
||||
# The digest is the linux/amd64 manifest of the tag, which is what podman
|
||||
# verifies and what RepoDigests records on this single-arch host. Pinning the
|
||||
# manifest-list digest instead would let a re-pull resolve to a different image.
|
||||
# The upstream version tags stop at 2.19.0 (built 2025-02-02) while `latest` is
|
||||
# newer by a year and a half, so `latest@digest` is the only pin that names the
|
||||
# build actually running — same reasoning as the jellyfin and romm units.
|
||||
|
||||
[Unit]
|
||||
Description=Audiobookshelf audiobook server container
|
||||
After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=audiobookshelf
|
||||
Image=ghcr.io/advplyr/audiobookshelf:latest@sha256:37c2057e150694970442579815c646fe6bdc01c95b9d1547c4c9068be8844ef3
|
||||
Network=pasta
|
||||
# The names are the image's own: the server reads PORT, CONFIG_PATH, METADATA_PATH
|
||||
# and ROUTER_BASE_PATH (measured against the digest below — `podman inspect` shows
|
||||
# PORT=80 CONFIG_PATH=/config METADATA_PATH=/metadata in the image ENV, and the
|
||||
# server's source greps for exactly these). The ABS_* family printed by the server
|
||||
# README is the npm package's naming and is INERT here: with ABS_PORT=8000 the
|
||||
# server listened on :80 under base path /audiobookshelf, so the unit sat
|
||||
# `active (running)` while every probe to :8000 was refused (measured 2026-11-02,
|
||||
# first deploy).
|
||||
# ROUTER_BASE_PATH MUST be /audiobookshelf: the SPA in /app/client/dist/index.html is
|
||||
# built with `routerBasePath:"/audiobookshelf"` baked in, so it asks for its assets under
|
||||
# that prefix no matter what the server is mounted at. With `/` the server answers the
|
||||
# page but every script request falls through to the SPA fallback and comes back as
|
||||
# text/html — Chrome refuses to execute it and the UI never leaves "Loading…"
|
||||
# (measured 2026-11-03; the API and /healthcheck were fine, which is why an API-only
|
||||
# verification missed it).
|
||||
Environment=PORT=8000
|
||||
Environment=ROUTER_BASE_PATH=/audiobookshelf
|
||||
Environment=CONFIG_PATH=/config
|
||||
Environment=METADATA_PATH=/metadata
|
||||
Environment=TZ=America/Vancouver
|
||||
PublishPort=8000:8000
|
||||
Volume=/home/dank/Containers/audiobookshelf/config:/config
|
||||
Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata
|
||||
Volume=/mnt/entropy/Audiobooks:/audiobooks:ro
|
||||
# The image is Alpine: it ships wget and no curl, so the health check must name
|
||||
# the binary that exists or the container is unhealthy while serving perfectly.
|
||||
# /healthcheck answers `OK` at the root AND under the base path (measured on both
|
||||
# base paths), so the probe below is correct whatever ROUTER_BASE_PATH says.
|
||||
HealthCmd=wget -q -O /dev/null http://127.0.0.1:8000/healthcheck
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
|
||||
[Service]
|
||||
# Refuse to start until the array is a real mount — with the bind source missing,
|
||||
# podman invents an empty directory and Audiobookshelf records an empty library as
|
||||
# truth. See the header of scripts/wait-for-mount.sh for the blind-boot incident.
|
||||
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /mnt/entropy
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
# The first boot scans the whole audiobook tree. Weighted LOW (default 100) so a
|
||||
# scan loses to playback when the array is contended. Proportional share, not a
|
||||
# cap: inert while the disk is idle.
|
||||
IOWeight=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -47,6 +47,12 @@ Volume=/home/dank/Containers/jellyfin/cache:/cache
|
||||
# is the guard: started against a plain, unmounted view directory a scan decides
|
||||
# the library is empty and deletes what it cannot see.
|
||||
Volume=/var/lib/media-meta/view:/data
|
||||
# Big-screen theme for the TV. The repo file replaces the container's "Media
|
||||
# Center" theme slot; the upstream dark palette stays live because the file
|
||||
# @imports it, so an image bump cannot strand a vendored CSS snapshot. Read-only
|
||||
# — the web app only fetches it. A client opts in with
|
||||
# Display Settings -> Theme -> "Media Center".
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/jellyfin/theme/wmc/theme.css:/jellyfin/jellyfin-web/themes/wmc/theme.css:ro
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
HealthCmd=curl -f http://localhost:8096/health
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
/*
|
||||
* Dank fleet big-screen theme for jellyfin-web.
|
||||
*
|
||||
* Deployed by bind-mounting THIS file over the container's "Media Center"
|
||||
* theme (`/jellyfin/jellyfin-web/themes/wmc/theme.css`) — see the Volume line
|
||||
* in dankserver/jellyfin/quadlet/jellyfin.container. The slot is deliberately
|
||||
* one nobody picks by accident, and its name is the only thing that changes:
|
||||
* the colors are the live upstream dark theme, pulled in by @import below, so
|
||||
* this file never vendors a snapshot of upstream CSS and cannot drift when the
|
||||
* Jellyfin image moves.
|
||||
*
|
||||
* Why it exists: the living-room client (Jellyfin Desktop on danklivingroom)
|
||||
* renders this server-served web app at 3840x2160 with no client-side scaling,
|
||||
* so 14 px body text lands on a big panel viewed from three metres. `zoom` on
|
||||
* the root element is the one lever that scales the whole UI — the layout code
|
||||
* measures layout px, so card grids re-flow to fewer, larger columns instead of
|
||||
* overflowing.
|
||||
*
|
||||
* Adoption is per device, not global: Display Settings -> Theme ->
|
||||
* "Media Center" on the screen that wants it. That is the point — a phone
|
||||
* browser on the same server must not get a 1.75x interface.
|
||||
*
|
||||
* Change the size by editing the single number. 1.5 is mild, 1.75 is the
|
||||
* three-metre default, 2.0 is for a projector.
|
||||
*/
|
||||
|
||||
@import url("../dark/theme.css");
|
||||
|
||||
:root {
|
||||
/* Marker the e2e gate reads back; also a tweak hook for a per-client
|
||||
override injected by anyone who really wants one. */
|
||||
--dank-big-screen: 1;
|
||||
--dank-big-screen-zoom: 1.75;
|
||||
}
|
||||
|
||||
html {
|
||||
zoom: var(--dank-big-screen-zoom);
|
||||
}
|
||||
|
||||
/* A remote moves focus in discrete jumps, so the ring has to survive a
|
||||
three-metre read. */
|
||||
.emby-focus,
|
||||
button:focus-visible,
|
||||
a:focus-visible,
|
||||
.inputElement:focus-visible {
|
||||
outline: 3px solid var(--accentColor, #00a4dc) !important;
|
||||
outline-offset: 2px !important;
|
||||
}
|
||||
|
||||
/* Scrollbars are mouse furniture; on a TV they are noise at this size. */
|
||||
* {
|
||||
scrollbar-width: none;
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env python3
|
||||
"""abs-post-reorg.py — drive the Audiobookshelf side of the 2026-11-04 relayout.
|
||||
|
||||
Runs ON Dankserver (the ABS host). Credentials come from the machine-local
|
||||
file ~/.config/secrets/audiobookshelf and never enter argv; requests go
|
||||
through urllib so neither the password nor the bearer token is world-readable
|
||||
in /proc. The delete arm is the DB-row arm ONLY — `hard=1` is fs.remove() of
|
||||
the item folder and must never reach this file (see the Audiobookshelf vault
|
||||
page; after the move the covering paths are whole books).
|
||||
|
||||
Subcommands
|
||||
covers --list LIST.tsv copy each ABS /metadata cover onto the book
|
||||
folder ("srchost<TAB>dstarray" per line);
|
||||
run BEFORE the mover so the cover rides the
|
||||
book's move, or AFTER with dst already final
|
||||
wipe [--apply] delete every library item DB row (progress
|
||||
included — the campaign accepted losing the
|
||||
11 progress rows); refuses without --apply
|
||||
scan [--wait] trigger the library scan; --wait polls
|
||||
isFetchingLibrary then prints the item total
|
||||
status item total + isMissing census
|
||||
|
||||
Paging law (measured twice): page on `page` (0-based) + `limit`; `offset` is
|
||||
ignored outright. Count unique ids; stop on `total`.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
BASE = "http://127.0.0.1:8000/audiobookshelf"
|
||||
SECRET = os.environ.get("ABS_SECRET", "/home/dank/.config/secrets/audiobookshelf")
|
||||
|
||||
|
||||
def cred():
|
||||
kv = {}
|
||||
with open(SECRET) as fh:
|
||||
for line in fh:
|
||||
if "=" in line:
|
||||
k, _, v = line.strip().partition("=")
|
||||
kv[k] = v
|
||||
return kv.get("ABS_ADMIN_USER", "frank"), kv.get("ABS_ADMIN_PASSWORD")
|
||||
|
||||
|
||||
def call(method, path, body=None, token=None, base=BASE):
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
req = urllib.request.Request(base + path, data=data, method=method)
|
||||
req.add_header("Content-Type", "application/json")
|
||||
if token:
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
raw = r.read().decode()
|
||||
try:
|
||||
return r.status, json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
return r.status, raw.strip()
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read().decode()[:200]
|
||||
|
||||
|
||||
def login():
|
||||
user, password = cred()
|
||||
if not password:
|
||||
sys.exit(f"refused: {SECRET} carries no ABS_ADMIN_PASSWORD")
|
||||
code, body = call("POST", "/login", {"username": user, "password": password})
|
||||
if code != 200:
|
||||
sys.exit(f"login failed {code}: {body}")
|
||||
return body["user"]["token"] # the token is inside user, not top level
|
||||
|
||||
|
||||
def library_id(token, name="Audiobooks"):
|
||||
code, libs = call("GET", "/api/libraries", token=token)
|
||||
if code != 200:
|
||||
sys.exit(f"libraries failed {code}: {libs}")
|
||||
for lib in libs.get("libraries", []):
|
||||
if lib["name"] == name:
|
||||
return lib["id"]
|
||||
sys.exit(f"no library named {name}")
|
||||
|
||||
|
||||
def all_item_ids(token, lid):
|
||||
ids, page = [], 0
|
||||
code, one = call("GET", f"/api/libraries/{lid}/items?limit=1", token=token)
|
||||
total = one.get("total", -1) if code == 200 else -1
|
||||
while True:
|
||||
code, body = call("GET", f"/api/libraries/{lid}/items?page={page}&limit=500",
|
||||
token=token)
|
||||
if code != 200:
|
||||
sys.exit(f"items page {page} failed {code}")
|
||||
results = body.get("results", [])
|
||||
if not results:
|
||||
break
|
||||
ids += [it["id"] for it in results]
|
||||
page += 1
|
||||
if total >= 0 and page * 500 >= total:
|
||||
break
|
||||
return list(dict.fromkeys(ids)), total
|
||||
|
||||
|
||||
def cmd_covers(args):
|
||||
copied = missing = exists = 0
|
||||
for line in open(args.list):
|
||||
src, _, dst = line.rstrip("\n").partition("\t")
|
||||
if not dst:
|
||||
continue
|
||||
if os.path.exists(dst):
|
||||
exists += 1
|
||||
continue
|
||||
if not os.path.exists(src):
|
||||
missing += 1
|
||||
continue
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
shutil.copy2(src, dst)
|
||||
copied += 1
|
||||
print(f"covers copied: {copied} already there: {exists} missing source: {missing}")
|
||||
|
||||
|
||||
def cmd_wipe(args, token, lid):
|
||||
ids, total = all_item_ids(token, lid)
|
||||
print(f"library {lid}: {len(ids)} items (total={total})")
|
||||
if not args.apply:
|
||||
print("dry run: nothing deleted (add --apply)")
|
||||
return
|
||||
gone = 0
|
||||
for i, iid in enumerate(ids):
|
||||
code, _ = call("DELETE", f"/api/items/{iid}", token=token)
|
||||
if code == 200:
|
||||
gone += 1
|
||||
elif i < 3:
|
||||
print(f"delete {iid}: {code}")
|
||||
if i % 500 == 499:
|
||||
print(f" {i + 1}/{len(ids)} deleted")
|
||||
print(f"deleted {gone}/{len(ids)} item rows")
|
||||
|
||||
|
||||
def cmd_scan(args, token, lid):
|
||||
code, body = call("POST", f"/api/libraries/{lid}/scan", token=token)
|
||||
print(f"scan: {code} {body if code != 200 else 'accepted'}")
|
||||
if not args.wait:
|
||||
return
|
||||
last = -1
|
||||
while True:
|
||||
time.sleep(30)
|
||||
code, lib = call("GET", f"/api/libraries/{lid}", token=token)
|
||||
code2, one = call("GET", f"/api/libraries/{lid}/items?limit=1", token=token)
|
||||
total = one.get("total", -1) if code2 == 200 else -1
|
||||
fetching = lib.get("isFetchingLibrary") if code == 200 else None
|
||||
print(f" fetching={fetching} items={total}")
|
||||
if fetching is False and total == last and total >= 0:
|
||||
break
|
||||
last = total
|
||||
print(f"scan settled at {total} items")
|
||||
|
||||
|
||||
def cmd_status(token, lid):
|
||||
missing = 0
|
||||
seen, page = set(), 0
|
||||
while True:
|
||||
code, body = call("GET", f"/api/libraries/{lid}/items?page={page}&limit=500",
|
||||
token=token)
|
||||
results = body.get("results", []) if code == 200 else []
|
||||
if not results:
|
||||
break
|
||||
for it in results:
|
||||
if it["id"] not in seen:
|
||||
seen.add(it["id"])
|
||||
if it.get("isMissing"):
|
||||
missing += 1
|
||||
page += 1
|
||||
code, lib = call("GET", f"/api/libraries/{lid}", token=token)
|
||||
print(f"items: {len(seen)} isMissing: {missing} "
|
||||
f"pending tasks: {lib.get('pendingTasks') if code == 200 else '?'}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
c = sub.add_parser("covers")
|
||||
c.add_argument("--list", required=True)
|
||||
w = sub.add_parser("wipe")
|
||||
w.add_argument("--apply", action="store_true")
|
||||
s = sub.add_parser("scan")
|
||||
s.add_argument("--wait", action="store_true")
|
||||
sub.add_parser("status")
|
||||
args = ap.parse_args()
|
||||
token = login()
|
||||
lid = library_id(token)
|
||||
if args.cmd == "covers":
|
||||
cmd_covers(args)
|
||||
elif args.cmd == "wipe":
|
||||
cmd_wipe(args, token, lid)
|
||||
elif args.cmd == "scan":
|
||||
cmd_scan(args, token, lid)
|
||||
else:
|
||||
cmd_status(token, lid)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,311 @@
|
||||
#!/usr/bin/env bash
|
||||
# audiobookshelf-init.sh — make an Audiobookshelf server usable, from nothing.
|
||||
#
|
||||
# Why this is a script and not a browser session (live 2026-11-02, Dankserver):
|
||||
# a freshly deployed Audiobookshelf answers /healthcheck with 200 while it has no
|
||||
# user, no library and no scan — "healthy" says nothing about usefulness, and a
|
||||
# service that only works after someone remembers to click through a first-run
|
||||
# wizard is not deployed, it is installed. Everything below is the API the React
|
||||
# client itself calls, so the same procedure runs unattended and lands in git.
|
||||
#
|
||||
# The API facts this script depends on, read off the pinned image rather than
|
||||
# guessed (dist-server/server/Server.js inside the digest this repo pins):
|
||||
# * the creation route is POST /init with {"newRoot":{"username","password"}}.
|
||||
# There is NO /setup route in 2.37.x — POST /setup answers "Unauthorized",
|
||||
# which reads exactly like an auth fault and is instead a missing route.
|
||||
# * /status (no /api prefix) reports {"isInit": <has a root user>}; every route
|
||||
# under /api needs the bearer token that POST /login returns.
|
||||
# * the library count is NOT a field on the library DTO; the door is
|
||||
# GET /api/libraries/{id}/items -> {"total": N}.
|
||||
#
|
||||
# The credential is generated once into a mode-600 host file and never enters
|
||||
# the repo, the URL, or a process argument list. Same pattern as calibre-server.
|
||||
#
|
||||
# Single sign-on (authentik) is armed by the same run, and only when the credential
|
||||
# file carries ABS_OIDC_CLIENT_SECRET, so a local-only deployment is unchanged by a
|
||||
# re-run. These OIDC facts came from the same pinned image, and each one is a trap:
|
||||
# * isOpenIDAuthSettingsValid demands issuer + authorization + token + userinfo +
|
||||
# jwks + clientID + clientSecret + signingAlgorithm at once. Miss one and the
|
||||
# server SILENTLY strips 'openid' from authActiveAuthMethods and falls back to
|
||||
# ['local'] while answering 200 everywhere — so the only honest check is what
|
||||
# /status says AFTER the write, which is what this script does.
|
||||
# * the algorithm is HS256, not the image's RS256 default: no provider on the
|
||||
# fleet's authentik carries a signing key, so the client secret IS the signature
|
||||
# key (scripts/authentik-audiobookshelf-sso.sh). Left at RS256, every id_token
|
||||
# fails verification and the failure looks like a broken IdP.
|
||||
# * the auth block lives on PATCH /api/auth-settings. PATCH /api/settings has a
|
||||
# 20-key allowlist (PATCHABLE_SETTINGS_KEYS) with NO auth key in it, so writing
|
||||
# the OIDC config there answers 200 and changes nothing — measured on this host
|
||||
# 2026-11-02. The keys /api/auth-settings accepts are exactly its own
|
||||
# authenticationSettings getters; anything else is skipped with a debug line.
|
||||
# * authOpenIDSubfolderForRedirectURLs must be sent explicitly. The 2.17.4 migration only
|
||||
# fills it on a DB that ALREADY had OIDC on, so a database created before SSO leaves it
|
||||
# UNDEFINED and the server builds redirect_uri
|
||||
# "http://host/undefined/auth/openid/callback" — authentik answers the authorize request
|
||||
# with "Redirect URI Error" (measured 2026-11-02). This script derives the value from
|
||||
# ABS_URL's path, so it cannot drift from the unit's ROUTER_BASE_PATH.
|
||||
# * 'local' stays in authActiveAuthMethods on purpose: if authentik is ever down,
|
||||
# the root account must still be able to sign into its own server.
|
||||
#
|
||||
# usage: audiobookshelf-init.sh # create what is missing, then scan
|
||||
# ABS_SCAN=0 audiobookshelf-init.sh # bootstrap without a scan
|
||||
# ABS_OIDC=0 audiobookshelf-init.sh # never touch the auth methods
|
||||
#
|
||||
# environment (defaults are the deployed shape):
|
||||
# ABS_URL server base URL INCLUDING the mounted base path
|
||||
# (http://127.0.0.1:8000/audiobookshelf)
|
||||
# ABS_SECRET credential file (/home/dank/.config/secrets/audiobookshelf)
|
||||
# ABS_LIBRARY library name to ensure (Audiobooks)
|
||||
# ABS_FOLDER folder to bind inside the server (/audiobooks)
|
||||
# ABS_SCAN 1 triggers a scan (default), 0 skips it
|
||||
# ABS_OIDC auto (default) arms SSO when the credential file has a client
|
||||
# secret, 1 requires one, 0 leaves the auth methods alone
|
||||
# ABS_OIDC_ISSUER authentik issuer URL for this client
|
||||
# ABS_OIDC_CLIENT_ID provider client id (audiobookshelf)
|
||||
# ABS_OIDC_ALG id_token signing algorithm the IdP actually uses (HS256)
|
||||
# ABS_OIDC_SUBFOLDER override for the redirect-URI subfolder (default: ABS_URL's path)
|
||||
#
|
||||
# exit 0 every step is satisfied (created or already present)
|
||||
# exit 1 a step was refused or the server misbehaved (named on stderr)
|
||||
# exit 2 usage / unreadable credential file
|
||||
set -euo pipefail
|
||||
|
||||
ABS_URL=${ABS_URL:-http://127.0.0.1:8000/audiobookshelf}
|
||||
ABS_SECRET=${ABS_SECRET:-/home/dank/.config/secrets/audiobookshelf}
|
||||
ABS_LIBRARY=${ABS_LIBRARY:-Audiobooks}
|
||||
ABS_FOLDER=${ABS_FOLDER:-/audiobooks}
|
||||
ABS_SCAN=${ABS_SCAN:-1}
|
||||
ABS_OIDC=${ABS_OIDC:-auto}
|
||||
ABS_OIDC_ISSUER=${ABS_OIDC_ISSUER:-https://auth.franksdank.cloud/application/o/audiobookshelf/}
|
||||
ABS_OIDC_CLIENT_ID=${ABS_OIDC_CLIENT_ID:-audiobookshelf}
|
||||
ABS_OIDC_ALG=${ABS_OIDC_ALG:-HS256}
|
||||
|
||||
die() { printf 'refused: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
case "$ABS_SCAN" in
|
||||
0 | 1) ;;
|
||||
*) printf 'refused: ABS_SCAN wants 0 or 1, got %s\n' "$ABS_SCAN" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
case "$ABS_OIDC" in
|
||||
auto | 0 | 1) ;;
|
||||
*) printf 'refused: ABS_OIDC wants auto, 0 or 1, got %s\n' "$ABS_OIDC" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
# ---- credential -------------------------------------------------------------
|
||||
umask 077
|
||||
if [ ! -f "$ABS_SECRET" ]; then
|
||||
mkdir -p "$(dirname "$ABS_SECRET")"
|
||||
user=${ABS_ADMIN_USER:-frank}
|
||||
# `tr … | head -c 24` is the obvious line and it is a trap under
|
||||
# `set -o pipefail`: head exits at 24 bytes, tr takes SIGPIPE, the pipeline
|
||||
# status is 141 and set -e kills the script before the credential is ever
|
||||
# written (measured 2026-11-02). od reads a FIXED byte count and exits on its
|
||||
# own, so no consumer closes a pipe early; bash slices the length.
|
||||
pass=$(od -An -tx1 -N 32 /dev/urandom | tr -d ' \n')
|
||||
pass=${pass:0:24}
|
||||
[ "${#pass}" -eq 24 ] || die "could not generate a password (no /dev/urandom)"
|
||||
printf 'ABS_ADMIN_USER=%s\nABS_ADMIN_PASSWORD=%s\n' "$user" "$pass" > "$ABS_SECRET"
|
||||
printf 'credential: created %s\n' "$ABS_SECRET"
|
||||
fi
|
||||
[ -r "$ABS_SECRET" ] || die "$ABS_SECRET exists but is not readable by $(id -un)"
|
||||
chmod 600 "$ABS_SECRET"
|
||||
|
||||
# ---- everything else, over HTTP ---------------------------------------------
|
||||
# python3 rather than curl so the password and the bearer token never appear in
|
||||
# any argv (they would be world-readable in /proc/<pid>/cmdline otherwise).
|
||||
ABS_URL="$ABS_URL" ABS_SECRET="$ABS_SECRET" ABS_LIBRARY="$ABS_LIBRARY" \
|
||||
ABS_FOLDER="$ABS_FOLDER" ABS_SCAN="$ABS_SCAN" ABS_OIDC="$ABS_OIDC" \
|
||||
ABS_OIDC_ISSUER="$ABS_OIDC_ISSUER" ABS_OIDC_CLIENT_ID="$ABS_OIDC_CLIENT_ID" \
|
||||
ABS_OIDC_ALG="$ABS_OIDC_ALG" python3 - <<'PY'
|
||||
import json, os, sys, urllib.error, urllib.parse, urllib.request
|
||||
|
||||
BASE = os.environ["ABS_URL"].rstrip("/")
|
||||
LIB = os.environ["ABS_LIBRARY"]
|
||||
FOLDER = os.environ["ABS_FOLDER"]
|
||||
SCAN = os.environ["ABS_SCAN"] == "1"
|
||||
cred = {}
|
||||
for line in open(os.environ["ABS_SECRET"]):
|
||||
if "=" in line:
|
||||
k, _, v = line.strip().partition("=")
|
||||
cred[k] = v
|
||||
user = cred.get("ABS_ADMIN_USER") or "frank"
|
||||
password = cred.get("ABS_ADMIN_PASSWORD") or ""
|
||||
if not password:
|
||||
sys.exit("refused: %s has no ABS_ADMIN_PASSWORD" % os.environ["ABS_SECRET"])
|
||||
|
||||
|
||||
def call(method, path, body=None, token=None, want_json=True):
|
||||
data = None
|
||||
headers = {"X-Requested-With": "audiobookshelf-init"}
|
||||
if body is not None:
|
||||
data = json.dumps(body).encode()
|
||||
headers["Content-Type"] = "application/json"
|
||||
if token:
|
||||
headers["Authorization"] = "Bearer " + token
|
||||
req = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
raw = r.read()
|
||||
if not raw or not want_json:
|
||||
return r.status, raw
|
||||
try:
|
||||
return r.status, json.loads(raw)
|
||||
except ValueError:
|
||||
# Express answers sendStatus() with the STATUS TEXT as the body:
|
||||
# POST /init and the scan route both reply "OK" (measured on the
|
||||
# pinned 2.37.2 image). A successful call must never be judged by
|
||||
# whether its body happens to parse as JSON.
|
||||
return r.status, raw.decode("utf-8", "replace")[:200]
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read()
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except Exception:
|
||||
payload = raw.decode("utf-8", "replace")[:200]
|
||||
return e.code, payload
|
||||
except urllib.error.URLError as e:
|
||||
# Not an HTTP answer at all: nothing listening, DNS dead, connection
|
||||
# refused. HTTPError is a URLError subclass, so this arm must come after.
|
||||
return 0, "unreachable (%s)" % e.reason
|
||||
|
||||
|
||||
status, st = call("GET", "/status")
|
||||
if status != 200 or not isinstance(st, dict):
|
||||
sys.exit("refused: %s/status answered %s (%s) — is the service up?" % (BASE, status, st))
|
||||
if st.get("isInit"):
|
||||
print("account: already registered")
|
||||
else:
|
||||
code, body = call("POST", "/init", {"newRoot": {"username": user, "password": password}})
|
||||
if code != 200:
|
||||
sys.exit("refused: POST /init answered %s (%s)" % (code, body))
|
||||
print("account: created (%s)" % user)
|
||||
|
||||
code, login = call("POST", "/login", {"username": user, "password": password})
|
||||
if code != 200 or not isinstance(login, dict) or not login.get("user", {}).get("token"):
|
||||
sys.exit("refused: login answered %s (%s) — the credential file does not match the server"
|
||||
% (code, login))
|
||||
token = login["user"]["token"]
|
||||
print("token: obtained (%s)" % login["user"].get("type", "user"))
|
||||
|
||||
code, libs = call("GET", "/api/libraries", token=token)
|
||||
if code != 200 or not isinstance(libs, dict) or "libraries" not in libs:
|
||||
sys.exit("refused: GET /api/libraries answered %s (%s)" % (code, libs))
|
||||
lib = next((l for l in libs["libraries"] if l.get("name") == LIB), None)
|
||||
if lib:
|
||||
print("library: exists %s (%s)" % (LIB, lib["id"]))
|
||||
else:
|
||||
body = {
|
||||
"name": LIB,
|
||||
"provider": "audiobook",
|
||||
"icon": "book",
|
||||
"folders": [{"fullPath": FOLDER}],
|
||||
"settings": {
|
||||
"coverEncoding": "jpg",
|
||||
"skipMatchingMediaWithAsin": False,
|
||||
"skipMatchingMediaWithIsbn": False,
|
||||
"autoDetectCrossoverAudiobooks": False,
|
||||
"ignoreAudiobookMetadata": False,
|
||||
"useAudiobookFileMetadata": False,
|
||||
},
|
||||
"providerMetadata": {},
|
||||
}
|
||||
code, made = call("POST", "/api/libraries", body, token=token)
|
||||
if code != 200:
|
||||
sys.exit("refused: POST /api/libraries answered %s (%s)" % (code, made))
|
||||
lib = made
|
||||
print("library: created %s (%s) folder %s" % (LIB, lib["id"], FOLDER))
|
||||
|
||||
if SCAN:
|
||||
code, body = call("POST", "/api/libraries/%s/scan" % lib["id"], {}, token=token,
|
||||
want_json=False)
|
||||
if code != 200:
|
||||
sys.exit("refused: scan answered %s (%s)" % (code, body))
|
||||
print("scan: accepted")
|
||||
else:
|
||||
print("scan: skipped (ABS_SCAN=0)")
|
||||
|
||||
code, items = call("GET", "/api/libraries/%s/items?limit=1" % lib["id"], token=token)
|
||||
if code == 200 and isinstance(items, dict):
|
||||
print("items: %s (a fresh scan raises this from 0; watch it, not /healthcheck)"
|
||||
% items.get("total"))
|
||||
|
||||
# ---- single sign-on (authentik) --------------------------------------------------
|
||||
# The provider this points at is provisioned by scripts/authentik-audiobookshelf-sso.sh;
|
||||
# the two share exactly one value, the client secret in the credential file.
|
||||
mode = os.environ["ABS_OIDC"]
|
||||
oidc_secret = cred.get("ABS_OIDC_CLIENT_SECRET") or ""
|
||||
if mode == "0":
|
||||
print("oidc: skipped (ABS_OIDC=0)")
|
||||
elif not oidc_secret:
|
||||
if mode == "1":
|
||||
sys.exit("refused: ABS_OIDC=1 but %s has no ABS_OIDC_CLIENT_SECRET — run "
|
||||
"scripts/authentik-audiobookshelf-sso.sh first" % os.environ["ABS_SECRET"])
|
||||
print("oidc: not configured (no ABS_OIDC_CLIENT_SECRET in the credential file)"
|
||||
" — local auth only")
|
||||
else:
|
||||
client_id = os.environ["ABS_OIDC_CLIENT_ID"]
|
||||
issuer = os.environ["ABS_OIDC_ISSUER"].rstrip("/") + "/"
|
||||
# redirect_uri = <origin><subfolder>/auth/openid/callback, and the subfolder IS the
|
||||
# base path the app is mounted under — taken from ABS_URL so unit and config agree.
|
||||
subfolder = os.environ.get("ABS_OIDC_SUBFOLDER")
|
||||
if subfolder is None:
|
||||
subfolder = urllib.parse.urlparse(os.environ["ABS_URL"]).path.rstrip("/")
|
||||
origin = issuer.split("/application/")[0]
|
||||
body = {
|
||||
"authActiveAuthMethods": ["local", "openid"],
|
||||
"authOpenIDIssuerURL": issuer,
|
||||
"authOpenIDAuthorizationURL": origin + "/application/o/authorize/",
|
||||
"authOpenIDTokenURL": origin + "/application/o/token/",
|
||||
"authOpenIDUserInfoURL": origin + "/application/o/userinfo/",
|
||||
"authOpenIDJwksURL": issuer + "jwks/",
|
||||
"authOpenIDLogoutURL": issuer + "end-session/",
|
||||
"authOpenIDClientID": client_id,
|
||||
"authOpenIDClientSecret": oidc_secret,
|
||||
"authOpenIDTokenSigningAlgorithm": os.environ["ABS_OIDC_ALG"],
|
||||
"authOpenIDSubfolderForRedirectURLs": subfolder,
|
||||
"authOpenIDButtonText": "Sign in with Authentik",
|
||||
# not auto-launch: the login page keeps offering the local form, which is the
|
||||
# break-glass path when the IdP is unreachable.
|
||||
"authOpenIDAutoLaunch": False,
|
||||
# a household user's first SSO sign-in creates their account. The role comes from
|
||||
# the abs_groups claim, which grants the listener role to every signer; only the
|
||||
# authentik admin group escalates. Revoking a listener therefore happens HERE in
|
||||
# ABS (disable or delete the user), never by removing an IdP group membership.
|
||||
"authOpenIDAutoRegister": True,
|
||||
"authOpenIDGroupClaim": "abs_groups",
|
||||
# The app's own scheme lives HERE, never in the IdP: ABS swaps it for
|
||||
# <base>/auth/openid/mobile-redirect before the provider sees a redirect_uri.
|
||||
"authOpenIDMobileRedirectURIs": ["audiobookshelf://oauth"],
|
||||
}
|
||||
code, res = call("PATCH", "/api/auth-settings", body, token=token)
|
||||
if code != 200:
|
||||
sys.exit("refused: PATCH /api/auth-settings answered %s (%s)" % (code, res))
|
||||
# Verify twice: what was stored, and what the server will honour. A config that
|
||||
# misses any of the eight required OIDC values is accepted with 200 and then
|
||||
# silently dropped at the next load, leaving authMethods=['local'].
|
||||
code, stored = call("GET", "/api/auth-settings", token=token)
|
||||
if code != 200 or not isinstance(stored, dict):
|
||||
sys.exit("refused: GET /api/auth-settings answered %s (%s)" % (code, stored))
|
||||
for key, want in (("authOpenIDIssuerURL", issuer),
|
||||
("authOpenIDTokenSigningAlgorithm", os.environ["ABS_OIDC_ALG"]),
|
||||
("authOpenIDSubfolderForRedirectURLs", subfolder),
|
||||
("authOpenIDButtonText", "Sign in with Authentik")):
|
||||
if stored.get(key) != want:
|
||||
sys.exit("refused: stored %s=%r, expected %r" % (key, stored.get(key), want))
|
||||
if sorted(stored.get("authActiveAuthMethods") or []) != ["local", "openid"]:
|
||||
sys.exit("refused: stored authActiveAuthMethods=%r, expected local+openid "
|
||||
"('openid' is dropped when the block is incomplete)"
|
||||
% stored.get("authActiveAuthMethods"))
|
||||
code, after = call("GET", "/status")
|
||||
methods = after.get("authMethods") if isinstance(after, dict) else None
|
||||
if not isinstance(methods, list) or "openid" not in methods:
|
||||
sys.exit("refused: /status reports authMethods=%s after a stored config that "
|
||||
"looks complete — restart the service, the strategy is only armed at "
|
||||
"load for a config that was invalid when the server read it" % methods)
|
||||
print("oidc: armed (issuer %s | alg %s | claim abs_groups | redirect subfolder %r | "
|
||||
"client %s | secret %s chars | authMethods %s)"
|
||||
% (issuer, os.environ["ABS_OIDC_ALG"], subfolder, client_id, len(oidc_secret),
|
||||
methods))
|
||||
PY
|
||||
exit 0
|
||||
@@ -0,0 +1,294 @@
|
||||
#!/usr/bin/env bash
|
||||
# authentik-audiobookshelf-sso.sh — provision the authentik side of Audiobookshelf SSO.
|
||||
#
|
||||
# Why a script: this state lives in authentik's DATABASE, not in any repo file, and a
|
||||
# blueprint re-import or a rebuild-from-scratch reverts hand-made provider rows
|
||||
# silently. Same reasoning (and same door) as
|
||||
# authentik-set-login-session-duration.sh: ssh into Dankclaw, `ak shell` from stdin.
|
||||
# Re-run this after such a rebuild.
|
||||
#
|
||||
# The contract with Audiobookshelf, read off the pinned image (2.37.2,
|
||||
# dist-server/server/auth/OidcAuthStrategy.js + objects/settings/ServerSettings.js)
|
||||
# rather than a forum, because each of these is a dead-login trap:
|
||||
#
|
||||
# * Two URIs per origin ABS is reached at — <base>/auth/openid/callback and
|
||||
# <base>/auth/openid/mobile-redirect — for EVERY entry in BASES (the LAN address
|
||||
# and the public vhost). ABS SWAPS the app's own audiobookshelf://oauth for the
|
||||
# mobile-redirect URL before the IdP ever sees a redirect_uri, so registering the
|
||||
# app scheme with authentik — the obvious move — is what makes phone logins fail.
|
||||
# The app scheme belongs in ABS's authOpenIDMobileRedirectURIs list (its built-in
|
||||
# default), not here. The mobile URI exists on every origin because the app sends
|
||||
# the one belonging to the door it was configured with: the vhost started serving
|
||||
# non-browser clients on 2026-11-03 (it left the guard chain), so a phone pointed
|
||||
# at the public name sends the public mobile-redirect URI, and STRICT matching
|
||||
# answers an unregistered one with "Redirect URI Error".
|
||||
# * <base> CARRIES THE BASE PATH. ABS builds redirect_uri as
|
||||
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback, and that subfolder is
|
||||
# the ROUTER_BASE_PATH the container is mounted at — /audiobookshelf, which is also the only
|
||||
# base path the shipped SPA can boot under. Left unset (the fresh-DB default) the URL
|
||||
# becomes "/undefined/…"; a provider registered without the prefix mismatches the same way
|
||||
# (measured 2026-11-02, "Redirect URI Error"). audiobookshelf-init.sh derives its copy from
|
||||
# ABS_URL, so the two sides each move with one variable.
|
||||
# * ABS requests the scopes "openid profile email <GroupClaim> [<PermsClaim>]": the
|
||||
# CLAIM NAME is a scope name. So authentik needs a scope mapping whose scope_name
|
||||
# is exactly the claim name, or the claim never arrives and the callback dies on
|
||||
# "Group claim <name> not found in userinfo".
|
||||
# * the claim's VALUES must be ABS role names — the callback picks the first of
|
||||
# ['admin','user','guest'] present in the list. Group names like
|
||||
# audiobookshelf_admins are therefore translated by the mapping, never passed
|
||||
# through. A user in neither group gets [] and is denied (fail closed).
|
||||
# * tokens are HS256 here: no provider on this box carries a signing_key (census
|
||||
# 2026-11-02), which makes the client secret the signature key. ABS's own default
|
||||
# is RS256, so the ABS side MUST be set to HS256 or every id_token fails
|
||||
# verification.
|
||||
# * provider.authorization_flow must be non-NULL or /application/o/authorize/
|
||||
# answers a 500 with 'NoneType' object has no attribute 'slug' (measured on
|
||||
# grafana, same authentik build).
|
||||
# * provider.grant_types MUST contain authorization_code. authentik 2026.8 defaults an
|
||||
# ORM-created provider to the EMPTY list, and /application/o/authorize/ then refuses the
|
||||
# code flow with oauthlib's generic "The request is otherwise malformed"; the only line
|
||||
# that names the cause is the server log's 'Invalid grant_type for provider'. The
|
||||
# redirect-URI check runs FIRST, so a provider carrying both faults shows only the URI
|
||||
# error until the URI is fixed (measured 2026-11-02, where the login survived the URI fix
|
||||
# and died here).
|
||||
# * property_mappings must carry the built-in OpenID scope mappings ('openid', 'profile',
|
||||
# 'email') alongside the claim mapping. A provider holding only the claim mapping grants a
|
||||
# token scoped shorter than the scopes ABS requests, and ABS's /userinfo call then dies
|
||||
# 'insufficient_scope (The request requires higher privileges than provided by the access
|
||||
# token)' after authentik has already issued a good code (measured 2026-11-02).
|
||||
#
|
||||
# Idempotent: every object is created or asserted. Safe to re-run.
|
||||
#
|
||||
# usage: ABS_OIDC_CLIENT_SECRET=<secret> authentik-audiobookshelf-sso.sh
|
||||
# (the secret is the value stored in
|
||||
# /home/dank/.config/secrets/audiobookshelf on Dankserver — get it with
|
||||
# ssh dankserver awk -F= '/^ABS_OIDC_CLIENT_SECRET=/{print $2}' \
|
||||
# ~/.config/secrets/audiobookshelf)
|
||||
#
|
||||
# environment:
|
||||
# ABS_OIDC_CLIENT_SECRET REQUIRED. The provider's client secret.
|
||||
# ABS_OIDC_CLIENT_ID provider/client id (audiobookshelf)
|
||||
# ABS_OIDC_GROUPS_CLAIM claim ABS reads for roles (abs_groups)
|
||||
# ABS_OIDC_USERNAME authentik user made admin (akadmin)
|
||||
# There is deliberately NO user list. The claim grants the LISTENER role to every
|
||||
# authentik account that signs in (2026-11-03), so onboarding a listener is exactly
|
||||
# "create the authentik account". Before this, a household account authenticated at
|
||||
# the IdP and was then refused by ABS for being in no group — a policy decision that
|
||||
# read as a broken login. Revocation lives on the ABS side (disable or delete the
|
||||
# user there): a group can no longer withhold a role the claim always sends.
|
||||
# ABS_E2E_ADMIN authentik user kept in the ADMIN group for the Playwright gate
|
||||
# (qa-e2e-admin — tests/e2e/specs/23-audiobookshelf-sso.spec.ts
|
||||
# signs in as this identity, so a re-run restores the fixture)
|
||||
# ABS_BASE ABS URL as the browser sees it, base path included
|
||||
# (http://192.168.0.235:8000/audiobookshelf — the LAN
|
||||
# origin, reached directly, which is also what the phone
|
||||
# app is configured with)
|
||||
# ABS_EXTRA_BASES space-separated additional origins, base path included
|
||||
# (default: https://audiobooks.franksdank.cloud/audiobookshelf
|
||||
# — the guarded vhost, dankclaw/Caddyfile).
|
||||
# ABS builds its redirect_uri from the ORIGIN of the
|
||||
# request, so each origin the app is served at needs its
|
||||
# own registered URI — an unregistered origin is a
|
||||
# redirect_uri_mismatch at the IdP, before any credential.
|
||||
# Set it empty to register the LAN origin alone.
|
||||
# AUTHENTIK_HOST ssh target (dankclaw)
|
||||
set -euo pipefail
|
||||
|
||||
SECRET="${ABS_OIDC_CLIENT_SECRET:?set ABS_OIDC_CLIENT_SECRET (see the header)}"
|
||||
[ "${#SECRET}" -ge 32 ] || { echo "refused: ABS_OIDC_CLIENT_SECRET is too short to be a secret" >&2; exit 2; }
|
||||
|
||||
CLIENT_ID="${ABS_OIDC_CLIENT_ID:-audiobookshelf}"
|
||||
CLAIM="${ABS_OIDC_GROUPS_CLAIM:-abs_groups}"
|
||||
ADMIN_GROUP="audiobookshelf_admins"
|
||||
SSO_USER="${ABS_OIDC_USERNAME:-akadmin}"
|
||||
E2E_ADMIN="${ABS_E2E_ADMIN:-qa-e2e-admin}"
|
||||
BASE="${ABS_BASE:-http://192.168.0.235:8000/audiobookshelf}"
|
||||
EXTRA_BASES="${ABS_EXTRA_BASES:-https://audiobooks.franksdank.cloud/audiobookshelf}"
|
||||
HOST="${AUTHENTIK_HOST:-dankclaw}"
|
||||
|
||||
# The claim expression: every signer is a listener; only the admin group escalates.
|
||||
# Stored verbatim so a later read of the provider shows exactly this policy. The default
|
||||
# lives HERE and nowhere else — ABS reads this list off the token, so an account in no
|
||||
# Audiobookshelf group still arrives with ["user"] and is auto-registered as a listener
|
||||
# instead of being refused after a successful authentication.
|
||||
read -r -d '' EXPR <<PYEXPR || true
|
||||
groups = {g.name for g in user.groups.all()}
|
||||
# Order is deliberate: an admin's claim comes out ["admin","user"], byte-identical to what
|
||||
# this mapping emitted before the default existed, because ABS's group-tag matching has never
|
||||
# been measured for order-sensitivity and the working admin login is not the place to find out.
|
||||
roles = []
|
||||
if "$ADMIN_GROUP" in groups:
|
||||
roles.append("admin")
|
||||
roles.append("user")
|
||||
return {"$CLAIM": roles}
|
||||
PYEXPR
|
||||
|
||||
# NOTE: the heredoc below is UNQUOTED on purpose — its body interpolates the variables
|
||||
# above. The consequence is that a backtick anywhere in that python body is a LOCAL command
|
||||
# substitution whose output is spliced into the program before it ever runs, and comments
|
||||
# are not exempt, because bash does not know what a python comment is. Write names plainly
|
||||
# in the body: measured 2026-11-03, two comments in this body had the shell run
|
||||
# audiobookshelf_users and user as commands on every execution of this script.
|
||||
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
|
||||
from authentik.core.models import Application, Group, User
|
||||
from authentik.flows.models import Flow
|
||||
from authentik.providers.oauth2.models import (
|
||||
ClientType, OAuth2Provider, RedirectURI, RedirectURIMatchingMode, RedirectURIType, ScopeMapping)
|
||||
|
||||
ADMIN_GROUP = "$ADMIN_GROUP"
|
||||
CLAIM, CLIENT_ID, SECRET, BASE = "$CLAIM", "$CLIENT_ID", "$SECRET", "$BASE"
|
||||
BASES = [BASE] + """$EXTRA_BASES""".split()
|
||||
SSO_USER, E2E_ADMIN = "$SSO_USER", "$E2E_ADMIN"
|
||||
EXPR = '''$EXPR'''
|
||||
# The expression reads user.groups, NOT user.ak_groups: authentik 2026.8 logs
|
||||
# "User.ak_groups is deprecated … use User.groups" on every evaluation of the old
|
||||
# name (measured on this run), and a stored mapping that leans on a deprecated
|
||||
# accessor is a future upgrade failure sitting inside the login path.
|
||||
|
||||
Group.objects.get_or_create(name=ADMIN_GROUP)
|
||||
admin_g = Group.objects.get(name=ADMIN_GROUP)
|
||||
# The audiobookshelf_users group is retired. Its membership used to be the only door to the
|
||||
# listener role; the claim now sends that role to every signer, so the group decides
|
||||
# nothing — and a group that decides nothing is a trap for the next reader, who will
|
||||
# assume it gates something and start adding people to it.
|
||||
for stale in Group.objects.filter(name="audiobookshelf_users"):
|
||||
stale.delete()
|
||||
print("group: retired audiobookshelf_users (the claim grants every signer)")
|
||||
|
||||
# --- the claim mapping (scope_name IS the claim name ABS requests) ---------------
|
||||
mapping = ScopeMapping.objects.filter(name="audiobookshelf-groups").first()
|
||||
if mapping is None:
|
||||
mapping = ScopeMapping.objects.create(name="audiobookshelf-groups", scope_name=CLAIM,
|
||||
expression=EXPR,
|
||||
description="Audiobookshelf roles: every signer is a listener, the admin group escalates")
|
||||
print("mapping: created (scope_name %s)" % CLAIM)
|
||||
elif mapping.scope_name != CLAIM or mapping.expression != EXPR:
|
||||
mapping.scope_name, mapping.expression = CLAIM, EXPR
|
||||
mapping.save(update_fields=["scope_name", "expression"])
|
||||
print("mapping: scope/expression updated (scope_name %s)" % CLAIM)
|
||||
else:
|
||||
print("mapping: already current")
|
||||
|
||||
# Prove the expression runs and yields an ABS role BEFORE anyone tries to log in:
|
||||
# authentik execs the expression inside a function whose context has the name "user".
|
||||
_src = "def _probe(user, request=None):\n" + "".join(" %s\n" % ln for ln in EXPR.splitlines())
|
||||
_ns = {}
|
||||
exec(_src, _ns)
|
||||
|
||||
# --- provider --------------------------------------------------------------------
|
||||
# One callback pair per origin (LAN address and the public vhost): ABS derives the
|
||||
# browser callback from the request origin and the mobile callback from the door the
|
||||
# app was configured with, so each origin needs both or that door cannot sign in.
|
||||
uris = [RedirectURI(matching_mode=RedirectURIMatchingMode.STRICT, url="%s%s" % (b, p))
|
||||
for b in BASES
|
||||
for p in ("/auth/openid/callback", "/auth/openid/mobile-redirect")]
|
||||
authz_flow = Flow.objects.get(slug="default-provider-authorization-implicit-consent")
|
||||
|
||||
# Three things beyond the name, each one a silent dead login: the grant-type list (2026.8
|
||||
# defaults it empty), the built-in OpenID scope mappings beside the claim mapping, and the
|
||||
# redirect URIs with the base path baked in.
|
||||
GRANTS = ["authorization_code", "refresh_token"]
|
||||
BUILTIN_SCOPES = ["authentik default OAuth Mapping: OpenID 'openid'",
|
||||
"authentik default OAuth Mapping: OpenID 'profile'",
|
||||
"authentik default OAuth Mapping: OpenID 'email'"]
|
||||
scoped = {m.pk: m for m in ScopeMapping.objects.filter(name__in=BUILTIN_SCOPES)}
|
||||
scoped[mapping.pk] = mapping
|
||||
_have = {m.name for m in scoped.values()}
|
||||
if len(_have) < len(BUILTIN_SCOPES):
|
||||
print("scope mappings: NOT PROVIDED by this authentik install: %s"
|
||||
% ", ".join(sorted(set(BUILTIN_SCOPES) - _have)))
|
||||
|
||||
p = OAuth2Provider.objects.filter(name="audiobookshelf").first()
|
||||
if p is None:
|
||||
p = OAuth2Provider(name="audiobookshelf", client_id=CLIENT_ID,
|
||||
client_type=ClientType.CONFIDENTIAL, client_secret=SECRET,
|
||||
authorization_flow=authz_flow, grant_types=list(GRANTS))
|
||||
p.redirect_uris = uris
|
||||
p.save()
|
||||
p.property_mappings.set(list(scoped.values()))
|
||||
print("provider: created (grant_types %s)" % GRANTS)
|
||||
else:
|
||||
changed = []
|
||||
if p.client_id != CLIENT_ID:
|
||||
p.client_id = CLIENT_ID
|
||||
changed.append("client_id")
|
||||
if p.client_secret != SECRET:
|
||||
p.client_secret = SECRET
|
||||
changed.append("client_secret")
|
||||
if p.authorization_flow_id != authz_flow.pk:
|
||||
p.authorization_flow = authz_flow
|
||||
changed.append("authorization_flow")
|
||||
if list(p.grant_types) != GRANTS:
|
||||
p.grant_types = list(GRANTS)
|
||||
changed.append("grant_types")
|
||||
if [(u.url, str(u.matching_mode)) for u in p.redirect_uris] != \
|
||||
[(u.url, str(u.matching_mode)) for u in uris]:
|
||||
p.redirect_uris = uris
|
||||
changed.append("redirect_uris")
|
||||
if changed:
|
||||
p.save()
|
||||
if sorted(m.pk for m in p.property_mappings.all()) != sorted(scoped):
|
||||
p.property_mappings.set(list(scoped.values()))
|
||||
changed.append("property_mappings")
|
||||
print("provider: asserted (%s)" % (", ".join(changed) or "nothing to change"))
|
||||
|
||||
# --- application (the launchpad entry; binds the provider) ------------------------
|
||||
app = Application.objects.filter(slug="audiobookshelf").first()
|
||||
if app is None:
|
||||
app = Application.objects.create(name="audiobookshelf", slug="audiobookshelf",
|
||||
provider=p, meta_launch_url=BASE)
|
||||
print("application: created")
|
||||
elif app.provider_id != p.pk or app.meta_launch_url != BASE:
|
||||
app.provider, app.meta_launch_url = p, BASE
|
||||
app.save(update_fields=["provider", "meta_launch_url"])
|
||||
print("application: asserted")
|
||||
else:
|
||||
print("application: already bound")
|
||||
|
||||
# --- membership --------------------------------------------------------------------
|
||||
root = User.objects.get(username=SSO_USER)
|
||||
root.groups.add(admin_g)
|
||||
# The deploy gate's Playwright spec signs in as this identity, so the admin fixture is
|
||||
# provisioned with the rest: a blueprint re-import that drops it fails the gate, not a phone.
|
||||
e2e = User.objects.filter(username=E2E_ADMIN).first()
|
||||
if e2e is None:
|
||||
print("e2e fixture: authentik user %s absent — the e2e SSO spec will fail" % E2E_ADMIN)
|
||||
else:
|
||||
e2e.groups.add(admin_g)
|
||||
print("e2e fixture: %s in %s" % (E2E_ADMIN, ADMIN_GROUP))
|
||||
print("admin: %s in %s" % (SSO_USER, ADMIN_GROUP))
|
||||
# The default is printed at provision time, not merely asserted: an expression that regressed
|
||||
# to returning nothing for a non-member refuses every household login, and this is where a
|
||||
# re-run notices it. The population matters — authentik ships an AnonymousUser row, a
|
||||
# superuser, and ak-outpost-* service identities, and proving the default on one of those says
|
||||
# nothing about the friend who logs in. Prefer the fleet's plain-user marker group (owned by
|
||||
# serverAppConfigs-docs/Infrastructure/Authentik.md; a preference only, since that group
|
||||
# belongs to other consumers), which normally lands on qa-e2e — the identity the e2e gate
|
||||
# signs in as. is_superuser is a python property on authentik's User, not a queryable field.
|
||||
pool = User.objects.filter(is_active=True).exclude(groups__name__in=[ADMIN_GROUP])
|
||||
pool = pool.exclude(username__in=[root.username, E2E_ADMIN, "AnonymousUser"])
|
||||
pool = [u for u in pool.order_by("username")
|
||||
if not u.is_superuser and not u.username.startswith("ak-outpost-")]
|
||||
household = [u for u in pool if any(g.name == "Non-Admin Users" for g in u.groups.all())]
|
||||
for u in (household or pool)[:3]:
|
||||
print("default probe: %s %s -> %s" % (
|
||||
u.username, sorted(g.name for g in u.groups.all()),
|
||||
_ns["_probe"](user=u, request=None)))
|
||||
|
||||
# --- verification: the claim the callback will actually receive ---------------------
|
||||
for probe in [root] + ([e2e] if e2e else []):
|
||||
print("claim for %s -> %s" % (probe.username, _ns["_probe"](user=probe, request=None)))
|
||||
p.refresh_from_db()
|
||||
print("provider %s | client_id %s | flow %s | grants %s | uris %s" % (
|
||||
p.name, p.client_id, p.authorization_flow.slug, list(p.grant_types),
|
||||
[u.url for u in p.redirect_uris]))
|
||||
print("scopes: %s" % sorted(m.scopemapping.scope_name for m in p.property_mappings.all()))
|
||||
PYEOF
|
||||
|
||||
cat <<DONE
|
||||
|
||||
Next: arm the Audiobookshelf side (issuer, client, HS256, claim) —
|
||||
ssh dankserver 'ABS_OIDC=1 bash /home/dank/serverAppConfigs/scripts/audiobookshelf-init.sh'
|
||||
It reads the same secret from /home/dank/.config/secrets/audiobookshelf and verifies
|
||||
/status answers openid among its auth methods.
|
||||
DONE
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env bash
|
||||
# Census and repair the OAuth2 defaults on every authentik PROXY provider.
|
||||
#
|
||||
# A proxy provider built by hand in `ak shell` never runs
|
||||
# ProxyProvider.set_oauth_defaults() — the REST serializer calls it, the ORM does
|
||||
# not — so the row looks perfectly configured (external_host, internal_host,
|
||||
# application bound, embedded-outpost claim present) while three OAuth facts are
|
||||
# blank. Each one kills the guarded login in a different, unrelated-looking way,
|
||||
# all BEFORE any credential is asked for:
|
||||
#
|
||||
# * redirect_uris empty -> authentik answers its own page:
|
||||
# "Redirect URI Error — The request fails due to a missing, invalid, or
|
||||
# mismatching redirection URI (redirect_uri)". Measured on the Audiobookshelf
|
||||
# guard provider, 2026-11-03.
|
||||
# * grant_types empty -> the authorize endpoint refuses the outpost's
|
||||
# authorization-code request with a bodiless HTTP 400 (Chromium renders
|
||||
# "This page isn't working / HTTP ERROR 400"). Same provider, one step
|
||||
# later, after the URIs were filled in by hand.
|
||||
# * authorization_flow NULL -> the authorize view crashes building its flow
|
||||
# planner: AttributeError 'NoneType' object has no attribute 'slug', HTTP
|
||||
# 500. That is what authentik-grafana-sso-fix.sh exists for.
|
||||
#
|
||||
# So: a guard that shows ANY of those three pages is not an Anubis problem and
|
||||
# not a Caddy problem — run this script before touching the route.
|
||||
#
|
||||
# Repair calls the VENDOR's own set_oauth_defaults() (grant_types = authz code +
|
||||
# client credentials + password, confidential client, the stock scope mappings,
|
||||
# redirect_uris derived from external_host by _get_callback_url) and points the
|
||||
# authorization flow at default-provider-authorization-implicit-consent, the flow
|
||||
# every working provider here uses. Providers that already match are untouched,
|
||||
# so a deliberate per-provider deviation is never silently flattened.
|
||||
#
|
||||
# This is authentik DB state. It is NOT in any deployable file: a blueprint
|
||||
# re-import, or an ak-shell re-creation of a provider, reverts these facts and
|
||||
# the symptom returns verbatim. Re-run after any such rebuild. Idempotent.
|
||||
#
|
||||
# Usage: authentik-proxy-provider-defaults.sh [--check]
|
||||
# no argument repair what drifts, print one line per provider
|
||||
# --check census only, exit 1 if anything drifts (gate use)
|
||||
# override host: AUTHENTIK_HOST=…
|
||||
set -euo pipefail
|
||||
HOST="${AUTHENTIK_HOST:-dankclaw}"
|
||||
MODE="${1:-}"
|
||||
case "$MODE" in
|
||||
"" | --check) : ;;
|
||||
*) echo "usage: $(basename "$0") [--check]" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
|
||||
from authentik.flows.models import Flow
|
||||
from authentik.providers.proxy.models import ProxyProvider, _get_callback_url
|
||||
|
||||
CHECK = $( [ "$MODE" = "--check" ] && echo True || echo False )
|
||||
FLOW = "default-provider-authorization-implicit-consent"
|
||||
WANT_GRANTS = ["authorization_code", "client_credentials", "password"]
|
||||
|
||||
flow = Flow.objects.filter(slug=FLOW).first()
|
||||
if not flow and not CHECK:
|
||||
raise SystemExit("flow %s is missing; nothing was changed" % FLOW)
|
||||
|
||||
drift = 0
|
||||
for p in ProxyProvider.objects.all().order_by("pk"):
|
||||
want_uris = [u.url for u in _get_callback_url(p.external_host)]
|
||||
have_uris = [u.url for u in p.redirect_uris]
|
||||
bad = []
|
||||
if [str(g) for g in p.grant_types] != WANT_GRANTS:
|
||||
bad.append("grant_types=%s" % [str(g) for g in p.grant_types])
|
||||
if have_uris != want_uris:
|
||||
bad.append("redirect_uris=%s" % have_uris)
|
||||
if p.authorization_flow_id is None:
|
||||
bad.append("authorization_flow=None")
|
||||
if not bad:
|
||||
print("ok %s (ext %s)" % (p.name, p.external_host))
|
||||
continue
|
||||
drift += 1
|
||||
if CHECK:
|
||||
print("DRIFT %s: %s" % (p.name, "; ".join(bad)))
|
||||
continue
|
||||
print("fix %s: %s" % (p.name, "; ".join(bad)))
|
||||
p.set_oauth_defaults()
|
||||
if p.authorization_flow_id is None:
|
||||
p.authorization_flow = flow
|
||||
p.save()
|
||||
after = ProxyProvider.objects.get(pk=p.pk)
|
||||
print(" -> grants %s, %d redirect URIs, flow %s"
|
||||
% ([str(g) for g in after.grant_types], len(after.redirect_uris),
|
||||
after.authorization_flow and after.authorization_flow.slug))
|
||||
|
||||
print("providers checked, drift: %d" % drift)
|
||||
raise SystemExit(1 if (CHECK and drift) else 0)
|
||||
PYEOF
|
||||
|
||||
echo "Verify: walk the guarded vhost in a browser — Anubis solves, authentik"
|
||||
echo "renders its login form (no Redirect URI Error, no HTTP 400), and the"
|
||||
echo "callback lands back on the app."
|
||||
@@ -60,6 +60,7 @@ SERVICE_ORDER = (
|
||||
"calibre-server",
|
||||
"plex",
|
||||
"tautulli",
|
||||
"audiobookshelf",
|
||||
"prowlarr",
|
||||
"qbittorrent",
|
||||
"radarr",
|
||||
@@ -123,6 +124,7 @@ DANKSERVER_SERVICES = {
|
||||
"calibre-server",
|
||||
"plex",
|
||||
"tautulli",
|
||||
"audiobookshelf",
|
||||
"prowlarr",
|
||||
"qbittorrent",
|
||||
"radarr",
|
||||
@@ -420,6 +422,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
"copyparty": {"copyparty"},
|
||||
"plex": {"plex"},
|
||||
"tautulli": {"tautulli"},
|
||||
"audiobookshelf": {"audiobookshelf"},
|
||||
}
|
||||
_add(services, *quadlet_services.get(parts[1], set()))
|
||||
if len(parts) >= 2 and parts[1] == "host":
|
||||
|
||||
@@ -95,6 +95,7 @@ service_is_allowed() {
|
||||
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:audiobookshelf|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||
@@ -322,11 +323,13 @@ copy_alloy_config() {
|
||||
# The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
|
||||
# carries the host's name, so Loki streams say WHICH machine's journal a
|
||||
# line came from without a second config file per host.
|
||||
# *_CONF_ROOT are test seams: the bats cases execute these copiers for
|
||||
# real against a sandbox with a stubbed run_privileged.
|
||||
# *_CONF_ROOT and ALLOY_CONFIG_SRC are test seams: the bats cases execute
|
||||
# these copiers for real against a sandbox with a stubbed run_privileged;
|
||||
# ALLOY_CONFIG_SRC lets the refusal case feed a deliberately broken config.
|
||||
local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
|
||||
local src="${ALLOY_CONFIG_SRC:-shared/alloy/config.alloy}"
|
||||
tmp="$(mktemp)"
|
||||
sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
|
||||
sed "s/GRAFANA_HOST_LABEL/$TARGET/g" "$src" > "$tmp"
|
||||
if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
|
||||
echo "alloy config: host-label substitution failed" >&2
|
||||
rm -f "$tmp"
|
||||
@@ -1212,6 +1215,10 @@ ensure_media_view() {
|
||||
# scripts/wait-for-mount.sh, which waits and then refuses the start. A
|
||||
# relayout must be followed by `media-view.sh reap`
|
||||
# (see the Media Relayout page), which this is not.
|
||||
# A fresh host has no upper base, and `install` refuses to invent one — so
|
||||
# without this the media deploy dies on a rebuilt machine with
|
||||
# "put it on the SSD first" (measured 2026-10-31).
|
||||
run_privileged bash scripts/media-view.sh ensure-base
|
||||
run_privileged bash scripts/media-view.sh install
|
||||
bash scripts/media-view.sh verify
|
||||
}
|
||||
@@ -1296,6 +1303,37 @@ deploy_calibre_web() {
|
||||
systemctl --user restart calibre-web.service
|
||||
}
|
||||
|
||||
# Audiobookshelf keeps its whole state (SQLite library, accounts, progress,
|
||||
# cached covers) on the NVMe under /home/dank/Containers/audiobookshelf; the
|
||||
# audio is a read-only bind off the array, so the only thing the deployer has to
|
||||
# create is the two state roots. mkdir names the owner — letting podman invent
|
||||
# the directories leaves the SQLite file unwritable by the container user on a
|
||||
# rootless first boot. No `systemctl --user enable`: a generated Quadlet unit
|
||||
# cannot be enabled, and WantedBy=default.target is what carries it across a
|
||||
# reboot.
|
||||
deploy_audiobookshelf() {
|
||||
cp "dankserver/audiobookshelf/quadlet/audiobookshelf.container" "$quadlet_unit_dir/"
|
||||
systemctl --user daemon-reload
|
||||
mkdir -p /home/dank/Containers/audiobookshelf/config \
|
||||
/home/dank/Containers/audiobookshelf/metadata
|
||||
systemctl --user restart audiobookshelf.service
|
||||
# /healthcheck answers once the HTTP server is bound, at the root whatever
|
||||
# ROUTER_BASE_PATH says (measured on both base paths); the library scan that
|
||||
# follows runs in the background and must not gate the deploy.
|
||||
wait_for_health audiobookshelf http://127.0.0.1:8000/healthcheck 90
|
||||
# A server with no root user and no library is "healthy" and useless, so the
|
||||
# bootstrapper runs after the health gate and its failure fails the deploy:
|
||||
# it creates the account (credential in the mode-600 host secret file), the
|
||||
# Audiobooks library over the /audiobooks bind, and fires the scan. Every
|
||||
# step is idempotent, so a re-deploy costs one incremental scan.
|
||||
# The bootstrapper has to speak the same base path the unit mounts (the API, the
|
||||
# login route and the OIDC redirect subfolder all hang off it), so the URL is read
|
||||
# out of the unit file here instead of being restated in the script's default.
|
||||
abs_base=$(awk -F= '/^Environment=ROUTER_BASE_PATH=/{print $3}' \
|
||||
dankserver/audiobookshelf/quadlet/audiobookshelf.container)
|
||||
ABS_URL="http://127.0.0.1:8000${abs_base}" scripts/audiobookshelf-init.sh
|
||||
}
|
||||
|
||||
# The content server's single admin account and the desktop's basic-auth
|
||||
# credential (same pair) are bootstrapped exactly once: the env file is the
|
||||
# plaintext store (mode 600), the userdb's pw column is plaintext too, so
|
||||
@@ -2172,6 +2210,7 @@ deploy_service() {
|
||||
copyparty) deploy_copyparty ;;
|
||||
plex) deploy_plex ;;
|
||||
tautulli) deploy_tautulli ;;
|
||||
audiobookshelf) deploy_audiobookshelf ;;
|
||||
alloy) deploy_alloy ;;
|
||||
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
||||
gpu-exporter) deploy_gpu_exporter ;;
|
||||
|
||||
@@ -148,6 +148,7 @@ valid_service() {
|
||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:audiobookshelf|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||
dankserver:prowlarr|dankserver:frank-archon|\
|
||||
|
||||
@@ -35,7 +35,8 @@
|
||||
# Subcommands
|
||||
# plan print the resolved shape and every pending change, touch nothing
|
||||
# install fetch+verify the plugin, then write the keys (backing up first)
|
||||
# verify prove the plugin and both keys are actually in place
|
||||
# verify prove the plugin and both keys are in place, and that no library
|
||||
# is a silent file index (providers off / image fetchers empty)
|
||||
#
|
||||
# Nothing here restarts Jellyfin: the deployer owns that
|
||||
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
||||
@@ -134,7 +135,18 @@ text = escape(val)
|
||||
if pat.search(src):
|
||||
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
|
||||
else:
|
||||
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
|
||||
# The plugin owns this file and writes it under ITS OWN root element
|
||||
# (TheTVDB's is <PluginConfiguration>), and its serializer omits a
|
||||
# null-valued setting entirely — so the element being filled can be
|
||||
# absent outright, and a hardcoded </Configuration> anchor then matches
|
||||
# nothing. Anchor on the document's real root instead.
|
||||
out = src
|
||||
root = re.search(r"<([A-Za-z_][\w.:-]*)[^>]*>", src)
|
||||
if root and ("</%s>" % root.group(1)) in src:
|
||||
name = root.group(1)
|
||||
out = src.replace(
|
||||
"</%s>" % name, f" <{elem}>{text}</{elem}>\n</{name}>", 1
|
||||
)
|
||||
if out == src:
|
||||
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
|
||||
|
||||
@@ -340,6 +352,64 @@ cmd_install() {
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
# A library whose options.xml carries EnableInternetProviders=false, or an empty
|
||||
# ImageFetchers list beside a populated MetadataFetchers list, is a SILENT FILE
|
||||
# INDEX: it indexes every file, answers 204 to every refresh, stores no provider
|
||||
# id, plot, rating or artwork, and logs nothing — so the item count reads like a
|
||||
# healthy scan. Jellyfin 12 keeps these options in root/default/<Library>/options.xml
|
||||
# (it was library.xml) and stores NONE of them in the database, so this audit is
|
||||
# the only place the state can be read without a browser session.
|
||||
verify_library_options() {
|
||||
local root="$JF_ROOT/root/default" f lib problems
|
||||
if [ ! -d "$root" ]; then
|
||||
printf '= no library options under %s (nothing audited)\n' "$root"
|
||||
return 0
|
||||
fi
|
||||
local rc=0
|
||||
for f in "$root"/*/options.xml; do
|
||||
[ -f "$f" ] || continue
|
||||
lib="$(basename "$(dirname "$f")")"
|
||||
problems="$(python3 -s - "$f" <<'PY'
|
||||
import sys, xml.dom.minidom as X
|
||||
root = X.parse(sys.argv[1]).documentElement
|
||||
def child(el, name):
|
||||
for n in el.childNodes:
|
||||
if n.nodeType == n.ELEMENT_NODE and n.tagName == name:
|
||||
return n
|
||||
def flag(el, name):
|
||||
n = child(el, name)
|
||||
return None if n is None else (n.firstChild.data.strip() if n.firstChild else "")
|
||||
def names(el, name):
|
||||
n = child(el, name)
|
||||
return None if n is None else [s.firstChild.data.strip() for s in n.childNodes
|
||||
if s.nodeType == s.ELEMENT_NODE and s.firstChild]
|
||||
bad = []
|
||||
ip = flag(root, "EnableInternetProviders")
|
||||
if ip is not None and ip.lower() != "true":
|
||||
bad.append("EnableInternetProviders=%s" % (ip or "empty"))
|
||||
types = child(root, "TypeOptions")
|
||||
if types is not None:
|
||||
for t in types.childNodes:
|
||||
if t.nodeType != t.ELEMENT_NODE or t.tagName != "TypeOptions":
|
||||
continue
|
||||
meta = names(t, "MetadataFetchers") or []
|
||||
if meta and not (names(t, "ImageFetchers") or []):
|
||||
bad.append("%s: ImageFetchers empty while MetadataFetchers names %d"
|
||||
% (flag(t, "Type") or "?", len(meta)))
|
||||
print("; ".join(bad))
|
||||
PY
|
||||
)"
|
||||
if [ -n "$problems" ]; then
|
||||
printf 'SILENT INDEX %s: %s\n' "$lib" "$problems" >&2
|
||||
rc=1
|
||||
else
|
||||
printf '= %s: library options fetch from providers\n' "$lib"
|
||||
fi
|
||||
done
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
cmd_verify() {
|
||||
check_root
|
||||
local bad=0
|
||||
@@ -378,6 +448,7 @@ cmd_verify() {
|
||||
printf '%s %s chars\n' "$elem" "${#n}"
|
||||
fi
|
||||
done
|
||||
verify_library_options || bad=1
|
||||
[ "$bad" -eq 0 ] || return 1
|
||||
printf 'providers configured\n'
|
||||
}
|
||||
|
||||
@@ -0,0 +1,420 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-ab-reorg.py — plan the Audiobooks dedupe + ABS-structure relayout.
|
||||
|
||||
2026-11-04 campaign. Turns the Audiobooks library into the layout Audiobookshelf
|
||||
documents (https://audiobookshelf.org/docs/documentation/libraries/book-library/
|
||||
directory-structure/):
|
||||
|
||||
{Author}/[{Series}/]{seq} - {year} - {Title} {Narrator}/…files…
|
||||
|
||||
The authority for every book's author/series/title/year/narrator is the ABS
|
||||
database dump (--items, JSON lines: [relPath, isMissing, title, authors, series,
|
||||
seq, year, narratorsJSON, coverPath, ebookFlag]); files are attached to their
|
||||
book by longest-prefix match on the item's relPath, which is how ABS's own
|
||||
grouping works. Folder names are then re-emitted so ABS's folder parser reads
|
||||
the same metadata back off disk.
|
||||
|
||||
Three outputs, all consumable by scripts/media-apply.py (run everything with
|
||||
--root /mnt/entropy; hold/intake/park live outside the library on purpose):
|
||||
|
||||
--plan mkdir/keep/move rows: dedupe losers -> --hold, ebook-only items
|
||||
and comics -> --intake (the eBooks pile feeds Calibre next),
|
||||
unowned media -> --park, everything else -> the new layout.
|
||||
--covers rows "srchost \\t dstarray": covers ABS keeps in /metadata get a
|
||||
copy beside the book (cp, it crosses filesystems); run this
|
||||
BEFORE apply so the cover lands inside the moved folder.
|
||||
--exceptions every judgement call: name collisions, byte-dups that pair two
|
||||
different books, collapsed items, stray files.
|
||||
|
||||
Dedupe law: only CONFIRMED byte-identities (--dups from
|
||||
media-ab-verifydups.py) remove a file. One copy survives per book: inside a
|
||||
folder the cleanest-named file wins; across folders the item whose root matches
|
||||
its author wins. Equal bytes under two different book identities keep both and
|
||||
raise an exception — a trailer embedded in two books is not a duplicate.
|
||||
|
||||
Rerunnable: a second run against a moved library emits keep rows, not moves.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections import defaultdict
|
||||
|
||||
AUDIO = {"m4b", "mp3", "m4a", "flac", "opus", "ogg", "oga", "mp4", "aac", "wma",
|
||||
"aiff", "aif", "wav", "webm", "webma", "mka", "awb", "caf", "mpg",
|
||||
"mpg2", "mpeg"}
|
||||
EBOOK = {"epub", "pdf", "mobi", "azw3", "cbr", "cbz", "fb2"}
|
||||
COMIC = {"cbr", "cbz"}
|
||||
MEDIA = AUDIO | {"epub", "pdf", "mobi", "azw3", "cbr", "cbz"}
|
||||
PLACEHOLDER_AUTHOR = {"unknown", "unknown artist", "various", "various artists",
|
||||
"various authors", "n/a", ""}
|
||||
BAD_CHARS = re.compile(r'[<>:"/\\|?*]')
|
||||
DISC_DIR = re.compile(r"^(disc|cd|disk)\s*\d", re.I)
|
||||
KEEP_HINT = re.compile(r"\s*\((?:2|3|copy|backup|duplicate)\)\s*$", re.I)
|
||||
META_HOST = "/home/dank/Containers/audiobookshelf"
|
||||
|
||||
|
||||
def clean(name):
|
||||
name = BAD_CHARS.sub(" ", name or "")
|
||||
name = re.sub(r"\s+", " ", name).strip().strip(".")
|
||||
return name or "Untitled"
|
||||
|
||||
|
||||
def clean_author(name):
|
||||
"""DB author string -> folder name: one author, no 'zz' sort artifact."""
|
||||
name = re.sub(r"^zz", "", (name or "").strip())
|
||||
return clean(name)
|
||||
|
||||
|
||||
def first_author(authors):
|
||||
parts = [clean_author(p) for p in (authors or "").split("|")]
|
||||
parts = [p for p in parts if p]
|
||||
return parts[0] if parts else "Unknown"
|
||||
|
||||
|
||||
def parse_narrators(blob):
|
||||
try:
|
||||
v = json.loads(blob) if blob else []
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
v = []
|
||||
if isinstance(v, str):
|
||||
v = [v]
|
||||
return [str(x).strip() for x in (v or []) if str(x).strip()]
|
||||
|
||||
|
||||
def core_title(title, author):
|
||||
"""Title as the book folder should show it: release prefixes stripped."""
|
||||
t = (title or "Untitled").strip()
|
||||
a = (author or "").strip()
|
||||
if a and t.lower().startswith(a.lower() + " -"):
|
||||
t = t[len(a):].lstrip(" -")
|
||||
return t.strip() or "Untitled"
|
||||
|
||||
|
||||
def bookname(seq, year, title, narrators):
|
||||
seq = (seq or "").strip()
|
||||
seq = seq if re.fullmatch(r"\d+(\.\d+)?", seq) else ""
|
||||
year = (year or "").strip()
|
||||
year = year if re.fullmatch(r"(1[5-9]|20)\d\d", year) else ""
|
||||
t = title
|
||||
if seq:
|
||||
# drop a sequence prefix already carried inside the title
|
||||
t = re.sub(rf"^{re.escape(seq)}\s*[\-.]\s+", "", t)
|
||||
if year:
|
||||
t = re.sub(rf"^{year}\s*[\-.]\s+", "", t)
|
||||
t = re.sub(rf"\s*\({year}\)\s*$", "", t)
|
||||
name = t
|
||||
if seq and year:
|
||||
name = f"{seq} - {year} - {name}"
|
||||
elif seq:
|
||||
name = f"{seq} - {name}"
|
||||
elif year:
|
||||
name = f"{year} - {name}"
|
||||
if len(narrators) == 1:
|
||||
name = f"{name} {{{clean(narrators[0])}}}"
|
||||
return clean(name)
|
||||
|
||||
|
||||
class Campaign:
|
||||
def __init__(self, args):
|
||||
self.root = args.root.rstrip("/")
|
||||
self.lib = self.root + "/Audiobooks"
|
||||
self.hold = args.hold
|
||||
self.intake = args.intake
|
||||
self.park = args.park
|
||||
self.items = {} # relPath -> item dict (healthy only)
|
||||
self.stale = [] # isMissing=1 rows
|
||||
self.moved = {} # src -> (dst, kind)
|
||||
self.claims = {} # dst -> src (collision guard)
|
||||
self.loser_files = set()
|
||||
self.loser_items = set()
|
||||
self.exc = defaultdict(list)
|
||||
|
||||
# ------------------------------------------------------------- inputs
|
||||
def load_items(self, path):
|
||||
for line in open(path):
|
||||
r = json.loads(line)
|
||||
rel = r[0]
|
||||
item = {"rel": rel, "title": r[2] or "", "authors": r[3] or "",
|
||||
"series": r[4] or "", "seq": r[5] or "", "year": r[6] or "",
|
||||
"narr": parse_narrators(r[7]), "cover": r[8] or "",
|
||||
"ebook": r[9]}
|
||||
if r[1]:
|
||||
self.stale.append(rel)
|
||||
continue
|
||||
self.items[rel] = item
|
||||
|
||||
def load_files(self, path):
|
||||
files = []
|
||||
for line in open(path):
|
||||
sz, ext, p = line.rstrip("\n").split("\t", 2)
|
||||
if p.startswith(self.lib + "/"):
|
||||
files.append((int(sz), ext.lstrip(".").lower(), p))
|
||||
return files
|
||||
|
||||
def owner(self, d):
|
||||
"""The healthy DB item whose folder is d or an ancestor of d."""
|
||||
rel = os.path.relpath(d, self.lib)
|
||||
while rel and rel != ".":
|
||||
if rel in self.items:
|
||||
return rel
|
||||
rel = os.path.dirname(rel)
|
||||
return None
|
||||
|
||||
# ------------------------------------------------------------- dedupe
|
||||
def identity(self, item_rel):
|
||||
if item_rel is None:
|
||||
return None
|
||||
it = self.items[item_rel]
|
||||
return (first_author(it["authors"]).lower(),
|
||||
core_title(it["title"], first_author(it["authors"])).lower(),
|
||||
(it["year"] or "").strip())
|
||||
|
||||
def root_score(self, item_rel):
|
||||
it = self.items[item_rel]
|
||||
top = item_rel.split("/")[0]
|
||||
s = 0
|
||||
if top.lower() == first_author(it["authors"]).lower():
|
||||
s += 2
|
||||
if not top.startswith("_") and "various" not in top.lower():
|
||||
s += 1
|
||||
return (s, -len(item_rel))
|
||||
|
||||
def dedupe(self, files, dups_path):
|
||||
by_hash = defaultdict(list)
|
||||
if dups_path:
|
||||
for line in open(dups_path):
|
||||
h, sz, p = line.rstrip("\n").split("\t", 2)
|
||||
if p.startswith(self.lib + "/"):
|
||||
by_hash[h].append(p)
|
||||
owner_of = {}
|
||||
dirs = {os.path.dirname(p) for _, _, p in files}
|
||||
for d in dirs:
|
||||
owner_of[d] = self.owner(d)
|
||||
files_by_owner = defaultdict(list)
|
||||
for _, _, p in files:
|
||||
files_by_owner[owner_of[os.path.dirname(p)]].append(p)
|
||||
|
||||
for h, paths in by_hash.items():
|
||||
owners = {owner_of.get(os.path.dirname(p)) for p in paths}
|
||||
owners = {o for o in owners if o is not None}
|
||||
per_item = defaultdict(list)
|
||||
for p in paths:
|
||||
per_item[owner_of.get(os.path.dirname(p))].append(p)
|
||||
if owners:
|
||||
ids = {o: self.identity(o) for o in owners}
|
||||
# group items by identity; loose (None) files follow their own rule
|
||||
same_as_loser = []
|
||||
keepers = []
|
||||
byid = defaultdict(list)
|
||||
for o in owners:
|
||||
byid[ids[o]].append(o)
|
||||
for ident, os_ in byid.items():
|
||||
os_.sort(key=self.root_score, reverse=True)
|
||||
keepers.append(os_[0])
|
||||
same_as_loser.extend(os_[1:])
|
||||
keepers = set(keepers)
|
||||
loser_items = set(same_as_loser)
|
||||
else:
|
||||
loser_items = set()
|
||||
keepers = set()
|
||||
for o, ps in per_item.items():
|
||||
if o is None:
|
||||
continue # loose files decided in placement
|
||||
if o in loser_items:
|
||||
self.loser_files.update(ps)
|
||||
continue
|
||||
if len(ps) > 1 and o in keepers:
|
||||
# redundant copies inside the surviving folder
|
||||
keeper = min(ps, key=lambda p: (
|
||||
bool(KEEP_HINT.search(os.path.splitext(p)[0])),
|
||||
len(p), p))
|
||||
self.loser_files.update(p for p in ps if p != keeper)
|
||||
for o in loser_items:
|
||||
self.loser_items.add(o)
|
||||
# collapsed items: every media file lost
|
||||
for o, ps in files_by_owner.items():
|
||||
if o is None:
|
||||
continue
|
||||
media = [p for p in ps if os.path.splitext(p)[1].lstrip(".").lower()
|
||||
in MEDIA]
|
||||
if media and all(p in self.loser_files for p in media):
|
||||
self.loser_items.add(o)
|
||||
self.exc["collapsed items (all files were byte-duplicates)"].append(o)
|
||||
|
||||
# ---------------------------------------------------------- placement
|
||||
def target(self, item):
|
||||
author = first_author(item["authors"])
|
||||
series = clean(item["series"]) if item["series"] else ""
|
||||
title = core_title(item["title"], author)
|
||||
name = bookname(item["seq"], item["year"], title, item["narr"])
|
||||
parts = [self.lib, clean(author)]
|
||||
if series:
|
||||
parts.append(clean(series))
|
||||
parts.append(name)
|
||||
return os.path.join(*parts)
|
||||
|
||||
def claim(self, dst, src, why):
|
||||
"""One destination, one source; collisions rename with a marker."""
|
||||
if dst not in self.claims:
|
||||
self.claims[dst] = src
|
||||
return dst
|
||||
stem, ext = os.path.splitext(dst)
|
||||
n = 2
|
||||
while f"{stem} ({n}){ext}" in self.claims:
|
||||
n += 1
|
||||
dst2 = f"{stem} ({n}){ext}"
|
||||
self.claims[dst2] = src
|
||||
self.exc[why].append(f"{src} -> renamed to {dst2}")
|
||||
return dst2
|
||||
|
||||
def place(self, files):
|
||||
audio_by_item = defaultdict(int)
|
||||
for _, ext, p in files:
|
||||
o = self.owner(os.path.dirname(p))
|
||||
if o and ext in AUDIO:
|
||||
audio_by_item[o] += 1
|
||||
cover_dirs = {}
|
||||
# Claim one BOOK FOLDER per item: two items for the same title (the
|
||||
# _unsorted era filed some books 3 times) must not merge their tracks
|
||||
# into one item — the loser gets a variant folder instead.
|
||||
for rel in sorted(self.items, key=lambda r: self.root_score(r),
|
||||
reverse=True):
|
||||
it = self.items[rel]
|
||||
t = self.target(it)
|
||||
if t in self.claims:
|
||||
n = 2
|
||||
while f"{t} (variant {n})" in self.claims:
|
||||
n += 1
|
||||
t = f"{t} (variant {n})"
|
||||
self.exc["same title filed by two DB items"].append(f"{rel} -> {t}")
|
||||
self.claims[t] = rel
|
||||
cover_dirs[rel] = t
|
||||
if (it["cover"].startswith("/metadata") and audio_by_item[rel]
|
||||
and rel not in self.loser_items):
|
||||
src = META_HOST + it["cover"]
|
||||
self.covers.append((src, os.path.join(t,
|
||||
"cover" + os.path.splitext(src)[1].lower())))
|
||||
# Containment gate: an item folder holding media files must never sit
|
||||
# above another item folder — ABS's grouping would swallow the child
|
||||
# (the docs' PROBLEM example). Rename the ANCESTOR book out of the way.
|
||||
for _ in range(3):
|
||||
moved = False
|
||||
folders = sorted(set(cover_dirs.values()))
|
||||
for anc in folders:
|
||||
for d in folders:
|
||||
if d != anc and d.startswith(anc + os.sep):
|
||||
n = 1
|
||||
cand = f"{anc} (book)" if n == 1 else f"{anc} (book {n})"
|
||||
while cand in self.claims:
|
||||
n += 1
|
||||
cand = f"{anc} (book {n})"
|
||||
self.exc["book folder nested above another book"].append(
|
||||
f"{anc} -> {cand}")
|
||||
self.claims[cand] = self.claims.pop(anc)
|
||||
for r, t in cover_dirs.items():
|
||||
if t == anc:
|
||||
cover_dirs[r] = cand
|
||||
self.covers = [(s, d.replace(anc + os.sep, cand + os.sep, 1))
|
||||
for s, d in self.covers]
|
||||
moved = True
|
||||
break
|
||||
if moved:
|
||||
break
|
||||
if not moved:
|
||||
break
|
||||
for _, ext, p in files:
|
||||
if p in self.loser_files:
|
||||
d = os.path.join(self.hold, os.path.relpath(p, self.root))
|
||||
self.move(p, d, "dup")
|
||||
continue
|
||||
d = os.path.dirname(p)
|
||||
o = self.owner(d)
|
||||
if o is None:
|
||||
rel = os.path.relpath(p, self.lib)
|
||||
if ext in MEDIA:
|
||||
self.move(p, os.path.join(self.park, rel), "park")
|
||||
else:
|
||||
self.move(p, os.path.join(self.park, rel), "park-junk")
|
||||
continue
|
||||
it = self.items[o]
|
||||
if ext in COMIC:
|
||||
key = clean(f"{first_author(it['authors'])} - {core_title(it['title'], '')}")
|
||||
dst = os.path.join(self.intake, "_comics", key, os.path.basename(p))
|
||||
self.move(p, self.claim(dst, p, "intake comic collisions"), "comic")
|
||||
continue
|
||||
if audio_by_item[o] == 0: # ebook-only item belongs to the ebook stack
|
||||
key = f"{first_author(it['authors'])} - {bookname(it['seq'], it['year'], core_title(it['title'], first_author(it['authors'])), it['narr'])}"
|
||||
dst = os.path.join(self.intake, key, os.path.relpath(p, os.path.join(self.lib, o)))
|
||||
self.move(p, self.claim(dst, p, "intake ebook collisions"), "ebook-only")
|
||||
continue
|
||||
rel_in = os.path.relpath(p, os.path.join(self.lib, o))
|
||||
dst = os.path.join(cover_dirs[o], rel_in)
|
||||
if dst == p:
|
||||
self.kept.append(p)
|
||||
self.claims.setdefault(dst, p)
|
||||
continue
|
||||
self.move(p, self.claim(dst, p, "target collisions"), "move")
|
||||
|
||||
def move(self, src, dst, kind):
|
||||
self.moves.append((src, dst, kind))
|
||||
|
||||
def write(self, plan_path, exc_path):
|
||||
dirs = {os.path.dirname(dst) for _, dst, _ in self.moves}
|
||||
dirs |= {os.path.dirname(d) for _, d in self.covers}
|
||||
all_dirs = set()
|
||||
for d in dirs:
|
||||
while d and d != self.root:
|
||||
all_dirs.add(d)
|
||||
d = os.path.dirname(d)
|
||||
with open(plan_path, "w") as fh:
|
||||
fh.write("action\tsrc\tdst\tkind\n")
|
||||
for d in sorted(all_dirs):
|
||||
fh.write(f"mkdir\t\t{d}\tcampaign\n")
|
||||
for p in self.kept:
|
||||
fh.write(f"keep\t{p}\t\tkeep\n")
|
||||
for src, dst, kind in sorted(self.moves):
|
||||
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||
if self.exc:
|
||||
with open(exc_path, "w") as fh:
|
||||
for why in sorted(self.exc):
|
||||
fh.write(f"# {why}\n")
|
||||
for line in sorted(set(self.exc[why])):
|
||||
fh.write(line + "\n")
|
||||
fh.write("\n")
|
||||
mv = [m for m in self.moves if m[2] != "dup"]
|
||||
print(f"plan: {len(all_dirs)} folders, {len(self.moves)} moves "
|
||||
f"({len(mv)} relayout, {len(self.moves)-len(mv)} dup-hold), "
|
||||
f"{len(self.kept)} keep, {len(self.covers)} cover-copies")
|
||||
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default="/mnt/entropy")
|
||||
ap.add_argument("--enum", required=True)
|
||||
ap.add_argument("--items", required=True)
|
||||
ap.add_argument("--dups", default=None)
|
||||
ap.add_argument("--hold", default="/mnt/entropy/.media-campaign/dup-hold-20261104")
|
||||
ap.add_argument("--intake", default="/mnt/entropy/eBooks/_from-audiobooks")
|
||||
ap.add_argument("--park", default="/mnt/entropy/NeedsProcessing/_from-audiobooks-20261104")
|
||||
ap.add_argument("--plan", required=True)
|
||||
ap.add_argument("--exceptions", required=True)
|
||||
ap.add_argument("--covers", required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
c = Campaign(args)
|
||||
c.moves, c.kept, c.covers = [], [], []
|
||||
c.load_items(args.items)
|
||||
files = c.load_files(args.enum)
|
||||
c.dedupe(files, args.dups)
|
||||
c.place(files)
|
||||
c.write(args.plan, args.exceptions)
|
||||
with open(args.covers, "w") as fh:
|
||||
for src, dst in sorted(set(c.covers)):
|
||||
fh.write(f"{src}\t{dst}\n")
|
||||
print(f"covers: {len(set(c.covers))} stale DB rows (delete after scan): {len(c.stale)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-ab-verifydups.py — confirm byte-duplicates among size-group survivors.
|
||||
|
||||
Stage 2 of the 2026-11-04 Audiobooks dedupe. Stage 1 (partial blake2b of the
|
||||
first+last 4 MiB, done by the campaign's hashcand.py) over-claims: two files
|
||||
of equal size and equal head+tail are CANDIDATES, not copies. This tool takes
|
||||
the candidate files from --partial (rows: size, hash, path), groups them by
|
||||
(size, partial-hash), full-hashes only members of groups with 2+ members, and
|
||||
writes verdicts: one line per file whose full hash ties it to at least one
|
||||
other file.
|
||||
|
||||
Output rows: full-hash <TAB> size <TAB> path (sorted by hash) — a hash that
|
||||
appears once is not written at all; singletons are not duplicates.
|
||||
|
||||
Read-only. Never moves, never deletes. Run it after the partial pass finishes
|
||||
(pgrep it first; two hashers on one output file have raced before).
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import os
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
def full_hash(path, chunk=1 << 20):
|
||||
h = hashlib.blake2b(digest_size=32)
|
||||
with open(path, "rb") as fh:
|
||||
for block in iter(lambda: fh.read(chunk), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--partial", required=True, help="partial-hashes.tsv")
|
||||
ap.add_argument("--out", required=True, help="full-hash verdicts TSV")
|
||||
ap.add_argument("--progress-every", type=int, default=50)
|
||||
args = ap.parse_args()
|
||||
|
||||
groups = defaultdict(list)
|
||||
with open(args.partial) as fh:
|
||||
for line in fh:
|
||||
f = line.rstrip("\n").split("\t")
|
||||
if len(f) != 3 or f[0] == "ERR":
|
||||
continue
|
||||
groups[(int(f[0]), f[1])].append(f[2])
|
||||
survivors = sorted({p for g in groups.values() if len(g) > 1 for p in g})
|
||||
print(f"survivor files to full-hash: {len(survivors)}")
|
||||
|
||||
done = set()
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out) as fh:
|
||||
for line in fh:
|
||||
done.add(line.rstrip("\n").split("\t", 2)[2])
|
||||
full = {}
|
||||
for line in open(args.out) if os.path.exists(args.out) else []:
|
||||
h, s, p = line.rstrip("\n").split("\t", 2)
|
||||
full[p] = h
|
||||
|
||||
todo = [p for p in survivors if p not in done]
|
||||
out = open(args.out, "a")
|
||||
for i, p in enumerate(todo):
|
||||
try:
|
||||
full[p] = full_hash(p)
|
||||
except OSError as e:
|
||||
print(f"SKIP {p}: {e}")
|
||||
continue
|
||||
if i % args.progress_every == args.progress_every - 1:
|
||||
out.flush()
|
||||
out.close()
|
||||
|
||||
# Rewrite the whole verdict file keeping only TRUE duplicates (2+ fulls).
|
||||
by_hash = defaultdict(list)
|
||||
for p in survivors:
|
||||
if p in full:
|
||||
by_hash[full[p]].append(p)
|
||||
with open(args.out, "w") as out:
|
||||
for h, ps in sorted(by_hash.items()):
|
||||
if len(ps) > 1:
|
||||
for p in sorted(ps):
|
||||
out.write(f"{h}\t{os.path.getsize(p)}\t{p}\n")
|
||||
dups = {h: ps for h, ps in by_hash.items() if len(ps) > 1}
|
||||
n = sum(len(ps) - 1 for ps in dups.values())
|
||||
waste = sum((len(ps) - 1) * os.path.getsize(ps[0]) for ps in dups.values())
|
||||
print(f"true dup sets: {len(dups)} redundant files: {n} "
|
||||
f"waste: {waste / 2**30:.1f} GiB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -121,14 +121,14 @@ class Journal:
|
||||
|
||||
def read_plan(path, root, force):
|
||||
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
|
||||
mkdirs, keeps, moves = [], [], []
|
||||
mkdirs, keeps, moves, bad = [], [], [], []
|
||||
with open(path) as fh:
|
||||
header = fh.readline()
|
||||
if not header.startswith("action\t"):
|
||||
raise Refuse(f"{path} is not a plan (first column is "
|
||||
f"{header.split(chr(9))[0]!r}, expected 'action') —"
|
||||
" regenerate it with scripts/media-plan.py")
|
||||
for line in fh:
|
||||
for n, line in enumerate(fh, 2):
|
||||
f = line.rstrip("\n").split("\t")
|
||||
if f[0] == "mkdir":
|
||||
mkdirs.append(f[2])
|
||||
@@ -136,6 +136,14 @@ def read_plan(path, root, force):
|
||||
keeps.append(f[1])
|
||||
elif f[0] == "move":
|
||||
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
|
||||
else:
|
||||
bad.append(f"line {n}: unknown action {f[0]!r} — this tool"
|
||||
" knows mkdir, keep and move (a rename is a move)")
|
||||
if bad:
|
||||
# A row this tool could not read is a plan bug, and a plan bug refuses:
|
||||
# silently dropping it turns a typo into a no-op run that reports success.
|
||||
head = "; ".join(bad[:3])
|
||||
raise Refuse(head + (f" (+{len(bad) - 3} more)" if len(bad) > 3 else ""))
|
||||
if not force:
|
||||
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
|
||||
if p and not (p == root or p.startswith(root + os.sep)):
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-ebook-dedupe.py — dedupe the eBooks pile against the Calibre library.
|
||||
|
||||
2026-11-04 campaign, ebook side. The Calibre library (/mnt/entropy/CalibreBooks,
|
||||
6,289 books) is DB-owned and healthy; /mnt/entropy/eBooks is a raw pile whose
|
||||
best books already live in the library, several of them twice in the pile.
|
||||
Audiobooks' orphaned ebooks/comics land in the pile first (media-ab-reorg's
|
||||
intake), so this campaign runs after the audiobook move.
|
||||
|
||||
Inputs
|
||||
--meta ebook-meta.tsv: JSON lines {p, sz, title, authors, ident} for
|
||||
epub (embedded OPF) and mobi/azw3 (Palm header title); the
|
||||
producer is the campaign's opfcensus.py.
|
||||
--enum enum TSV (size, ext, path) for every tree.
|
||||
--calibre calibre-books.tsv: JSON lines [id, title, authors, series, path,
|
||||
pubdate, formats] dumped from metadata.db.
|
||||
--dups full blake2b verdicts (hash, size, path) — only confirmed
|
||||
byte-identities remove a file. Optional: without it, identity
|
||||
rules alone run and every hash-dependent pair stays a KEEP.
|
||||
|
||||
Outputs
|
||||
--plan media-apply plan (--root /mnt/entropy): hash-duplicates of a
|
||||
file already in CalibreBooks -> HOLD; byte-identical files inside
|
||||
the pile keep the best format of one book once; non-book junk
|
||||
(mp4/mp3/images) -> PARK.
|
||||
--ingest one JSON line per surviving BOOK (a set of format files of one
|
||||
identity): {paths, title, authors, source}. The ingest runner
|
||||
feeds these to calibredb inside the calibre-web image.
|
||||
--exceptions every identity pair the file-level evidence could not settle.
|
||||
|
||||
Match law: identity is (core title, first author surname). Calibre's
|
||||
"Dinniman| Matt" and an OPF "Matt Dinniman" share surname "dinniman". Title
|
||||
core drops the leading article and all non-alphanumerics. Identity alone NEVER
|
||||
deletes against the pile — it only deletes when the same identity sits in the
|
||||
Calibre library with the same format (the pile copy adds nothing), or when a
|
||||
byte-hash confirms the pair. Format preference when the pile holds one book in
|
||||
several copies: epub > azw3 > mobi > pdf > fb2 > lit > rtf > txt.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import unicodedata
|
||||
from collections import defaultdict
|
||||
|
||||
BOOK_EXT = {"epub", "azw3", "mobi", "pdf", "fb2", "lit", "rtf", "txt", "cbr", "cbz"}
|
||||
FORMAT_RANK = {"epub": 0, "azw3": 1, "mobi": 2, "pdf": 3, "fb2": 4,
|
||||
"cbr": 5, "cbz": 5, "lit": 6, "rtf": 7, "txt": 8}
|
||||
ARTICLE = re.compile(r"^(the|a|an)\s+", re.I)
|
||||
|
||||
|
||||
def ascii_fold(s):
|
||||
return "".join(c for c in unicodedata.normalize("NFKD", s or "")
|
||||
if not unicodedata.combining(c))
|
||||
|
||||
|
||||
def title_core(title):
|
||||
t = ascii_fold(title).lower()
|
||||
t = ARTICLE.sub("", t)
|
||||
t = re.sub(r"\(.*?\)", " ", t) # (1998), (Unabridged) — noise
|
||||
t = re.sub(r"[^a-z0-9 ]", " ", t)
|
||||
t = re.sub(r"\b(1[5-9]|20)\d\d\b", " ", t) # years
|
||||
return " ".join(t.split())[:120]
|
||||
|
||||
|
||||
def surname(author):
|
||||
a = ascii_fold(author).strip()
|
||||
a = re.split(r"[;,]", a)[0] if "," in a else " ".join(a.split()[-1:])
|
||||
return re.sub(r"[^a-z]", "", a.lower())
|
||||
|
||||
|
||||
def first_author(authors):
|
||||
parts = [p.strip() for p in re.split(r"[;&]", authors or "") if p.strip()]
|
||||
return parts[0] if parts else ""
|
||||
|
||||
|
||||
def stem_title(path):
|
||||
s = os.path.splitext(os.path.basename(path))[0]
|
||||
return ascii_fold(s)
|
||||
|
||||
|
||||
class Campaign:
|
||||
def __init__(self, args):
|
||||
self.args = args
|
||||
self.root = args.root.rstrip("/")
|
||||
self.pile = self.root + "/eBooks"
|
||||
self.lib = self.root + "/CalibreBooks"
|
||||
self.hold = args.hold
|
||||
self.park = args.park
|
||||
self.meta = {} # path -> meta dict
|
||||
self.pile_files = [] # (sz, ext, path)
|
||||
self.cal_by_path = {} # file path under CalibreBooks -> calibre row
|
||||
self.cal_identity = {} # (tcore, surname) -> calibre row
|
||||
self.drop = {} # path -> reason
|
||||
self.exc = defaultdict(list)
|
||||
self.moves = []
|
||||
self.dirs_pile_book = defaultdict(list) # book dir (pile) -> files
|
||||
|
||||
# ------------------------------------------------------------- inputs
|
||||
def load(self):
|
||||
for line in open(self.args.meta):
|
||||
r = json.loads(line)
|
||||
self.meta[r["p"]] = r
|
||||
by_hash = defaultdict(list)
|
||||
if self.args.dups:
|
||||
for line in open(self.args.dups):
|
||||
h, sz, p = line.rstrip("\n").split("\t", 2)
|
||||
by_hash[h].append(p)
|
||||
self.dup_sets = [ps for ps in by_hash.values() if len(ps) > 1]
|
||||
for line in open(self.args.enum):
|
||||
sz, ext, p = line.rstrip("\n").split("\t", 2)
|
||||
ext = ext.lstrip(".").lower()
|
||||
if p.startswith(self.pile + "/") and p not in self.park:
|
||||
self.pile_files.append((int(sz), ext, p))
|
||||
for line in open(self.args.calibre):
|
||||
row = json.loads(line)
|
||||
self.cal_identity[(title_core(row[1]), surname(first_author(row[2])))] = row
|
||||
|
||||
def identity(self, p):
|
||||
m = self.meta.get(p)
|
||||
if m and m.get("title"):
|
||||
t = title_core(m["title"])
|
||||
a = surname(first_author(m.get("authors", "")))
|
||||
if t:
|
||||
return (t, a)
|
||||
return (title_core(stem_title(p)), "")
|
||||
|
||||
# ------------------------------------------------------------- rules
|
||||
def run(self):
|
||||
pile_paths = {p for _, _, p in self.pile_files}
|
||||
# R1/R2: confirmed byte-duplicates
|
||||
for ps in self.dup_sets:
|
||||
in_lib = [p for p in ps if p.startswith(self.lib + "/")]
|
||||
in_pile = [p for p in ps if p.startswith(self.pile + "/")]
|
||||
if in_lib:
|
||||
for p in in_pile:
|
||||
self.drop[p] = "hash-duplicate of CalibreBooks file"
|
||||
if len(in_pile) > 1:
|
||||
keep = min(in_pile, key=lambda p: (FORMAT_RANK.get(
|
||||
os.path.splitext(p)[1].lstrip(".").lower(), 9), len(p), p))
|
||||
for p in in_pile:
|
||||
if p != keep:
|
||||
self.drop[p] = "hash-duplicate inside the pile"
|
||||
# R3/R4: identity against the library (same format only deletes)
|
||||
cal_files = defaultdict(set) # identity -> formats in the library
|
||||
for key, row in self.cal_identity.items():
|
||||
cal_files[key] = set(f for f in row[6].split("; ") if f)
|
||||
for sz, ext, p in self.pile_files:
|
||||
if p in self.drop or ext not in BOOK_EXT:
|
||||
continue
|
||||
key = self.identity(p)
|
||||
if key[0] and key in self.cal_identity:
|
||||
if ext in cal_files[key]:
|
||||
self.drop[p] = f"same identity+format already in Calibre ({key[0][:40]})"
|
||||
else:
|
||||
self.exc["pile holds a format the library lacks (add-format on ingest)"].append(
|
||||
f"{p}\t{key[0][:50]}\t{self.cal_identity[key][0]}")
|
||||
# R5: surviving files grouped by (identity, parent-ish book dir)
|
||||
books = defaultdict(list)
|
||||
for sz, ext, p in self.pile_files:
|
||||
if p in self.drop:
|
||||
continue
|
||||
if ext not in BOOK_EXT:
|
||||
self.moves.append((p, os.path.join(self.park, os.path.relpath(p, self.root)),
|
||||
"non-book"))
|
||||
continue
|
||||
m = self.meta.get(p, {})
|
||||
if m.get("err") and ext in ("epub",):
|
||||
self.exc["unreadable container (park for a human)"].append(p)
|
||||
self.moves.append((p, os.path.join(self.park, os.path.relpath(p, self.root)),
|
||||
"broken-book"))
|
||||
continue
|
||||
key = self.identity(p)
|
||||
books[key].append(p)
|
||||
for key, ps in books.items():
|
||||
first = min(ps, key=lambda p: (FORMAT_RANK.get(
|
||||
os.path.splitext(p)[1].lstrip(".").lower(), 9), len(p), p))
|
||||
m = self.meta.get(first, {})
|
||||
self.ingest = {
|
||||
"key": list(key),
|
||||
"paths": sorted(ps),
|
||||
"title": m.get("title") or stem_title(first),
|
||||
"authors": m.get("authors") or "",
|
||||
"ident": m.get("ident") or "",
|
||||
}
|
||||
self.ingest_list.append(self.ingest)
|
||||
for p, why in self.drop.items():
|
||||
self.moves.append((p, os.path.join(self.hold, os.path.relpath(p, self.root)), "dup"))
|
||||
|
||||
def write(self):
|
||||
dirs = {os.path.dirname(d) for _, d, _ in self.moves}
|
||||
all_dirs = set()
|
||||
for d in dirs:
|
||||
while d and d != self.root:
|
||||
all_dirs.add(d)
|
||||
d = os.path.dirname(d)
|
||||
with open(self.args.plan, "w") as fh:
|
||||
fh.write("action\tsrc\tdst\tkind\n")
|
||||
for d in sorted(all_dirs):
|
||||
fh.write(f"mkdir\t\t{d}\tcampaign\n")
|
||||
for src, dst, kind in sorted(self.moves):
|
||||
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||
with open(self.args.ingest, "w") as fh:
|
||||
for b in self.ingest_list:
|
||||
fh.write(json.dumps(b, ensure_ascii=False) + "\n")
|
||||
with open(self.args.exceptions, "w") as fh:
|
||||
for why in sorted(self.exc):
|
||||
fh.write(f"# {why}\n")
|
||||
for line in sorted(set(self.exc[why])):
|
||||
fh.write(line + "\n")
|
||||
fh.write("\n")
|
||||
ndrop = sum(1 for _, _, k in self.moves if k == "dup")
|
||||
print(f"pile books to ingest: {len(self.ingest_list)} "
|
||||
f"files dropped as duplicates: {ndrop} "
|
||||
f"non-book/broken moved out: {sum(1 for _, _, k in self.moves if k in ('non-book', 'broken-book'))}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--root", default="/mnt/entropy")
|
||||
ap.add_argument("--meta", required=True)
|
||||
ap.add_argument("--enum", required=True)
|
||||
ap.add_argument("--calibre", required=True)
|
||||
ap.add_argument("--dups", default=None)
|
||||
ap.add_argument("--hold", default="/mnt/entropy/.media-campaign/dup-hold-ebooks-20261104")
|
||||
ap.add_argument("--park", default="/mnt/entropy/NeedsProcessing/_from-ebooks-20261104")
|
||||
ap.add_argument("--plan", required=True)
|
||||
ap.add_argument("--ingest", required=True)
|
||||
ap.add_argument("--exceptions", required=True)
|
||||
args = ap.parse_args()
|
||||
c = Campaign(args)
|
||||
c.ingest_list = []
|
||||
c.load()
|
||||
c.run()
|
||||
c.write()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,223 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Trigger and poll Jellyfin's library scan — the step that makes a media move visible.
|
||||
|
||||
media-refresh-jellyfin.py trigger [--force]
|
||||
media-refresh-jellyfin.py status [--after EPOCH]
|
||||
media-refresh-jellyfin.py wait [--after EPOCH] [--interval S] [--require-quiet N]
|
||||
|
||||
`status` answers {"scan": true, "taskState": "Running", "recentMedia": 17}: the
|
||||
Scan Media Library row plus the rows the database has taken in recently; with
|
||||
`--after` (the trigger's epoch — a file mtime is already passed by an earlier scan) it
|
||||
also answers `scannedAfter` and exits non-zero
|
||||
while the last finished scan predates it. `wait` calls the library settled only
|
||||
on the page's law: a sighting of the task Running — or a run that finished while
|
||||
it watched, or a `--after` the task already passes — and THEN `--require-quiet`
|
||||
consecutive reads with no scan and no writes. Both the scan and the metadata
|
||||
fetch must be quiet, and the Running sighting is what stops a watcher from
|
||||
declaring a pre-scan library settled (Dankserver/Jellyfin.md).
|
||||
|
||||
Laws this encodes:
|
||||
* Auth is `Authorization: MediaBrowser Token="…"`. A bare X-Emby-Token answers
|
||||
401 (measured on franksdankcloud 12.4.0), which reads exactly like a dead
|
||||
key. The key comes from a mode-600 file and never rides argv or a URL:
|
||||
/proc/<pid>/cmdline is world-readable.
|
||||
* `POST /Library/Refresh` (204) is the ONLY door that queues work on 12.4, and
|
||||
it walks every library. `POST /Items/{id}/Refresh` answers 204 and queues
|
||||
NOTHING (measured twice, 2026-11-02) — so there is no --library here.
|
||||
* A SECOND accepted POST cancels and restarts the scan already running, so
|
||||
progress looks frozen while you hammer it. `trigger` therefore refuses while
|
||||
the Scan Media Library task is not Idle; --force says you want the restart.
|
||||
* 429 means busy. Waiting is the only response that helps, and it is bounded
|
||||
so a wedged server ends the run with a named failure instead of a loop.
|
||||
* `RefreshStatus` on /Library/VirtualFolders stays Idle through a live scan;
|
||||
never poll it. The task table is the only honest read.
|
||||
|
||||
Defaults are this fleet's layout: http://127.0.0.1:8096, the key at
|
||||
~/.config/secrets/jellyfin, and the database found by trying, in order, $JELLYFIN_DB,
|
||||
~/.local/share/jellyfin/jellyfin.db and the quadlet's config bind.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SCAN_TASK = "Scan Media Library"
|
||||
IDLE = ("Idle",) # Running, Cancelling and Waiting all mean: not settled
|
||||
DB_CANDIDATES = ("~/.local/share/jellyfin/jellyfin.db",
|
||||
"/home/dank/Containers/jellyfin/config/data/jellyfin.db")
|
||||
|
||||
|
||||
def _epoch(iso: str | None) -> float | None:
|
||||
"""Jellyfin's `2026-11-04T18:31:33Z` EndTimeUtc as a unix epoch, or None."""
|
||||
if not iso:
|
||||
return None
|
||||
for fmt in ("%Y-%m-%dT%H:%M:%S.%fZ", "%Y-%m-%dT%H:%M:%SZ"):
|
||||
try:
|
||||
return datetime.strptime(iso, fmt).replace(tzinfo=timezone.utc).timestamp()
|
||||
except ValueError:
|
||||
continue
|
||||
sys.exit(f"the server sent an unparsable EndTimeUtc: {iso!r}")
|
||||
|
||||
|
||||
def read_key(value: str) -> str:
|
||||
path = pathlib.Path(value).expanduser()
|
||||
try:
|
||||
token = path.read_text().strip()
|
||||
except OSError as exc:
|
||||
sys.exit(f"no Jellyfin API key at {path} ({exc.strerror}); "
|
||||
"that file is the credential this driver authenticates with")
|
||||
if not token:
|
||||
sys.exit(f"Jellyfin API key file {path} is empty")
|
||||
return token
|
||||
|
||||
|
||||
def request(base: str, token: str, method: str, path: str,
|
||||
timeout: float = 30.0) -> tuple[int, bytes]:
|
||||
req = urllib.request.Request(
|
||||
base.rstrip("/") + path, method=method,
|
||||
headers={"Authorization": f'MediaBrowser Token="{token}"'})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return resp.status, resp.read()
|
||||
except urllib.error.HTTPError as exc:
|
||||
return exc.code, exc.read()
|
||||
|
||||
|
||||
def scan_task(base: str, token: str) -> dict:
|
||||
"""The one task row that means something, and when its last run finished."""
|
||||
code, body = request(base, token, "GET", "/ScheduledTasks")
|
||||
if code != 200:
|
||||
sys.exit(f"GET /ScheduledTasks -> HTTP {code}: {body[:200]!r}")
|
||||
for task in json.loads(body):
|
||||
if task.get("Name") == SCAN_TASK:
|
||||
state = task.get("State")
|
||||
return {"state": state, "running": state not in IDLE,
|
||||
"lastScan": _epoch(task.get("EndTimeUtc"))}
|
||||
sys.exit(f"the server does not list a {SCAN_TASK!r} task; refusing to guess")
|
||||
|
||||
|
||||
def find_db(value: str) -> pathlib.Path:
|
||||
wanted = [value] if value != "-" else []
|
||||
wanted += [p for p in (os.environ.get("JELLYFIN_DB"),) if p]
|
||||
wanted += list(DB_CANDIDATES)
|
||||
for candidate in wanted:
|
||||
path = pathlib.Path(candidate).expanduser()
|
||||
if path.is_file():
|
||||
return path
|
||||
sys.exit("no Jellyfin database found; tried: " + ", ".join(wanted))
|
||||
|
||||
|
||||
def recent_media(db: str, window: int) -> int:
|
||||
since = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(time.time() - window * 60))
|
||||
con = sqlite3.connect(f"file:{find_db(db)}?mode=ro", uri=True)
|
||||
try:
|
||||
return con.execute(
|
||||
"SELECT COUNT(*) FROM BaseItems WHERE DateModified >= ?", (since,)
|
||||
).fetchone()[0]
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("command", choices=["trigger", "status", "wait"])
|
||||
ap.add_argument("--url", default="http://127.0.0.1:8096")
|
||||
ap.add_argument("--key-file", default="~/.config/secrets/jellyfin")
|
||||
ap.add_argument("--db", default="-", help="jellyfin.db; - searches the known homes")
|
||||
ap.add_argument("--window", type=int, default=5,
|
||||
help="minutes of DB writes that still count as recent")
|
||||
ap.add_argument("--retries", type=int, default=3, help="extra attempts against a 429")
|
||||
ap.add_argument("--retry-sleep", type=float, default=20.0)
|
||||
ap.add_argument("--force", action="store_true",
|
||||
help="trigger even though a scan is already running")
|
||||
ap.add_argument("--interval", type=float, default=12.0, help="wait: seconds between reads")
|
||||
ap.add_argument("--require-quiet", type=int, default=3,
|
||||
help="wait: consecutive quiet reads before the library counts as settled")
|
||||
ap.add_argument("--after", type=float, default=None,
|
||||
help="status/wait: an epoch the scan must have finished after — pass "
|
||||
"the trigger's, not a file mtime an earlier scan already passes")
|
||||
ap.add_argument("--start-window", type=float, default=120.0,
|
||||
help="wait: seconds to allow the scan to start before reporting "
|
||||
"that the 204 queued nothing")
|
||||
ap.add_argument("--timeout", type=float, default=1800.0, help="wait: seconds before giving up")
|
||||
args = ap.parse_args()
|
||||
token = read_key(args.key_file)
|
||||
|
||||
def state() -> dict:
|
||||
task = scan_task(args.url, token)
|
||||
out = {"scan": task["running"], "taskState": task["state"],
|
||||
"lastScan": task["lastScan"],
|
||||
"recentMedia": recent_media(args.db, args.window)}
|
||||
if args.after is not None:
|
||||
out["scannedAfter"] = bool(task["lastScan"] and task["lastScan"] >= args.after)
|
||||
return out
|
||||
|
||||
if args.command == "status":
|
||||
current = state()
|
||||
print(json.dumps(current))
|
||||
if args.after is not None and not current["scannedAfter"]:
|
||||
sys.exit(f"the last scan finished before {args.after:.0f}; "
|
||||
"a refresh is still owed")
|
||||
return
|
||||
|
||||
if args.command == "wait":
|
||||
start = time.monotonic()
|
||||
ended_at_entry, quiet, started, first = None, 0, False, True
|
||||
while True:
|
||||
current = state()
|
||||
if first:
|
||||
first, ended_at_entry = False, current["lastScan"]
|
||||
if (current["scan"]
|
||||
or (current["lastScan"] and current["lastScan"] != ended_at_entry)
|
||||
or current.get("scannedAfter")):
|
||||
# The settle law: a Running sighting (or a run that finished while
|
||||
# watching, or a --after the task already passes) must come BEFORE
|
||||
# any claim that the library is quiet.
|
||||
started = True
|
||||
busy = current["scan"] or current["recentMedia"]
|
||||
quiet = quiet + 1 if started and not busy else 0
|
||||
print(json.dumps(current | {"quiet": quiet}), flush=True)
|
||||
if quiet >= args.require_quiet:
|
||||
return
|
||||
elapsed = time.monotonic() - start
|
||||
if not started and elapsed >= args.start_window:
|
||||
sys.exit(f"{SCAN_TASK} never started: it read {current['taskState']!r} "
|
||||
f"for {elapsed:.0f}s and the database took no writes. Either "
|
||||
"the POST queued nothing — a POST that arrives while a scan "
|
||||
"is already running cancels it, and every later 204 is "
|
||||
"accepted and discarded, so restart the service and trigger "
|
||||
"once — or the scan finished before this watch began, in "
|
||||
"which case ask with --after <journal mtime> instead.")
|
||||
if elapsed >= args.timeout:
|
||||
sys.exit(f"still busy after {args.timeout:.0f}s: {current}")
|
||||
time.sleep(args.interval)
|
||||
|
||||
if scan_task(args.url, token)["running"] and not args.force:
|
||||
sys.exit(f"{SCAN_TASK} is already running on {args.url}: a second "
|
||||
"POST /Library/Refresh cancels and restarts it, so the index "
|
||||
"looks frozen while you wait. Poll `wait` instead of triggering "
|
||||
"again (--force overrides).")
|
||||
|
||||
for attempt in range(args.retries + 1):
|
||||
code, body = request(args.url, token, "POST", "/Library/Refresh")
|
||||
if code == 204:
|
||||
print("refresh accepted (204); the scan walks every library")
|
||||
return
|
||||
if code == 429:
|
||||
if attempt < args.retries:
|
||||
time.sleep(args.retry_sleep)
|
||||
continue
|
||||
sys.exit(f"Jellyfin is busy: {args.retries + 1} attempts all answered "
|
||||
"429. Nothing was queued; run `wait`, then trigger again.")
|
||||
sys.exit(f"POST /Library/Refresh -> HTTP {code}: {body[:200]!r}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env bash
|
||||
# Trigger and poll ONE Plex library section — the step that makes a media move visible.
|
||||
#
|
||||
# media-refresh-plex.sh trigger --library KEY
|
||||
# media-refresh-plex.sh status --library KEY [--after EPOCH]
|
||||
#
|
||||
# One section only: refreshing all five costs a full walk of each and buys
|
||||
# nothing for a rename that touched one tree. And a rename on the array is
|
||||
# invisible until the section is rescanned — the server is bound at the SSD
|
||||
# metadata view, so an array-side move is not an event its watcher ever sees.
|
||||
#
|
||||
# Two host realities shape this script:
|
||||
# * PMS's HTTPD closes without a single byte on a request that arrives from
|
||||
# another netns (the host and the 10.89.x.x container address both read as
|
||||
# "empty reply"), and it serves TLS to remote clients. So the request rides
|
||||
# plain http://127.0.0.1:32400 INSIDE the container, shipped in as a file —
|
||||
# a three-layer ssh/podman/sh quoting chain only produces parse errors.
|
||||
# * The token is `PlexOnlineToken` in the container's Preferences.xml (not
|
||||
# PlexHTTPToken, which a claimed server may not carry). It is read inside
|
||||
# the container and used from a curl config there: never argv (any process
|
||||
# on the host reads /proc/<pid>/cmdline), never a URL (proxies log URLs).
|
||||
#
|
||||
# Progress needs no token and no container: `section_locations.scanned_at` in
|
||||
# the library database is the same epoch the API prints as `scannedAt`
|
||||
# (verified 2026-11-04), so `status` reads the database and compares it against
|
||||
# --after — which makes it a census of the last RECORDED scan, never a waiter: PMS
|
||||
# stamps that epoch when it accepts the request, so the door answers `settled` in the
|
||||
# same second its own `trigger` ran (measured 2026-11-04; no epoch survives it).
|
||||
# `trigger` answering 000 means PMS is still warming — retry, never
|
||||
# restart the service over it. Proof that a rename actually landed is
|
||||
# scripts/media-verify.py replaying the journal; this script only moves the
|
||||
# scanner along.
|
||||
set -euo pipefail
|
||||
|
||||
CONTAINER=plex
|
||||
URL="http://127.0.0.1:32400"
|
||||
PREFS='/config/Library/Application Support/Plex Media Server/Preferences.xml'
|
||||
DB="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
|
||||
SECTION=""
|
||||
AFTER=""
|
||||
TRIES=6
|
||||
WARM_SLEEP=10
|
||||
|
||||
die() { echo "$*" >&2; exit 1; }
|
||||
|
||||
CMD="${1:-}"; shift || true
|
||||
case "$CMD" in
|
||||
trigger|status) ;;
|
||||
*) die "usage: $(basename "$0") trigger|status --library KEY [--after EPOCH] [--container NAME] [--db PATH]" ;;
|
||||
esac
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--library) SECTION="${2:-}"; shift 2 ;;
|
||||
--after) AFTER="${2:-}"; shift 2 ;;
|
||||
--tries) TRIES="${2:-}"; shift 2 ;;
|
||||
--warm-sleep) WARM_SLEEP="${2:-}"; shift 2 ;;
|
||||
--container) CONTAINER="${2:-}"; shift 2 ;;
|
||||
--url) URL="${2:-}"; shift 2 ;;
|
||||
--prefs) PREFS="${2:-}"; shift 2 ;;
|
||||
--db) DB="${2:-}"; shift 2 ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$SECTION" ] || die "--library is required (the section id from GET /library/sections; Movies is 1)"
|
||||
command -v python3 >/dev/null || die "python3 is required to read the library database"
|
||||
|
||||
if [ "$CMD" = "status" ]; then
|
||||
[ -r "$DB" ] || die "no readable Plex library database at $DB"
|
||||
python3 - "$DB" "$SECTION" "$AFTER" <<'PY'
|
||||
import datetime, sqlite3, sys
|
||||
|
||||
db, section, after = sys.argv[1], int(sys.argv[2]), sys.argv[3]
|
||||
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
rows = con.execute(
|
||||
"SELECT root_path, available, scanned_at FROM section_locations "
|
||||
"WHERE library_section_id = ? ORDER BY id", (section,)).fetchall()
|
||||
if not rows:
|
||||
sys.exit(f"section {section} has no location in {db}; is the id right?")
|
||||
worst = min(r[2] or 0 for r in rows)
|
||||
for root, available, at in rows:
|
||||
iso = (datetime.datetime.fromtimestamp(at, datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
if at else "never")
|
||||
print(f" available={available} scanned_at={iso} {root}")
|
||||
print(f"section={section} scanned_at={worst}")
|
||||
if after:
|
||||
if worst < int(after):
|
||||
sys.exit(f"section {section} was last scanned before {after}; a refresh is still owed")
|
||||
PY
|
||||
exit 0
|
||||
fi
|
||||
|
||||
command -v podman >/dev/null || die "podman is required to reach PMS inside its container"
|
||||
|
||||
INNER="$(mktemp -t pxscan.XXXXXX)"
|
||||
trap 'rm -f "$INNER"' EXIT
|
||||
cat > "$INNER" <<'EOS'
|
||||
# Runs INSIDE the plex container: $1 section key, $2 base URL, $3 Preferences.xml.
|
||||
key="$1"; url="$2"; prefs="$3"
|
||||
[ -r "$prefs" ] || { echo "no readable $prefs inside the container" >&2; exit 3; }
|
||||
T=$(sed -n 's/.*PlexOnlineToken="\([^"]*\)".*/\1/p' "$prefs" | head -1)
|
||||
[ -n "$T" ] || { echo "no PlexOnlineToken in $prefs" >&2; exit 3; }
|
||||
cfg=$(mktemp)
|
||||
( umask 077
|
||||
{ printf 'header = "X-Plex-Token: %s"\n' "$T"
|
||||
printf 'silent\nshow-error\nrequest = "POST"\n'
|
||||
printf 'url = "%s/library/sections/%s/refresh"\n' "$url" "$key"; } > "$cfg" )
|
||||
code=$(curl -K "$cfg" -o /dev/null -w '%{http_code}' || true)
|
||||
shred -u "$cfg" 2>/dev/null || rm -f "$cfg"
|
||||
[ "$code" = "401" ] && { echo "section $key: HTTP 401 — the token in $prefs is not accepted" >&2; exit 4; }
|
||||
echo "$code"
|
||||
EOS
|
||||
|
||||
attempt=1
|
||||
while :; do
|
||||
podman cp "$INNER" "$CONTAINER:/tmp/media-refresh-plex.sh" \
|
||||
|| die "could not copy the runner into container $CONTAINER"
|
||||
code="$(podman exec "$CONTAINER" sh /tmp/media-refresh-plex.sh "$SECTION" "$URL" "$PREFS" || true)"
|
||||
podman exec "$CONTAINER" rm -f /tmp/media-refresh-plex.sh >/dev/null 2>&1 || true
|
||||
case "$code" in
|
||||
204|200) echo "refresh accepted ($code) for section $SECTION"; exit 0 ;;
|
||||
000)
|
||||
[ "$attempt" -ge "$TRIES" ] && die "PMS never answered after $TRIES attempts (000); it may still be warming — do not restart the service over it"
|
||||
echo "section $SECTION: no answer yet ($attempt/$TRIES), PMS is warming" >&2 ;;
|
||||
*) die "refresh of section $SECTION failed: HTTP ${code:-curl-error}" ;;
|
||||
esac
|
||||
attempt=$((attempt + 1))
|
||||
sleep "$WARM_SLEEP"
|
||||
done
|
||||
@@ -78,10 +78,16 @@ one journal is one undo:
|
||||
|
||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
|
||||
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
|
||||
--tvdb --tmdb --out /mnt/entropy/media-tv-plan.tsv \
|
||||
--park /mnt/entropy/media-park/TVShows
|
||||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
|
||||
--tvdb --tmdb --out /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||
--park /mnt/entropy/NeedsProcessing/TVShows
|
||||
python3 scripts/media-apply.py apply \
|
||||
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-tv-relayout.jsonl
|
||||
|
||||
Every plan, journal and log lives in /mnt/entropy/.media-campaign/ — the array
|
||||
root holds libraries only. media-apply.py defaults its journal to
|
||||
<root>/media-relayout.jsonl, so pass --journal explicitly or the run puts the
|
||||
log back on the root.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
|
||||
@@ -17,13 +17,20 @@ the executor's own path guard still has teeth, and give it its own journal —
|
||||
one journal is one undo, and a TV move must not be welded to a movie relayout:
|
||||
|
||||
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
|
||||
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
|
||||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||
--library /mnt/entropy/TVShows \
|
||||
--out /mnt/entropy/.media-campaign/media-tv-plan.tsv
|
||||
python3 scripts/media-apply.py apply \
|
||||
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl
|
||||
python3 scripts/media-apply.py verify \
|
||||
--plan /mnt/entropy/.media-campaign/media-tv-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl
|
||||
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||
--journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl --apply
|
||||
|
||||
Plans and journals live in `/mnt/entropy/.media-campaign/`, never loose on the
|
||||
array root: `media-apply.py` defaults its journal to `<root>/media-relayout.jsonl`,
|
||||
so `--journal` is always explicit here.
|
||||
|
||||
Guards, and why each one is here:
|
||||
|
||||
|
||||
@@ -0,0 +1,475 @@
|
||||
#!/usr/bin/env python3
|
||||
"""media-tvdupes.py — find duplicate TV episodes and plan their staging, one
|
||||
EPISODE at a time.
|
||||
|
||||
A TV library grows the same show into several folders: a tagged season box, an
|
||||
untagged packager drop, a `Complete` folder that is not complete. The folders
|
||||
look like copies of each other and are not, so the unit of comparison is the
|
||||
episode and never the folder. A file is droppable only when EVERY episode it
|
||||
carries is also held, somewhere else in the same show, by a file at least as
|
||||
large — Frank's rule: the same episode in two places keeps the larger one. A
|
||||
file that holds one episode nothing else has stays, whatever else it carries.
|
||||
|
||||
It writes a plan and moves nothing; `scripts/media-apply.py` is the only thing
|
||||
that renames. The plan stages into a dated hold, so the pass is undoable from
|
||||
the journal and frees no space until the hold is purged.
|
||||
|
||||
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||
--hold /mnt/entropy/.AGENTTRASH/20261102-tv-duplicates \
|
||||
--out /mnt/entropy/.media-campaign/tv-dupes-plan.tsv
|
||||
python3 scripts/media-apply.py apply \
|
||||
--plan /mnt/entropy/.media-campaign/tv-dupes-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/tv-dupes.jsonl \
|
||||
--metadata-view /var/lib/media-meta/view --apply
|
||||
python3 scripts/media-apply.py verify \
|
||||
--plan /mnt/entropy/.media-campaign/tv-dupes-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/tv-dupes.jsonl
|
||||
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||
--journal /mnt/entropy/.media-campaign/tv-dupes.jsonl --apply # empty folders only
|
||||
scripts/media-view.sh reap # ghost sidecars above
|
||||
|
||||
Then both servers have to be told: the plan moves files on the raw array, and no
|
||||
watcher crosses the metadata view — [[Media Relayout]] owns the refresh order.
|
||||
|
||||
Guards, and why each one is here:
|
||||
|
||||
* **Identity before geometry.** Two folders holding the same words are not the
|
||||
same show: the 2026-11-02 census found `Avatar: The Last Airbender` (1999,
|
||||
tvdb 74852) beside the 2024 remake (385925), `Star Trek: Voyager` beside
|
||||
`Star Trek: The Animated Series`, and a German thriller Jellyfin had titled
|
||||
`Millennium (2022) [tvdbid-472933]` beside `Millennium (1996)`. A folder's
|
||||
identity is read from three places, in this order: its `[tvdbid-N]` tag, the
|
||||
`tvshow.nfo` Jellyfin wrote — which lives in the metadata view, not on the
|
||||
array, and is what `--meta` reads — and a `folder<TAB>id` map passed with
|
||||
`--ids`. The tag outranks the derived layers: an id that contradicts a folder's
|
||||
own tag is reported and ignored, never unioned — measured 2026-11-02, Jellyfin's
|
||||
provider table carried a 2026 special's id on a folder tagged `[tvdbid-73838]`,
|
||||
and unioning them makes one show of the 2000 series and the special, whose bigger
|
||||
files then dominate the original's pilot. Two folders are one show when they share
|
||||
an id, or when neither has one and name and year both agree. Two disjoint id sets
|
||||
is an absolute stop: the pair is reported as a conflict and nothing under either is
|
||||
staged. Same name with nothing to decide on is reported unresolved — the fix is the
|
||||
tag, not a guess.
|
||||
* **A folder-level verdict destroys content.** 10 of the 25 shows that looked
|
||||
duplicated held COMPLEMENTARY halves: `Game of Thrones Complete` carries no
|
||||
S01 and no S08, `Modern Family S05` is the only copy of season 5, `Loiter
|
||||
Squad Uncensored` is the only copy of season 1. Judged per folder, 20 shows
|
||||
and some 340 GB of unique episodes would have been staged; judged per episode
|
||||
the true figure was a sixth of that.
|
||||
* **A bare number is never an episode.** `256491.mp4` may be an episode id, a
|
||||
poster, or a season, so a name that states no `SxxEyy` / `1x04` marker is
|
||||
unkeyed: never staged, and counted in the report. A media server's own episode
|
||||
table does know those files — pass it with `--witness` and its rows key
|
||||
whatever the server has matched.
|
||||
* **The audit re-reads the disk.** Sizes and the survivor's existence are
|
||||
checked again when the plan is written, because the census is a snapshot: a
|
||||
file that grew, vanished, or landed in the hold in between must fail the plan,
|
||||
not surprise the executor.
|
||||
* **A sidecar that shares an episode's stem follows it.** Subtitles and `.nfo`
|
||||
left at the old coordinates are ghost bait for `media-view.sh reap`.
|
||||
* A multi-episode file (`S01E01-02`) is dominated only when every episode in its
|
||||
range is held elsewhere — and it competes for the survivor seat on each of
|
||||
them, because the byte law compares files. A pack that states its range in
|
||||
the name is a name both media servers parse the same way, so the pack winning
|
||||
against two single-episode rips is the law applied, not a misread.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||
eng = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(eng)
|
||||
|
||||
# The executor's definition of a media file, plus the two the title engine omits:
|
||||
# an audit that disagrees with media-apply.py is a plan that dies at its
|
||||
# preflight instead of at the census.
|
||||
VIDEO = tuple(sorted(set(eng.VIDEO_SUFFIX) | {".ts", ".wmv"}))
|
||||
TAG_RE = re.compile(r"\[\s*tvdbid[\s._-]*(\d+)\s*\]", re.I)
|
||||
YEAR_RE = re.compile(r"[(\[]\s*((?:19|20)\d\d)\s*[)\]]")
|
||||
SEASON_EP = re.compile(r"(?i)\bs(?:eason)?[\s._-]*(\d{1,2})[\s._-]*e(?:p(?:isode)?)?"
|
||||
r"[\s._-]*(\d{1,3})(?:[\s._-]*(?:-|to|\+)[\s._-]*e?"
|
||||
r"[\s._-]*(\d{1,3}))?")
|
||||
NXN = re.compile(r"(?i)(?<!\d)(\d{1,2})x(\d{1,3})(?!\d)")
|
||||
|
||||
|
||||
def parse_args():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--library", default="/mnt/entropy/TVShows",
|
||||
help="the TV library to census (never written)")
|
||||
ap.add_argument("--hold", required=True,
|
||||
help="dated trash the plan stages into, e.g. "
|
||||
"/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates")
|
||||
ap.add_argument("--out", default=None,
|
||||
help="plan TSV for scripts/media-apply.py (default: "
|
||||
"tv-dupes-plan.tsv beside --library)")
|
||||
ap.add_argument("--witness", default=None,
|
||||
help="TSV of path<TAB>season<TAB>episode out of a media "
|
||||
"server's episode table; keys files whose names carry"
|
||||
" no marker")
|
||||
ap.add_argument("--meta", default=None,
|
||||
help="the merged metadata view (on the host "
|
||||
"/var/lib/media-meta/view): tvshow.nfo lives in the"
|
||||
" upper layer, so this is where an untagged folder's"
|
||||
" tvdb id is readable")
|
||||
ap.add_argument("--ids", default=None,
|
||||
help="TSV of folder<TAB>tvdbid, one line per folder, for"
|
||||
" identity a lookup settled and the folder name does"
|
||||
" not state")
|
||||
ap.add_argument("--json", default=None,
|
||||
help="also dump the analysis as JSON")
|
||||
return ap.parse_args()
|
||||
|
||||
|
||||
ARGS = parse_args()
|
||||
|
||||
|
||||
def episode_keys(name):
|
||||
"""Every (season, episode) an episode name states.
|
||||
|
||||
`S01E03`, `Season 1 E3`, `1x04`, and a range `S01E01-02`, which expands so a
|
||||
two-in-one file is only droppable when both halves are held elsewhere. A
|
||||
bare number states nothing and yields nothing."""
|
||||
keys = set()
|
||||
for m in SEASON_EP.finditer(name):
|
||||
s, first = int(m.group(1)), int(m.group(2))
|
||||
keys.add((s, first))
|
||||
if m.group(3):
|
||||
keys.update((s, e) for e in range(first + 1, int(m.group(3)) + 1))
|
||||
keys.update((int(m.group(1)), int(m.group(2))) for m in NXN.finditer(name))
|
||||
return keys
|
||||
|
||||
|
||||
def folder_identity(folder):
|
||||
"""(fold_sep name, year, tvdb ids) for a top-level show folder.
|
||||
|
||||
Tag and year come in both orders — `Name (2008) [tvdbid-84859]` and `Name
|
||||
[tvdbid-84859] (2008)` are both in the library — so neither is assumed to
|
||||
sit at the end, and both are cut out of the name folders compare on."""
|
||||
ids = frozenset(int(x) for x in TAG_RE.findall(folder))
|
||||
year = None
|
||||
for _ in range(3):
|
||||
y = YEAR_RE.search(folder)
|
||||
if not y:
|
||||
break
|
||||
year = year or y.group(1)
|
||||
folder = TAG_RE.sub(" ", f"{folder[:y.start()]} {folder[y.end():]}")
|
||||
return eng.fold_sep(TAG_RE.sub(" ", folder)), year, ids
|
||||
|
||||
|
||||
def nfo_id(path):
|
||||
"""The tvdb id a `tvshow.nfo` states, or None.
|
||||
|
||||
Three shapes are in the library: Jellyfin's `<uniqueid type="tvdb">`, Kodi's
|
||||
`<tvdbid>`, and a packager's `<id>` + `<id_type>` pair; some carry the id
|
||||
only in the title line as `[tvdbid-N]`."""
|
||||
try:
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
text = fh.read(16384)
|
||||
except OSError:
|
||||
return None
|
||||
for rx in (r"<tvdbid\s*>\s*(\d+)\s*<",
|
||||
r"<uniqueid[^>]*type\s*=\s*[\"']?tvdb[^>]*>\s*(\d+)\s*<",
|
||||
r"<id\s*>\s*(\d+)\s*</id>\s*<id_type>\s*(?:tvdb|thedotdb)",
|
||||
r"\[\s*tvdbid[\s._-]*(\d+)\s*\]"):
|
||||
m = re.search(rx, text, re.I)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def collect_ids(library, meta, ids_file):
|
||||
"""top-level folder -> {tvdb ids} from the nfo layer and an operator map.
|
||||
|
||||
`meta` is the merged metadata view: Jellyfin's sidecars live in the upper
|
||||
layer, so a folder's `tvshow.nfo` is not on the raw array. Reading the id
|
||||
from there is what makes an untagged packager folder identifiable at all —
|
||||
without it the pair stays unresolved and nothing under it is staged."""
|
||||
out = collections.defaultdict(set)
|
||||
if meta:
|
||||
try:
|
||||
tops = sorted(os.listdir(library))
|
||||
except OSError:
|
||||
tops = []
|
||||
for folder in tops:
|
||||
i = nfo_id(os.path.join(meta, folder, "tvshow.nfo"))
|
||||
if i:
|
||||
out[folder].add(i)
|
||||
if ids_file:
|
||||
with open(ids_file, encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
cols = line.rstrip("\n").split("\t")
|
||||
if len(cols) < 2 or not cols[1].strip().isdigit():
|
||||
continue
|
||||
out[cols[0]].add(int(cols[1].strip()))
|
||||
return out
|
||||
|
||||
|
||||
def witness_rows(path):
|
||||
"""absolute path -> {(season, episode)} from a media server's episode table."""
|
||||
out = collections.defaultdict(set)
|
||||
if not path:
|
||||
return out
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
cols = line.rstrip("\n").split("\t")
|
||||
if len(cols) < 3 or not cols[0].startswith("/"):
|
||||
continue
|
||||
try:
|
||||
out[os.path.abspath(cols[0])].add((int(cols[1]), int(cols[2])))
|
||||
except ValueError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def sidecars(path):
|
||||
"""The files beside PATH carrying its exact stem — subtitles, nfo, artwork."""
|
||||
folder, stem = os.path.split(path)
|
||||
stem = os.path.splitext(stem)[0]
|
||||
out = []
|
||||
try:
|
||||
entries = sorted(os.listdir(folder))
|
||||
except OSError:
|
||||
return out
|
||||
for e in entries:
|
||||
p = os.path.join(folder, e)
|
||||
if p == path or os.path.islink(p) or not os.path.isfile(p):
|
||||
continue
|
||||
if os.path.splitext(e)[0] == stem:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def scan(library, hold, witness):
|
||||
"""Every video file under LIBRARY: relative path, size, episode keys.
|
||||
|
||||
A file keys from the witness when it has a row (a server matched it), and
|
||||
from its own name otherwise; with neither it is unkeyed."""
|
||||
files = []
|
||||
for dirpath, dirnames, filenames in os.walk(library):
|
||||
dirnames[:] = sorted(d for d in dirnames
|
||||
if not d.startswith(".") and d != "@eaDir"
|
||||
and os.path.abspath(os.path.join(dirpath, d)) != hold)
|
||||
for f in sorted(filenames):
|
||||
p = os.path.join(dirpath, f)
|
||||
if os.path.islink(p) or not f.lower().endswith(VIDEO):
|
||||
continue
|
||||
try:
|
||||
size = os.path.getsize(p)
|
||||
except OSError:
|
||||
continue
|
||||
files.append({"path": os.path.abspath(p),
|
||||
"rel": os.path.relpath(os.path.abspath(p), library),
|
||||
"size": size,
|
||||
"keys": set(witness.get(os.path.abspath(p))
|
||||
or episode_keys(os.path.splitext(f)[0]))})
|
||||
return files
|
||||
|
||||
|
||||
def show_groups(folders, extra_ids):
|
||||
"""(show groups, conflicts, unresolved, disagreements) over top-level folders.
|
||||
|
||||
A folder's id is its `[tvdbid-N]` tag; the nfo layer and `--ids` fill a folder
|
||||
only when it carries no tag. The rule, in order of trust: an id in common makes
|
||||
one show; two disjoint id sets are two shows, reported as a conflict and never
|
||||
merged; no id on either side merges only on name AND year (the 1999 miniseries
|
||||
and the 2015 series are not one show); one side ided and the other bare is
|
||||
unresolved, because the fix is the tag.
|
||||
|
||||
The derived layers never overrule a tag. Measured 2026-11-02: Jellyfin's
|
||||
provider table carried a 2026 special's tvdb id on the folder tagged
|
||||
`[tvdbid-73838]`, and unioning the two made one show of the 2000 series and the
|
||||
special — whose bigger files then dominate the original's pilot. A contradiction
|
||||
is reported instead, because the tag is the operator's word and the provider
|
||||
table is a guess that can be wrong.
|
||||
"""
|
||||
names = {f: folder_identity(f) for f in folders}
|
||||
tags = {f: set(names[f][2]) for f in folders}
|
||||
derived = {f: set(extra_ids.get(f, ())) for f in folders}
|
||||
disagree = {f: (sorted(tags[f]), sorted(derived[f] - tags[f]))
|
||||
for f in folders if tags[f] and derived[f] - tags[f]}
|
||||
ids = {f: (tags[f] or derived[f]) for f in folders}
|
||||
parent = {f: f for f in folders}
|
||||
|
||||
def find(f):
|
||||
while parent[f] != f:
|
||||
parent[f] = parent[parent[f]]
|
||||
f = parent[f]
|
||||
return f
|
||||
|
||||
conflicts, unresolved = [], []
|
||||
order = sorted(folders)
|
||||
for i, a in enumerate(order):
|
||||
for b in order[i + 1:]:
|
||||
na, ya, _ia = names[a]
|
||||
nb, yb, _ib = names[b]
|
||||
if na != nb:
|
||||
continue
|
||||
ia, ib = ids[a], ids[b]
|
||||
if ia & ib:
|
||||
parent[find(b)] = find(a)
|
||||
elif ia and ib:
|
||||
conflicts.append((a, b, f"tvdb {sorted(ia)} vs {sorted(ib)}"))
|
||||
elif not ia and not ib and ya and ya == yb:
|
||||
parent[find(b)] = find(a)
|
||||
elif not ia and not ib:
|
||||
conflicts.append((a, b,
|
||||
f"same name, years {ya or '?'} vs {yb or '?'}"))
|
||||
else:
|
||||
unresolved.append((a, b, "one folder carries a tvdb id,"
|
||||
" the other none"))
|
||||
groups = collections.defaultdict(list)
|
||||
for f in order:
|
||||
groups[find(f)].append(f)
|
||||
return sorted(groups.values()), conflicts, unresolved, disagree
|
||||
|
||||
|
||||
def rank(rec):
|
||||
"""Survivor order: biggest, then shortest path, then lexicographic.
|
||||
|
||||
A strict total order per key means a key's winner is never itself dominated,
|
||||
so no chain of staging can cycle and exactly one copy of a byte-identical
|
||||
pair is staged."""
|
||||
return (-rec["size"], len(rec["path"]), rec["path"])
|
||||
|
||||
|
||||
def main():
|
||||
library = os.path.abspath(ARGS.library)
|
||||
hold = os.path.abspath(ARGS.hold)
|
||||
out = ARGS.out or os.path.join(os.path.dirname(library), "tv-dupes-plan.tsv")
|
||||
if not os.path.isdir(library):
|
||||
sys.exit(f"refused: {library} is not a directory")
|
||||
if hold == library or hold.startswith(library + os.sep):
|
||||
sys.exit(f"refused: --hold {hold} sits inside the library; a hold the"
|
||||
" census walks is a plan that eats its own output")
|
||||
|
||||
files = scan(library, hold, witness_rows(ARGS.witness))
|
||||
folders = sorted({r["rel"].split(os.sep)[0] for r in files})
|
||||
ids = collect_ids(library, ARGS.meta, ARGS.ids)
|
||||
groups, conflicts, unresolved, disagree = show_groups(folders, ids)
|
||||
|
||||
by_show = collections.defaultdict(list)
|
||||
for members in groups:
|
||||
for r in files:
|
||||
if r["rel"].split(os.sep)[0] in members:
|
||||
by_show[members[0]].append(r)
|
||||
|
||||
rows, audits, report, shows_json = [], [], [], []
|
||||
for show in sorted(by_show):
|
||||
recs = by_show[show]
|
||||
holders = collections.defaultdict(list)
|
||||
for r in recs:
|
||||
for k in r["keys"]:
|
||||
holders[k].append(r)
|
||||
winner = {k: min(v, key=rank) for k, v in holders.items()}
|
||||
dominated = [r for r in recs if r["keys"]
|
||||
and all(winner[k] is not r for k in r["keys"])]
|
||||
if not dominated:
|
||||
continue
|
||||
dup_keys = sorted({k for r in dominated for k in r["keys"]})
|
||||
sole = sum(1 for v in holders.values() if len(v) == 1)
|
||||
staged_folders = {os.path.dirname(r["rel"]) for r in dominated}
|
||||
for r in dominated:
|
||||
for k in r["keys"]:
|
||||
t = winner[k]
|
||||
try:
|
||||
now_src = os.path.getsize(r["path"])
|
||||
now_twin = os.path.getsize(t["path"])
|
||||
except OSError:
|
||||
audits.append(f"gone since the census: {r['rel']}"
|
||||
f" or its survivor {t['rel']}")
|
||||
break
|
||||
if now_twin < now_src:
|
||||
audits.append(f"survivor smaller on disk: {r['rel']}"
|
||||
f" {now_src} > {t['rel']} {now_twin}")
|
||||
for r in dominated:
|
||||
dst = os.path.join(hold, r["rel"])
|
||||
rows.append((r["path"], dst))
|
||||
for side in sidecars(r["path"]):
|
||||
rows.append((side, os.path.join(hold, os.path.relpath(side, library))))
|
||||
shows_json.append({"show": show, "files": len(dominated),
|
||||
"bytes": sum(r["size"] for r in dominated),
|
||||
"episodes": len(dup_keys),
|
||||
"sole-episode files left": sole,
|
||||
"folders": sorted(staged_folders)})
|
||||
n_ep, n_folder = len(dup_keys), len(staged_folders)
|
||||
report.append(
|
||||
f"{show}: stage {len(dominated)} files"
|
||||
f" ({sum(r['size'] for r in dominated) / 2**30:.1f} GiB) — {n_ep}"
|
||||
f" episode{n_ep != 1 and 's' or ''} held twice or more; {sole}"
|
||||
f" episode{sole != 1 and 's' or ''} exist in one copy only, so"
|
||||
f" {n_folder} folder{n_folder != 1 and 's' or ''} keep their files")
|
||||
|
||||
rows = sorted(set(rows))
|
||||
dests = set()
|
||||
for src, dst in rows:
|
||||
if dst in dests:
|
||||
audits.append(f"destination twice: {dst}")
|
||||
dests.add(dst)
|
||||
if os.path.exists(dst):
|
||||
audits.append(f"hold target occupied: {dst}")
|
||||
if not os.path.isfile(src):
|
||||
audits.append(f"source missing: {src}")
|
||||
|
||||
unkeyed = collections.Counter(os.path.dirname(r["rel"]) for r in files
|
||||
if not r["keys"])
|
||||
if ARGS.json:
|
||||
with open(ARGS.json, "w") as fh:
|
||||
json.dump({"shows": shows_json,
|
||||
"conflicts": [list(c) for c in conflicts],
|
||||
"unresolved": [list(u) for u in unresolved],
|
||||
"disagreement": {f: [t, d] for f, (t, d) in disagree.items()},
|
||||
"unkeyed": dict(unkeyed)}, fh, indent=1)
|
||||
|
||||
for line in report:
|
||||
print(line)
|
||||
for a, b, why in conflicts:
|
||||
print(f"CONFLICT (never merged): {a} | {b} — {why}")
|
||||
for a, b, why in unresolved:
|
||||
print(f"UNRESOLVED (tag it, then re-run): {a} | {b} — {why}")
|
||||
for f, (tagged, extra) in sorted(disagree.items()):
|
||||
print(f"DISAGREE (the tag wins): {f} — tag tvdb {tagged},"
|
||||
f" derived tvdb {extra}")
|
||||
if unkeyed:
|
||||
print(f"unkeyed, never staged: {sum(unkeyed.values())} files in"
|
||||
f" {len(unkeyed)} folders name no SxxEyy and have no --witness row;"
|
||||
" most: " + ", ".join(f"{d} ({n})" for d, n in unkeyed.most_common(5)))
|
||||
|
||||
# The gate runs before the plan is written: a plan file that exists is a plan
|
||||
# somebody will point the executor at, so a failed audit leaves no artifact.
|
||||
if audits:
|
||||
print(f"AUDIT FAILED ({len(audits)})", file=sys.stderr)
|
||||
for a in audits[:10]:
|
||||
print(f" {a}", file=sys.stderr)
|
||||
print(f"refused: no plan written to {out}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
mkdirs = sorted({os.path.dirname(dst) for _s, dst in rows},
|
||||
key=lambda p: (p.count(os.sep), p))
|
||||
with open(out, "w") as fh:
|
||||
fh.write("action\tsrc\tdst\tkind\n")
|
||||
for d in mkdirs:
|
||||
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||
for src, dst in rows:
|
||||
fh.write(f"move\t{src}\t{dst}\ttvdup\n")
|
||||
|
||||
held = sum(os.path.getsize(s) for s, _d in rows if os.path.isfile(s))
|
||||
print(f"plan: {len(rows)} moves ({held / 2**30:.1f} GiB) into"
|
||||
f" {len(mkdirs)} hold folders -> {out}")
|
||||
print("audit: every staged file has a survivor in the same show holding each of"
|
||||
" its episodes at >= its size on disk, one destination each")
|
||||
if rows:
|
||||
root = os.path.dirname(library)
|
||||
print(f"next: python3 scripts/media-apply.py apply --plan {out} --root"
|
||||
f" {root} --journal {os.path.join(root, 'tv-dupes.jsonl')}"
|
||||
" --metadata-view /var/lib/media-meta/view (dry run; add --apply)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -60,7 +60,7 @@ die() {
|
||||
}
|
||||
|
||||
usage() {
|
||||
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
|
||||
printf 'usage: %s plan|ensure-base|verify|install|reap|status [options]\n' "$0" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
@@ -98,6 +98,22 @@ require_upper_base() {
|
||||
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
||||
}
|
||||
|
||||
# The SSD placement of the metadata layer is a human choice, so `install` refuses
|
||||
# to invent a base directory: a mistyped --upper-base must not quietly create a
|
||||
# tree on whatever filesystem the caller landed on. The deployer owns the
|
||||
# documented location, so it calls this first — measured 2026-10-31, when a
|
||||
# fresh-host `install` died with "put it on the SSD first" and nothing in git
|
||||
# would have created it.
|
||||
do_ensure_base() {
|
||||
if [ -d "$upper_base" ]; then
|
||||
printf 'base: %s exists\n' "$upper_base"
|
||||
return 0
|
||||
fi
|
||||
install -d -m 0755 "$upper_base" 2>/dev/null \
|
||||
|| die "refused: cannot create $upper_base — it lives outside the checkout, run this as root"
|
||||
printf 'base: created %s\n' "$upper_base"
|
||||
}
|
||||
|
||||
do_plan() {
|
||||
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
||||
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
||||
@@ -234,6 +250,7 @@ do_status() {
|
||||
case "$cmd" in
|
||||
plan) do_plan ;;
|
||||
verify) do_verify ;;
|
||||
ensure-base) do_ensure_base ;;
|
||||
install) do_install ;;
|
||||
reap) do_reap ;;
|
||||
status) do_status ;;
|
||||
|
||||
@@ -104,6 +104,12 @@ for path in "${CHANGED[@]}"; do
|
||||
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
|
||||
add_test tests/media_overlay.bats
|
||||
;;
|
||||
# The Audiobookshelf bootstrapper (account + library + scan over the API)
|
||||
# and the deployer hook that runs it after the health gate.
|
||||
scripts/audiobookshelf-init.sh|tests/audiobookshelf_init.bats)
|
||||
add_test tests/audiobookshelf_init.bats
|
||||
add_test tests/deployment_flow.bats
|
||||
;;
|
||||
shared/homepage/*)
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/homepage_config.bats
|
||||
@@ -334,7 +340,7 @@ for path in "${CHANGED[@]}"; do
|
||||
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
||||
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
||||
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
||||
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*)
|
||||
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*|dankserver/audiobookshelf/*)
|
||||
add_test tests/dankserver_quadlet.bats
|
||||
add_test tests/deployment_flow.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
|
||||
@@ -68,8 +68,8 @@ service on the other host.
|
||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||
|
||||
The shared chat template maps to the inference service on the selected target.
|
||||
The `frank-npu` endpoint is retired () and has no deploy handler
|
||||
an unrelated large-model deployment never touches it.
|
||||
The `frank-npu` endpoint is retired and has no deploy handler, so an unrelated
|
||||
large-model deployment never touches it.
|
||||
|
||||
## Caddy safety
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ This repository tests its configuration at defined seams. The tests run with `ba
|
||||
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
|
||||
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
|
||||
| E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] |
|
||||
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
|
||||
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{audiobookshelf_init,anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
|
||||
|
||||
## Focused pre-commit testing
|
||||
|
||||
|
||||
@@ -222,5 +222,5 @@ harness bundle and measured the day of the flip:
|
||||
| Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
|
||||
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
|
||||
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
|
||||
| A consumer rejects `wilwheaton` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
|
||||
| A consumer rejects `wilwheaton` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed. In OpenWebUI the `browser-kokoro` choice is that user's own `user.settings` row (`ui.audio.tts.engine`) and the SPA re-serves it on every load, so flipping the dropdown does not stick; see [[OpenWebUI]] |
|
||||
---
|
||||
@@ -21,7 +21,7 @@ https://github.com/open-webui/open-webui
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) |
|
||||
| Deployed build | `0.11.4` — `GET http://127.0.0.1:3005/api/version` → `{"version":"0.11.4"}` (2026-10-18). `:main` is the rolling image, not a stable pin, so the running build lags upstream: releases `0.12.0` (2026-10-10) through `0.12.3` (2026-10-16) exist and are not deployed (https://github.com/open-webui/open-webui/releases). The 2026-09-21 `0.11.4` security/access-control advisory is what pinned this era of the page. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
|
||||
| Deployed build | `0.12.5` — `GET http://127.0.0.1:3005/api/version` → `{"version":"0.12.5"}` (re-measured 2026-11-04). The repo pin `@sha256:0b399a8b…` (`dankclaw/open-webui/quadlet/open-webui.container`, bumped 2026-10-30) is what makes that build reproducible; `:main` is a rolling tag, so re-measure `/api/version` instead of quoting a build from this page. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Backends | The picker's content is `webui.db` state, not repo state: as recorded on 2026-10-07 it exposes **one** model, `frank-zealot`, with `frank-dragoon`/`frank-archon` configured but disabled — *unverified 2026-10-18* (the DB is the authority and is not readable from the repo). What the repo does fix is the container's own OpenAI target: `OPENAI_API_BASE_URL=http://192.168.0.143:7770` (`dankclaw/open-webui/quadlet/open-webui.container:32`, same value in the live container env) — see [The LLM backends](#the-llm-backends) |
|
||||
@@ -109,6 +109,39 @@ The Ollama connection is disabled (`ollama.enable=false`).
|
||||
> connection here only hides it from this picker — the endpoint itself keeps
|
||||
> serving every other client (the harness, Hermes, API consumers).
|
||||
|
||||
## Default model, descriptions, and the admin API door
|
||||
|
||||
**The default model is `ui.default_models`** — one row of the config table, a
|
||||
comma-separated **string**, not a list. Live value `frank-archon` since
|
||||
2026-11-04 (per Frank). The front end resolves the picker as *the browser's own
|
||||
stored pick, else `config.default_models.split(",")`*, so the value decides what
|
||||
a fresh profile or a cleared-storage session lands on; after a user picks
|
||||
something themselves, their pick keeps winning. Verified 2026-11-04 in a clean
|
||||
browser: signed in with `localStorage` empty → the pill reads
|
||||
`Selected model: frank-archon`.
|
||||
|
||||
Two admin-token doors, both driven against loopback `:3005` — service-to-service
|
||||
traffic never rides the public vhost:
|
||||
|
||||
- **Config:** `POST /api/v1/configs/import` with
|
||||
`{"config":{"ui.default_models":"frank-archon"}}` — the general per-key setter
|
||||
(no `PATCH /configs/{key}` route exists in 0.12.5; verified against
|
||||
`routers/` + `main.py`), and it returns the whole config so the write reads
|
||||
back in the same call.
|
||||
- **A model row:** `POST /api/v1/models/model/update` with a full `ModelForm`
|
||||
(`id`, `name`, `meta`, `params`, `is_active`). **`meta` and `params` REPLACE,
|
||||
they do not merge:** read the row first, change one field, send everything
|
||||
back — a partial payload silently drops `meta.capabilities` and the
|
||||
profile/background images.
|
||||
|
||||
**An API key is not accepted everywhere.** Routes guarded by `get_human_user`
|
||||
(`/api/v1/auths/`, `utils/auth.py`) answer **403 `A human session is required.`**
|
||||
for any token whose `typ` is not `session` — that is not a dead key. The two
|
||||
mutation doors above take `get_admin_user` / `get_verified_user`, which do accept
|
||||
the key. Prove a key on `GET /api/v1/models` with **no trailing slash**:
|
||||
`/api/v1/models/` falls through to the SPA and answers `200` + `text/html`, which
|
||||
looks like success and parses as nothing.
|
||||
|
||||
## Container introspection (DB and backend python)
|
||||
|
||||
- Working dir for imports: `/app/backend`; package name `open_webui`
|
||||
@@ -163,14 +196,11 @@ account under Profile → API Keys). Anonymous web visitors get the `pending`
|
||||
role (`ui.default_user_role=pending`), which cannot chat. Verified
|
||||
2026-10-07 from loopback `:3005` and over the public vhost:
|
||||
`GET /api/v1/models/list`, `GET /api/v1/models` and
|
||||
`POST /api/chat/completions` all return **401** with no key (also with a
|
||||
bogus bearer token); only `/api/config`, `/api/version` and `/health`
|
||||
`POST /api/chat/completions` all return **401** with no key (also with a bogus bearer
|
||||
token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of
|
||||
them expose the model list. An external program needs an account + API key — there is
|
||||
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`
|
||||
`/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and
|
||||
`POST /api/chat/completions` → 401; `/api/config` additionally publishes
|
||||
no keyless access. Re-measured 2026-11-04 from loopback: `/api/version` → `0.12.5`,
|
||||
and `/api/config` publishes
|
||||
`features.enable_login_form=false`, `features.enable_signup=false` and
|
||||
`oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside.
|
||||
|
||||
@@ -219,18 +249,26 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
|
||||
into the existing local account with the same email — the pre-SSO admin
|
||||
(frank8465@protonmail.com) keeps its data. `ENABLE_OAUTH_SIGNUP=true` lets
|
||||
other `openwebui_users` create accounts on first login.
|
||||
- **First login mints role `pending`.** `ui.default_user_role=pending` (config
|
||||
DB, verified 2026-10-09), so a freshly-provisioned `openwebui_users` account
|
||||
completes SSO and then cannot chat (picker empty) until its role is flipped:
|
||||
`UPDATE "user" SET role='user' WHERE email='<user>@franksdank.cloud';` —
|
||||
run with host `python3` over ssh against
|
||||
- **First login mints the role from the claim, not from
|
||||
`ui.default_user_role`.** The quadlet sets `ENABLE_OAUTH_ROLE_MANAGEMENT=true`
|
||||
and `OAUTH_ROLES_CLAIM` defaults to `roles`, so the `roles: ["user"]` claim
|
||||
decides the role. Measured 2026-11-04: a freshly-provisioned
|
||||
`openwebui_users` account completed SSO in a clean browser and the row
|
||||
appeared as `role='user'` — chat answered and Read Aloud synthesized, with no
|
||||
manual step. `ui.default_user_role=pending` (config DB) is the fallback for an
|
||||
account the claim does *not* describe, and 0.12.5's `get_user_role` **refuses
|
||||
the login with 403** when role management is on and no roles claim is
|
||||
readable — so "SSO succeeded but I cannot chat" points at a missing
|
||||
`openwebui_users` group/entitlement, not at a row waiting to be flipped. Flip
|
||||
only a row that is actually stuck:
|
||||
`UPDATE "user" SET role='user' WHERE email='<user>@franksdank.cloud';` — run
|
||||
with host `python3` over ssh against
|
||||
`/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
|
||||
Dankclaw; table is `user`, singular; no `active` column). No per-user model
|
||||
grant is needed: the picker model `frank-zealot` carries the global
|
||||
`access_grant` row `(model, frank-zealot, user, *, read)`, so any
|
||||
`role='user'` account sees it. Flip on FIRST-LOGIN SIGHTING — the row does
|
||||
not exist until the account signs in (2026-10-09: five provisioned accounts
|
||||
had no row).
|
||||
Dankclaw; table is `user`, singular; no `active` column). The row does not
|
||||
exist until the account signs in, so census it after a first-login sighting
|
||||
(2026-10-09: five provisioned accounts had no row). No per-user model grant is
|
||||
needed: the picker models carry the global `access_grant` row
|
||||
`(model, <id>, user, *, read)`, so any `role='user'` account sees them.
|
||||
- **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
|
||||
cannot reach `authentik-server:9000` by name. It reaches the IdP over the
|
||||
public URL `https://auth.franksdank.cloud` — Caddy publishes `0.0.0.0:443`
|
||||
@@ -305,28 +343,58 @@ port is the rootless quadlet `frank-tts` (this repo:
|
||||
400 listing the voices that exist, never a silent fallback.
|
||||
|
||||
**The 29-voice `af_* … bm_fable` list is NOT frank-tts.** That enum belongs
|
||||
to **Kokoro.js running in the browser**: the deployed front end (0.11.4)
|
||||
ships an in-page Kokoro implementation (chunk carrying a frozen voice table
|
||||
`{af_heart:{name:"Heart",…}}` and `_validate_voice`, model
|
||||
`onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
|
||||
setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
|
||||
`browser-kokoro` is chosen the page synthesizes locally and rejects any
|
||||
voice outside its fixed pack — so `wilwheaton` fails with
|
||||
`Voice: "wilwheaton" not found. Should be one of: af_heart, …` **without a
|
||||
single request reaching the server**. Personal audio settings are
|
||||
browser: no `user.settings` row and no container env carries an audio
|
||||
override, yet the toast still appears ⇒ the choice lives in that browser's
|
||||
own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**,
|
||||
which hands synthesis to the admin config (frank-tts); picking `af_heart`
|
||||
instead keeps Kokoro, and Kokoro cannot clone — fixed pack only.
|
||||
to **Kokoro.js running in the browser**: the deployed front end ships an
|
||||
in-page Kokoro implementation (frozen voice table plus `_validate_voice`,
|
||||
model `onnx-community/Kokoro-82M-v1.0-ONNX`, and a personal *Kokoro.js dtype*
|
||||
setting). Its engine select takes `""` (Default) or `browser-kokoro`; with
|
||||
`browser-kokoro` the page synthesizes locally and rejects any voice outside
|
||||
its fixed pack, so `wilwheaton` fails with `Voice: "wilwheaton" not found.
|
||||
Should be one of: af_heart, …` **without a single request leaving the
|
||||
browser**. Kokoro cannot clone — fixed pack only.
|
||||
|
||||
Verified 2026-10-18 from inside the container with the exact OpenWebUI body
|
||||
(`{"model":"frank-tts","input":…,"voice":"narrator1",
|
||||
"response_format":"mp3","speed":1.0}`) → **200, `audio/mpeg`, 7,629 B**,
|
||||
while `podman logs open-webui` recorded **zero** `audio/speech` requests for
|
||||
the failing click — that contrast is the instrument: a server-path failure
|
||||
leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves
|
||||
neither.
|
||||
**The switch is a DB row, not browser storage.** Personal audio settings are
|
||||
the `settings` JSON column of table `user`, and in 0.12.x they nest under
|
||||
`ui`: `ui.audio.tts.engine`, `.voice`, `.defaultVoice`. In 0.11.x the same
|
||||
keys sat at the top level (`audio.tts.*`), and the 2026-10-18 probe listed
|
||||
only top-level keys, found no audio key, and concluded the choice lived in
|
||||
that browser's own storage — a probe-depth bug. The row carried the override
|
||||
the whole way; the 2026-10-30 pin bump to 0.12.5 moved the keys, not the bug.
|
||||
The SPA reads that row at load (`GET /api/v1/users/user/settings`) and posts
|
||||
the **whole** blob back whenever any preference changes
|
||||
(`POST …/settings/update`, body `{...settings}`). So the dropdown can fix it —
|
||||
but a second tab still holding the old copy writes `browser-kokoro` straight
|
||||
back, which is the loop that makes the toast return after it was "fixed". The
|
||||
user row is the authority here; the admin `config` table was never the problem
|
||||
(its `audio.tts.engine` has been `openai` throughout). Repair
|
||||
`settings.ui.audio.tts` with the image's stdlib `sqlite3` (`mode=ro`; quote
|
||||
identifiers, `user` and `group` are reserved words — there is no `sqlite3` CLI
|
||||
in the image or on the host), then reload every tab of that account.
|
||||
|
||||
**Repair the engine *and* the voice.** With engine `""` the front end still
|
||||
sends its stored voice in the request body, so a Kokoro-era `voice`/
|
||||
`defaultVoice` survives the engine fix as frank-tts's own rejection —
|
||||
measured 2026-11-04: `voice: "alloy"` → **400 `unknown voice 'alloy';
|
||||
available voices: dr.kern, garrickhagon, jeffhays, vaati, wilwheaton`**. Set
|
||||
`ui.audio.tts` to `{"engine": "", "voice": "wilwheaton", "defaultVoice":
|
||||
"wilwheaton"}` and reload. Engine `""` means "the admin config decides", and
|
||||
the handler fills `voice` from `audio.tts.voice` only when the body omits it.
|
||||
|
||||
**Two tells settle the case in one line:**
|
||||
|
||||
- a session that fetched `/wasm/ort-wasm-simd-threaded.jsep.wasm` loaded
|
||||
onnxruntime, so Kokoro.js is running in that browser — the server path
|
||||
never asks for that file;
|
||||
- a server-path failure leaves a uvicorn request line for
|
||||
`/api/v1/audio/speech` **and** a frank-tts log line; a `browser-kokoro`
|
||||
failure leaves neither. uvicorn logs every request, so silence is evidence.
|
||||
|
||||
Verified 2026-11-04: the handler's own outbound shape
|
||||
(`{"model":"frank-tts","input":…,"voice":"wilwheaton"}` with
|
||||
`Authorization: Bearer none`, the configured `audio.tts.openai.api_key`) →
|
||||
**200 `audio/wav`** — a body without `response_format` comes back WAV and
|
||||
either MIME is browser-playable, so OpenWebUI accepts it. The full route as a
|
||||
real user over the vhost, `POST /api/v1/audio/speech` → **200 `audio/mpeg`**,
|
||||
with frank-tts logging `speech voice=wilwheaton`.
|
||||
|
||||
## Retired instances
|
||||
|
||||
@@ -396,6 +464,21 @@ neither.
|
||||
app process env; the `.webui_secret_key` file happens to equal it — use it
|
||||
explicitly, `podman exec -e`).
|
||||
|
||||
- **2026-11-04 — decoration rows drifted again; default model set to
|
||||
`frank-archon`.** Two payload swaps had gone by without anyone touching the
|
||||
picker: `frank-dragoon`'s row still advertised halogen `0.16.2`, and
|
||||
`frank-zealot`'s still said `build 10777` / `3×262,144` / *MTP off* — live is
|
||||
`0.18.20`, `build 12118`, `3 × 200,704`, MTP armed. The prose is cosmetic; the
|
||||
`params.context.length` values had **swapped**, and that one bites:
|
||||
`frank-zealot` advertised 262,144 while the endpoint refuses anything past its
|
||||
200,704 slot, so a long zealot chat rides the endpoint's 400 instead of
|
||||
compacting, and `frank-dragoon` advertised 200,704 against a 262,144 ceiling,
|
||||
giving up 61k tokens of window for nothing. Both rows re-keyed through the API
|
||||
door in [Default model, descriptions, and the admin API door](#default-model-descriptions-and-the-admin-api-door),
|
||||
descriptions rewritten from the env files, and `ui.default_models` set to
|
||||
`frank-archon`. **Law: a model swap is not finished until that endpoint's
|
||||
`model` row is re-keyed as well — no deployer step touches `webui.db`.**
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
|
||||
@@ -0,0 +1,368 @@
|
||||
---
|
||||
type: reference
|
||||
description: "Audiobookshelf on Dankserver: quadlet shape, array bind, SSD state, API bootstrap, authentik SSO, public vhost, laws."
|
||||
tags:
|
||||
- dankserver
|
||||
- audiobooks
|
||||
- quadlet
|
||||
timestamp: 2026-11-03
|
||||
---
|
||||
|
||||
# Audiobookshelf
|
||||
|
||||
Audiobookshelf (ABS) serves the [[Entropy Volume]] audiobook library to browsers
|
||||
and phones: it scans `/mnt/entropy/Audiobooks`, keeps its own SQLite library with
|
||||
per-listener playback positions, and streams the audio. It is **not** a media
|
||||
server — it shares nothing with [[Plex]] or [[Jellyfin]] and owns no files on the
|
||||
array.
|
||||
|
||||
**Host:** Dankserver — **Unit:** `audiobookshelf.service` (rootless Quadlet, user manager) —
|
||||
**Doors:** LAN `http://192.168.0.235:8000/audiobookshelf` (plain HTTP, nothing in front) and public `https://audiobooks.franksdank.cloud/audiobookshelf/` — Caddy straight to the container, **exempt from the guard chain** because the app must answer non-browser clients ([[Caddy]] *Unguard a vhost*). The public name is the address every client carries: Pi-hole answers it with `192.168.0.143` on the LAN, Porkbun answers it with the WAN address outside, and Caddy serves a real certificate either way.
|
||||
|
||||
## Shape
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Unit file | `dankserver/audiobookshelf/quadlet/audiobookshelf.container` (the authority for every number here) |
|
||||
| Image | `ghcr.io/advplyr/audiobookshelf` pinned by **amd64 manifest digest** — see Laws |
|
||||
| Port | `Environment=PORT=8000` + `PublishPort=8000:8000` — web UI and API on one port |
|
||||
| Base path | `Environment=ROUTER_BASE_PATH=/audiobookshelf` — every URL below carries this prefix, and the shipped SPA cannot boot without it (Laws) |
|
||||
| Network | `pasta` (standalone; publishes nothing to other containers) |
|
||||
| Audio bind | `/mnt/entropy/Audiobooks:/audiobooks:ro` — read-only, by design |
|
||||
| State | `/home/dank/Containers/audiobookshelf/config` → `/config`, `…/metadata` → `/metadata` (both NVMe) |
|
||||
| Mount gate | `ExecStartPre=…/wait-for-mount.sh /mnt/entropy` |
|
||||
| Health | `wget` against `/healthcheck` inside the container, 30 s / 3 retries |
|
||||
| I/O share | `IOWeight=10` — a scan loses the array to every other tenant |
|
||||
| Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` |
|
||||
| Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` |
|
||||
| Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` |
|
||||
| Single sign-on | authentik provider `audiobookshelf`; the local `frank` account stays enabled as break-glass — see *Single sign-on* |
|
||||
| Library layout | one book per folder — ABS groups by the first directory holding a media file (see Laws); every move is journalled at `/mnt/entropy/.media-campaign/media-relayout-audiobooks.jsonl` |
|
||||
|
||||
## Bootstrap
|
||||
|
||||
`scripts/audiobookshelf-init.sh` — run by the deployer after the health gate, so a
|
||||
deployed ABS is a *usable* ABS: `/healthcheck` answers 200 on a server with no
|
||||
user, no library and no scan, and nothing else in the deploy notices the
|
||||
difference. The script is idempotent; re-running it costs one incremental scan.
|
||||
|
||||
| Step | Door | Printed |
|
||||
|------|------|---------|
|
||||
| credential | `~/.config/secrets/audiobookshelf` (mode 600, generated once, host-side) | `credential: created …` |
|
||||
| account | `POST /init` with `{"newRoot":{"username","password"}}` | `account: created (frank)` / `already registered` |
|
||||
| session | `POST /login` → bearer token | `token: obtained (root)` |
|
||||
| library | `POST /api/libraries`, folder `/audiobooks`, provider `audiobook` | `library: created Audiobooks (<id>)` / `exists` |
|
||||
| scan | `POST /api/libraries/<id>/scan` | `scan: accepted` |
|
||||
| proof | `GET /api/libraries/<id>/items?limit=1` | `items: <total>` |
|
||||
|
||||
Sign in at `http://192.168.0.235:8000/audiobookshelf` as `frank`; the password is
|
||||
`awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' ~/.config/secrets/audiobookshelf` on
|
||||
Dankserver. Change it in **Settings → Users → frank → Password**; the secret file
|
||||
then holds a stale value and only matters again if the `config` bind is ever
|
||||
destroyed (a fresh DB makes the script create a new owner from it).
|
||||
|
||||
That value is the live password, not a leftover: it bcrypt-matches `users.pash` in the state
|
||||
DB (2026-11-03), so `local` does open the door when authentik is down.
|
||||
|
||||
A full scan is slow and runs in the background; watch the `items:` total climb,
|
||||
never `/healthcheck`. It also reads files that are genuinely broken and logs
|
||||
`[AudioFileScanner] Invalid data found when processing input` per file — that is
|
||||
the file, not the bind: the same line count grows while the total climbs.
|
||||
|
||||
## Phone app
|
||||
|
||||
The official **Audiobookshelf** app connects with Server Address
|
||||
`https://audiobooks.franksdank.cloud/audiobookshelf` — the base path belongs in the address —
|
||||
username `frank`, and the same password as the web UI, or the SSO button. One address works
|
||||
at home and away: on the LAN Pi-hole answers that name with `192.168.0.143` and Caddy dials
|
||||
the container; off-LAN the same name resolves to the WAN address and takes the same route
|
||||
(inbound 443 is forwarded at the router — fetched from a datacenter node 2026-11-03). The raw
|
||||
LAN address still answers and is the break-glass door whenever Caddy is down. `/ping`
|
||||
(`{"success":true}`) and `/healthcheck` (`OK`) answer at the ROOT whatever `ROUTER_BASE_PATH`
|
||||
says, so the app's connectivity probe works before it resolves the base path at all, and
|
||||
`download: true` on the account is what makes offline listening work.
|
||||
|
||||
**A home device that bypasses Pi-hole cannot reach it.** The G36 does not hairpin: resolving
|
||||
the name to the WAN address from inside the LAN times out (measured 2026-11-03). Only a
|
||||
client pinned to an external resolver hits this — everything using the router's DNS gets the
|
||||
LAN answer.
|
||||
|
||||
**The login token is at `user.token`, not top level.** `POST /login` answers
|
||||
`{ user, serverSettings, ereaderDevices, Source, userDefaultLibraryId }` and the bearer sits
|
||||
inside the user object. A client that reads `response.token` gets the empty string, sends
|
||||
`Authorization: Bearer ` with nothing after it, and every `/api/*` call then answers `401`
|
||||
as though the password were wrong.
|
||||
|
||||
**`POST /api/login` is not a route.** Local login is `POST <base>/login`, and the `/api`
|
||||
bearer guard answers anything else with a bare `401 Unauthorized` while logging **nothing** —
|
||||
no `[LocalAuth]` line — so a wrong path is indistinguishable from a wrong password unless the
|
||||
journal is read too. Same family as the absent `/setup` route named in
|
||||
`scripts/audiobookshelf-init.sh`. To settle a suspected stale credential without signing in:
|
||||
bcrypt-compare the recorded value against `users.pash` in
|
||||
`/home/dank/Containers/audiobookshelf/config/absdatabase.sqlite`, opened read-only
|
||||
(`file:<path>?mode=ro`).
|
||||
|
||||
## Listeners (accounts)
|
||||
|
||||
The door is public now, so the account is the only gate between a listener and the library.
|
||||
Two ways in, both per listener:
|
||||
|
||||
- **Local account** — **Settings → Users → Add User**: type `user`, tick only the libraries
|
||||
they may hear, and leave `download` off unless they need offline listening. Hand them the
|
||||
server address above and their own password.
|
||||
- **Household SSO** — create the authentik account, and that is the whole of it: the first
|
||||
**Sign in with Authentik** auto-registers the ABS account (`authOpenIDAutoRegister`) as a
|
||||
listener, because the claim grants that role to every signer. To shut a listener off,
|
||||
disable or delete their user in Audiobookshelf — no IdP group membership can withhold it.
|
||||
|
||||
What auto-registration hands out (read off the state DB, 2026-11-03): `accessAllLibraries:
|
||||
true` and `download: true` — every household signer lands on the **whole** library with
|
||||
offline downloads already on. Narrow it per account in **Settings → Users → <name>**. The DB
|
||||
is the census when no admin token is at hand; `GET /api/users` returns the *limited* shape
|
||||
(no `active`, no `librariesAccessible`), so a missing key there is not a denial, and an
|
||||
account created by SSO has `pash` NULL with its IdP link at `extraData.authOpenIDSub`.
|
||||
|
||||
Nothing rate-limits `POST /audiobookshelf/login` — this Caddy build carries no `rate_limit`
|
||||
module and ABS 2.37.2 ships no lockout — so the form is a guessing target exactly like
|
||||
`jelly.`, `plex.` and the *arr logins already are. The mitigation is credential hygiene:
|
||||
long unique passwords per listener, never a reuse of the root password.
|
||||
|
||||
## Single sign-on (authentik)
|
||||
|
||||
Both doors take the household identity: the web UI puts **Sign in with Authentik** beside
|
||||
the local form, and the phone app offers SSO on its server-address screen. The local
|
||||
`frank` account stays enabled on purpose — `authActiveAuthMethods` is `local` **and**
|
||||
`openid`, and `local` is how you get in when the IdP is down.
|
||||
|
||||
| Piece | Owner |
|
||||
|-------|-------|
|
||||
| Provider `audiobookshelf`, the admin group, the scope mapping | `scripts/authentik-audiobookshelf-sso.sh` — re-run it after an authentik rebuild, because this state lives in the IdP's database and a blueprint re-import reverts it silently |
|
||||
| Client secret | `ABS_OIDC_CLIENT_SECRET` in `~/.config/secrets/audiobookshelf` on Dankserver (mode 600; never in the repo, a URL or an argv) |
|
||||
| The ABS half: issuer, the four IdP URLs, claim, auto-register | `scripts/audiobookshelf-init.sh`, armed automatically when that secret exists in the credential file |
|
||||
| Role at first SSO sign-in | the `abs_groups` claim: every signer gets `user`; `audiobookshelf_admins` adds `admin`. Nothing in the IdP can deny the listener role — the login itself is the gate (law since 2026-11-03; the `audiobookshelf_users` group was retired, it had zero members and every household account was being refused) |
|
||||
| Admin group members | `akadmin` (the operator's own authentik admin account — sign in as it and the claim grants site admin) and `qa-e2e-admin` (the identity the deploy gate signs in as, so the script re-adds it every run; drop it and the gate fails, not a phone) |
|
||||
| Guard state of the door | none since 2026-11-03: proxy provider `audiobooks-proxy` and application "Audiobookshelf Guard" were deleted and the embedded outpost un-claimed them, because that chain answers a native client with challenge HTML ([[Caddy]] *Unguard a vhost* for the order). The SSO relying party `audiobookshelf` above is a different object and stays |
|
||||
| Redirect URIs on the RP | `<base>/auth/openid/callback` **and** `<base>/auth/openid/mobile-redirect` for **both** origins (LAN address and public name): the app sends the mobile URI belonging to the door it was configured with, and STRICT matching answers an unregistered one with "Redirect URI Error" |
|
||||
|
||||
With `authOpenIDAutoRegister` on, a first sign-in creates the account and the claim alone
|
||||
decides its type — and the claim always carries `user`, so **the IdP is now the library gate**:
|
||||
any authentik account that can reach this application becomes a listener. That is the
|
||||
intent (the household signs in with the accounts it already has), and its breadth is
|
||||
accepted on purpose: no access policy is bound to the `audiobookshelf` application, and
|
||||
the wide auto-registered default (every library, downloads on) is **not a defect to fix** —
|
||||
Frank's call, 2026-11-03. Narrowing stays a per-account judgement in **Settings → Users**.
|
||||
The one thing still worth watching is authentik's own invitation/signup flow: it must
|
||||
remain something only Frank drives, because an account created there is a listener the
|
||||
moment it signs in. Revoking one person is a disable/delete of their ABS user; revoking
|
||||
everyone is that signup flow.
|
||||
|
||||
Both halves — authentik's provider row and ABS's settings — are asserted end to end by the Playwright gate named in *Verification*. `23-` signs in as `qa-e2e-admin` and must land inside the app with site-admin rights; `24-` signs in as the standard identity, which is in no Audiobookshelf group, and must land as a listener. Together they pin the escalation and the default: a mapping that stops granting silently locks the household out, and a mapping that over-grants turns the household into site admins.
|
||||
|
||||
## Laws
|
||||
|
||||
- **Speak the image's env names, not the README's.** The server reads `PORT`,
|
||||
`CONFIG_PATH`, `METADATA_PATH` and `ROUTER_BASE_PATH`; the `ABS_*` family printed
|
||||
by the server's own README is the npm package's naming and is **inert** in this
|
||||
image. Measured 2026-11-02 on the first deploy: with `ABS_PORT=8000` the server
|
||||
listened on `:80` under base path `/audiobookshelf`, so `systemctl` said
|
||||
`active (running)` while every probe of `:8000` was refused and the deploy gate
|
||||
failed at 180 s. Two seconds with the pinned digest settles it:
|
||||
`podman inspect --format '{{.Config.Env}}' <image>`.
|
||||
- **The URL carries `/audiobookshelf`, and that is not a style choice.** The SPA in
|
||||
`/app/client/dist/index.html` is built with `routerBasePath:"/audiobookshelf"`, so it asks
|
||||
for its assets under that prefix whatever the server is mounted at. Mounted at `/`, every
|
||||
script request falls through to the SPA fallback, comes back `text/html`, Chrome refuses to
|
||||
execute it, and the page never leaves "Loading…" — while `/healthcheck`, the API and the
|
||||
phone app stay perfectly healthy, which is how this hid behind an API-only verification
|
||||
(measured 2026-11-02). Prove a UI claim in a browser, never from `/health`.
|
||||
- **The OIDC redirect subfolder is a value you send, not a default you inherit.** The 2.17.4
|
||||
upgrade only fills `authOpenIDSubfolderForRedirectURLs` on a database that ALREADY had OIDC
|
||||
enabled, so a DB created before SSO leaves it unset and the server builds
|
||||
`http://host/undefined/auth/openid/callback`. authentik answers the authorize request with
|
||||
"Redirect URI Error" and nothing on the ABS side looks wrong (measured 2026-11-02). The
|
||||
provider's registered URI must equal `<origin><subfolder>/auth/openid/callback`, so each
|
||||
script derives the value from one variable of its own (`ABS_URL`, `ABS_BASE`) and
|
||||
`tests/audiobookshelf_init.bats` pins what reaches the wire.
|
||||
- **An ORM-created provider has no grant types.** authentik 2026.8 defaults `grant_types` to
|
||||
`[]`, and `/application/o/authorize/` then refuses the code flow with oauthlib's generic
|
||||
"The request is otherwise malformed" — the wire names nothing, the server log line that names
|
||||
it is `Invalid grant_type for provider`. Worse, the redirect-URI check runs *first*, so a
|
||||
provider carrying both faults shows only "Redirect URI Error" until the URI is fixed, and the
|
||||
second wall appears after the fix ships (measured 2026-11-02: the browser reached authentik,
|
||||
got `code`+`state`, and ABS still bounced to `/login?error=undefined`). The script sets
|
||||
`authorization_code` + `refresh_token`; a blueprint re-import reverts the row silently.
|
||||
- **The guarded door has to tell the truth about its scheme.** ABS validates every OIDC
|
||||
`callback` against the origin it believes it is served from — `getRequestOrigin()` is
|
||||
`x-forwarded-proto://Host` (`server/utils/requestUtils.js`, read in the pinned digest
|
||||
2026-11-03) — so behind a TLS edge that reaches the container over plain http it computes
|
||||
`http://audiobooks.franksdank.cloud` and answers its own https callback with a 400 "Invalid callback
|
||||
URL - must be same-origin" before the IdP is ever contacted. The Caddy `:8017` route pins
|
||||
`header_up X-Forwarded-Proto https` for exactly that reason ([[Caddy]] *Guard a new vhost*);
|
||||
the LAN door needs nothing, because there browser and server agree on `http`.
|
||||
- **The claim mapping is not the whole `property_mappings` list.** A provider holding only the
|
||||
custom mapping grants a token scoped shorter than the scopes ABS requests
|
||||
(`openid profile email abs_groups`): the code is issued and exchanged normally, and ABS's
|
||||
`/userinfo` call then dies `insufficient_scope (The request requires higher privileges than
|
||||
provided by the access token)` — a fault that reads like an ABS bug and is an IdP row. Attach
|
||||
the built-in OpenID `openid`/`profile`/`email` mappings beside the claim mapping, which is
|
||||
what every working RP on this IdP carries (measured 2026-11-02).
|
||||
- **With `authOpenIDMatchExistingBy` unset, an SSO identity never lands on an existing ABS
|
||||
account, and a name collision is a hard denial.** In that mode `findUserFromOpenIdUserInfo`
|
||||
matches on `sub` only, so every first sign-in auto-registers a NEW user; if the IdP username
|
||||
equals an ABS username — the local root `frank` is the one here — the create hits the unique
|
||||
constraint and the callback dies `Unauthorized`. So "create a `frank` user in authentik" does
|
||||
not reach the root account: either set `authOpenIDMatchExistingBy` to `username` (links the
|
||||
IdP account to the existing one, and `setUserGroup` refuses to downgrade a root — root plus an
|
||||
`admin` claim is allowed, root plus `user` is denied) or pick a non-colliding IdP name. Read
|
||||
off `server/models/User.js` and `server/auth/OidcAuthStrategy.js` in the pinned digest
|
||||
(2026-11-02).
|
||||
- **The audio bind stays `:ro`.** ABS writes nothing that belongs beside the
|
||||
audio: its library, covers, progress and sidecar metadata all land in
|
||||
`/metadata` on the SSD. A writable bind would put stray writes on the spindle
|
||||
and hand the array to a service with no reason to hold it. If per-book
|
||||
`.opf`/`.nfo` sidecars on disk are ever wanted, the rollback is one line —
|
||||
point the audio bind at the [[Media Metadata View]] the media servers already
|
||||
use — but that view's upper layer exists for those two servers' sidecars, so a
|
||||
third writer needs a reason first ([[Media Metadata View]]).
|
||||
- **Do not bind the raw array without the mount gate.** In a rootless unit
|
||||
`ConditionPathIsMountPoint=` is fail-*dead* and `RequiresMountsFor=` is inert,
|
||||
so a missing array does not stop the start: podman invents an empty directory,
|
||||
ABS indexes it, and an empty library becomes recorded truth. `wait-for-mount.sh`
|
||||
is the only gate that works here — same law the media servers obey
|
||||
([[Quadlet Lifecycle]]).
|
||||
- **The `HealthCmd` must use `wget`.** The image is Alpine and ships `wget`, not
|
||||
`curl`; a `curl` health command reads a perfectly healthy server as unhealthy
|
||||
and `Restart=always` turns that into a loop.
|
||||
- **The image pin is the amd64 manifest digest, not the index digest.**
|
||||
`podman inspect` reports the platform manifest (`RepoDigests[0]`) for what was
|
||||
pulled, which differs from the manifest-list digest `latest` points at. Pin the
|
||||
one this host runs. `latest` is newer than the newest version tag: upstream
|
||||
stopped publishing version tags after `2.19.0` while `latest` kept moving, so
|
||||
"highest version tag" is stale here — read the index, don't guess from tags.
|
||||
- **`IOWeight=10` is inert without the host drop-ins.** `dankserver/host/systemd/`
|
||||
carries `user.slice.d/io-accounting.conf` and the `user@1000` delegate, and the
|
||||
deployer installs both; without them systemd silently drops the weight
|
||||
([[Quadlet Lifecycle]]).
|
||||
- **No `systemctl --user enable`.** Generated Quadlet units cannot be enabled on
|
||||
this host; `WantedBy=default.target` in the unit file is what carries the
|
||||
service across a reboot ([[Quadlet Lifecycle]]).
|
||||
- **One door for every client, and it sits outside the guard chain on purpose.**
|
||||
`https://audiobooks.franksdank.cloud/audiobookshelf` is what browsers and apps use;
|
||||
`http://192.168.0.235:8000/audiobookshelf` is the raw LAN socket that same Caddy dials,
|
||||
so it doubles as break-glass when the proxy is down. Behind the chain (until
|
||||
2026-11-03) the vhost answered the phone app's `/ping` and `/socket.io/` handshake with
|
||||
Anubis' challenge HTML — Anubis demands a JavaScript proof and the outpost needs a
|
||||
session cookie, and a native client solves neither — so the app evaluated
|
||||
`'isInit' in e.data` on a non-object, reported "Failed to contact server", and the
|
||||
container's request log never recorded the attempt. Podcast RSS and OPDS readers died
|
||||
the same death. ABS's own auth is the gate, as it is for [[Plex]] and [[Jellyfin]]
|
||||
([[Caddy]] *Unguard a vhost*).
|
||||
- **There is no `/setup` route; the creation door is `POST /init`.** `/status`
|
||||
(no `/api` prefix) reports `isInit` = "a root user exists", `POST /init` with
|
||||
`{"newRoot":{"username","password"}}` creates it, `POST /login` returns the
|
||||
bearer every `/api/*` route wants. `POST /setup` answers `Unauthorized` — that
|
||||
is a missing route wearing an auth message. Read off `dist-server/server/Server.js`
|
||||
inside the pinned digest rather than from a forum.
|
||||
- **A 200 may carry plain text.** `res.sendStatus(200)` answers the body `OK`, and
|
||||
`/init` and the scan route use it; a client that parses every success as JSON
|
||||
crashes on the one call that matters (`tests/audiobookshelf_init.bats` pins it).
|
||||
- **The item count is not a field on the library DTO, and paging never ends
|
||||
itself.** The door is `GET /api/libraries/<id>/items?limit=1` → `total`; a poller
|
||||
that guesses a plausible name (`numItems`) prints `0` forever beside a scan that
|
||||
is working. Paging is on `page` (0-based) plus `limit`, and **`offset` is
|
||||
ignored**: `limit=500&offset=500` returns the same 500 ids as `offset=0`, while
|
||||
`page=1` shares none of them. A census that pages on `offset` therefore reads
|
||||
`ceil(total / limit) × limit` rows containing only `limit` distinct items — the
|
||||
exact origin of the "exactly 1,500" and "exactly 4,000" counts two runs
|
||||
reported for one library. Page on `page`, count *unique ids*, stop on `total`,
|
||||
and take the number after the pass settles (`total` moves during a scan).
|
||||
- **A fresh scan is the only thing that notices new audio.** `inotify` does not
|
||||
cross a rootless podman bind — measured for [[Jellyfin]], whose page owns that
|
||||
law — so the in-container watcher sees nothing a host process files onto the
|
||||
array, and the deployer's re-run of the bootstrapper is the cheap scan trigger.
|
||||
- **One stray media file at a collection root turns the whole collection into one
|
||||
book.** Every scan re-walks the tree from disk (`scanFolder` → `recurseFiles`),
|
||||
and `utils/scandir.js:groupFileItemsIntoLibraryItemDirs` assigns each file to the
|
||||
FIRST directory along its path that already has a group: the first file seen at a
|
||||
root owns every deeper file, and the folders below it never become items. One
|
||||
949 MB `.cbr` sitting at `_unsorted`'s root collapsed 652 book folders into a
|
||||
single item of 5,693 tracks, titled after whichever file supplied metadata.
|
||||
"Hundreds of books are missing" is a layout fault, not a scan fault.
|
||||
- **"Media file" means audio OR ebook, so an audio-only census cannot see the file
|
||||
that is hiding the library.** With `audiobooksOnly=false` (the deployed shape)
|
||||
the predicate is `SupportedAudioTypes` ∪ `SupportedEbookTypes`: 20 audio
|
||||
extensions (including `mp4`, `mpg`, `webm`) plus `epub pdf mobi azw3 cbr cbz`. A
|
||||
census reporting "that root holds no loose audio" can be correct and still
|
||||
useless — the file doing the shadowing here was a comic archive.
|
||||
- **Deleting the covering item does not un-shadow anything.** Grouping is recomputed
|
||||
from disk, so the next scan rebuilds the item from the same stray file (measured:
|
||||
`DELETE`, then scan → `1 Added`, the item back at the same path). The fix is the
|
||||
MOVE: give every stray root file a folder of its own (one file per folder is
|
||||
ABS's own single-book layout; formats of one book share a folder), through the
|
||||
journal-guarded mover, then delete the stale item so the old path is not kept,
|
||||
then scan.
|
||||
- **`?force=1` is neither a discovery arm nor a repair arm.** Both arms of
|
||||
`POST /libraries/:id/scan` walk the disk, so finding new folders needs neither
|
||||
and the deployer's cheap trigger stays on the plain arm. Force additionally
|
||||
re-scans every item that already exists (measured 1:33 against 0:04 on ~4,600
|
||||
items) and repaired nothing on a library whose residuals were file-level: item
|
||||
count, path set and media-less count identical before and after.
|
||||
- **`DELETE /api/items/:id` is a DB row; `?hard=1` is `fs.remove()`.** In
|
||||
`LibraryItemController.delete`, `hardDelete = req.query.hard == 1` and then
|
||||
`fs.remove(libraryItemPath)`. Pointed at a covering item, that path is the whole
|
||||
swallowed subtree — `hard=1` there deletes every book beneath it off the array.
|
||||
Assert the item's `path` first, and never pass `hard`.
|
||||
- **`Invalid data found when processing input` names the file, not the bind.** The
|
||||
scanner logs it per file and skips that file; a book whose files ALL fail never
|
||||
becomes an item at all, so the disk-vs-DB folder census shows it as a book with
|
||||
no item. That is corrupt media, not a lost book. Never clear a suspect file with
|
||||
a lenient probe: `ffprobe -show_entries format=duration` happily returns three
|
||||
hours for the WAV bytes wearing a `.mp3` name that the scanner refuses. Read the
|
||||
log's file paths instead, and `stat -c %s` first — 0-byte files are the common
|
||||
case, and a truncated download reports `moov atom not found`.
|
||||
- **A `Disc 01` / `CD 1` subfolder is part of a book, not a book.** The extension
|
||||
census counts those folders; ABS deliberately does not — its grouping attaches
|
||||
their files to the ancestor item (measured: 110 such folders under one item
|
||||
carrying all 192 tracks). Read the basename before believing a disk-vs-DB gap.
|
||||
- **`.cbr`/`.cbz` group like media and never play.** They sit in the grouping
|
||||
predicate, so they hide books, but the comic parser throws `ERAR_BAD_ARCHIVE`
|
||||
(`parseComicMetadata` → `CbrComicBookExtractor`) and the item lands with no
|
||||
media. A media-less row beside those lines is the file, not the scanner.
|
||||
- **A scan is finished when a NEW scan uuid completes.** Match
|
||||
`Library scan "<uuid>" completed in ...`. `podman logs --since 30m` is a sliding
|
||||
window: the set of visible completion lines changes on its own as old lines age
|
||||
out, so a poller keyed on that hash reads a mid-scan number and calls it done.
|
||||
|
||||
## Verification
|
||||
|
||||
`./tests/run.sh` covers this unit through `tests/dankserver_quadlet.bats`
|
||||
(boot persistence, the SSD state roots, the `:ro` array bind, the mount gate, the `wget`
|
||||
health check, the base path the shipped SPA needs), `tests/audiobookshelf_init.bats` (the
|
||||
bootstrapper against a stub of the API above: create-once, idempotent re-run, no secret in a
|
||||
URL, refusal by URL when the server does not answer, the deployer calling it only after the
|
||||
health gate, and the SSO arm sending the mounted base path as the redirect subfolder);
|
||||
`tests/media_overlay.bats` (only [[Plex]] and [[Jellyfin]] may reference the metadata view);
|
||||
and the live Playwright gates `tests/e2e/specs/23-audiobookshelf-sso.spec.ts` and
|
||||
`tests/e2e/specs/24-audiobookshelf-sso-default-listener.spec.ts`, run by `tests/e2e/run.sh` at
|
||||
the end of every `deploy.sh`. `23-` (the `admin` project) signs in as `qa-e2e-admin` against
|
||||
`http://192.168.0.235:8000/audiobookshelf` and asserts the two things no stub can see: every
|
||||
`.js` the app loads arrives as JavaScript (a bundle built for the wrong base path arrives as
|
||||
`text/html`, which boots nothing), and `/api/me` answers 200 with `type: "admin"` — true only
|
||||
while grant types, the OpenID scope mappings and the group claim all hold together. `24-` (the
|
||||
`standard` project) signs in as `qa-e2e`, an account in no Audiobookshelf group, and must come
|
||||
back `type: "user"`: the default grant that lets the household in, so a scope-mapping
|
||||
regression surfaces as a failed sign-in in the gate instead of as a friend's complaint.
|
||||
|
||||
The public door is pinned by the guard audit `tests/e2e/specs/14-guard.spec.ts`, which now
|
||||
carries an **exemption** assertion where it used to carry a guarded one: an anonymous raw
|
||||
client must get the app's own `{"success":true}` on `/ping`, a machine-format socket
|
||||
handshake, and the app's own `401` on `/audiobookshelf/api/libraries` — never challenge
|
||||
HTML. `tests/anubis.bats` pins the same shape in the Caddyfile: the vhost never proxies to
|
||||
`anubis:8923`, and the post-auth listener carries no `@audiobooks` route.
|
||||
|
||||
Five book folders hold media no scanner can read, so they stay out of the library:
|
||||
three 2024 Nebula nominees in the `_unsorted` box (two `.m4b` with no `moov` atom,
|
||||
one 36-file `.mp3` set that is not MPEG audio at all), `Anthony Heald/Bounty
|
||||
Hunter Wars - Book 1` and `Star Wars Full Cast Audio Drama/Dark Empire III` — the
|
||||
latter two also exist in readable form under their author folders. Fix by
|
||||
re-acquiring the files; the scanner names every one of them in its log.
|
||||
@@ -66,3 +66,24 @@ exactly describes the year this bug survived.
|
||||
volume absent they start `active (healthy)` against empty directories, which is
|
||||
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
||||
[[Storage Policy]] for what may be written here.
|
||||
|
||||
## Write doors
|
||||
|
||||
Something renamed movie folders on this array with no journal in any of the
|
||||
`/mnt/entropy/*.jsonl` plans, no bash history, no cron, no timer, and no job
|
||||
artifact. The census of things that can mutate it:
|
||||
|
||||
- **A local process** as `dank` or root. `scripts/media-apply.py` is the only
|
||||
one that journals; anything else (a hand `mv`, another session's script) is
|
||||
silent.
|
||||
- **SFTP by a human.** GNOME Files on a workstation mounts the array through
|
||||
`gvfsd-sftp`, which runs `sftp-server` on this host as `dank`. Renames and
|
||||
moves are direct `rename()` calls: no shell, no history, no journal. A folder
|
||||
rename bumps **only that directory's ctime** — its mtime and every child's
|
||||
ctime stay at their old values, so the files look untouched while their paths
|
||||
are gone. Diagnose it with `pgrep -af gvfsd-sftp` on the workstation and
|
||||
`ss -tnp` for the peer.
|
||||
- **Copyparty is not a door**: the bind is `ro` at the mount level, so no HTTP
|
||||
or WebDAV write reaches the array regardless of account flags ([[Copyparty]]).
|
||||
- **No SMB or NFS server exists here** — the `*arr` stack on FrankBook-Pro
|
||||
cannot reach the array at all, so it cannot rename, import, or rescan it.
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Jellyfin
|
||||
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
|
||||
tags: [jellyfin, dankserver, media-server]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
timestamp: 2026-11-03T11:40:00Z
|
||||
---
|
||||
|
||||
# Jellyfin
|
||||
@@ -18,7 +18,7 @@ clients.
|
||||
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
||||
|
||||
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
|
||||
## Facts (the repo quadlet — deployed as the live unit on 2026-10-31)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
||||
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
||||
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
||||
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
|
||||
| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**, live since the 2026-10-31 cutover: sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched |
|
||||
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
||||
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
||||
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
||||
@@ -37,26 +37,143 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
||||
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
||||
/var/lib/media-meta/view -> /data (git: the overlay view)
|
||||
/mnt/entropy -> /data (live today: the array, read-only)
|
||||
/mnt/entropy -> /data (the overlay's lower layer; also the rollback bind)
|
||||
```
|
||||
|
||||
Library folders point at `/data` paths: stable across container recreation and
|
||||
identical to the raw array layout below. Which side of that path the container
|
||||
is on is the open question on this host — the installed unit binds the read-only
|
||||
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
|
||||
sees is the [[Media Metadata View]] overlay, live since the 2026-10-31 cutover:
|
||||
`/var/lib/media-meta/view:/data`, read-write.
|
||||
|
||||
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
||||
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
|
||||
written once on the NVMe upper layer are files [[Plex]] can read too. While
|
||||
`/data` is bound read-only every item update logs the error below and no sidecar
|
||||
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
|
||||
that raises the view.
|
||||
reason the view exists: `movie.nfo`, `tvshow.nfo` and `episode.nfo` written once
|
||||
on the NVMe upper layer are files [[Plex]] can read too — and since the
|
||||
2026-10-31 cutover that is what happens: Jellyfin is the fleet's only metadata
|
||||
writer, and every sidecar it saves is an SSD byte the array never sees.
|
||||
**Artwork is not one of those sidecars.** No image saver is bound for any
|
||||
library, so posters and fanart go to the internal store
|
||||
(`config/metadata/library/<xx>/<item-id>/poster.jpg`), which [[Plex]] never
|
||||
reads — movie art has never existed beside the media on either layer.
|
||||
|
||||
**Library coverage is the first thing to check when a title is missing.** The
|
||||
four libraries are `movies → /data/Movies`, `tvshows → /data/TVShows`,
|
||||
`music → /data/Music`, `boxsets → /config/data/collections`
|
||||
(`GET /Library/VirtualFolders`). The review pile — `/data/NeedsProcessing`, the
|
||||
former `media-park`, drained into it 2026-11-02 — is in **none** of them, so
|
||||
everything filed there is invisible to Jellyfin by configuration,
|
||||
not by fault: a recursive refresh answers 204, burns CPU for minutes and
|
||||
indexes nothing, because it never looks there. Check coverage before re-scanning.
|
||||
|
||||
**Which read proves an item is indexed: the folder view, never the recursive type
|
||||
list.** `GET /Items?IncludeItemTypes=Movie&Recursive=true` answered 2,101 items on
|
||||
2026-11-02 while `GET /Items?ParentId=<Movies ItemId>` listed 2,548 children — every
|
||||
folder on disk — and the eight folders the recursive read "lost" (three `Gundam 00
|
||||
Special Edition`, five `Sailor Moon Memorial`) each answer as `Type: Movie` through
|
||||
`SearchTerm` and sit in the folder listing. The two reads are not interchangeable, so
|
||||
the recursive list is a wrong instrument for "did the refresh index what I moved": its
|
||||
shortfall is grouping, not missing media, and acting on it starts a scan that fixes
|
||||
nothing. Verify per item — `ParentId` listing or `SearchTerm` — and use the recursive
|
||||
count only as a trend line.
|
||||
|
||||
**Failure signature:** `System.IO.IOException: Read-only file system :
|
||||
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
||||
instead of the view. On Dankserver that is today's expected state — the view has
|
||||
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
|
||||
same line is a fault: the quadlet drifted or the mount died.
|
||||
instead of the view — always a fault on this host since the 2026-10-31 cutover.
|
||||
`scripts/media-view.sh status` says which: the quadlet drifted back to the raw
|
||||
array, or the mount died.
|
||||
|
||||
**`EnableInternetProviders` false turns a library into a silent file index.**
|
||||
Every refresh answers `204`, stores nothing and logs nothing — no provider ids,
|
||||
no plot, no rating, no art — while the item count reads like a healthy scan. The
|
||||
switch lives in `config/root/default/<Library>/options.xml`, Jellyfin 12's name
|
||||
for the per-library option file: grepping for the old `library.xml` concludes "no
|
||||
per-library options exist", and the database holds none, so the trail dies there.
|
||||
Two more traps in the same file: `TypeOptions/ImageFetchers` is the *enabled* set,
|
||||
so an empty `<ImageFetchers />` switches art off while `ImageFetcherOrder` stays
|
||||
fully populated — order is not enablement — and the populated `<MetadataFetchers>`
|
||||
built next to it makes the library look healthy.
|
||||
|
||||
**Live posture, measured 2026-11-02** from `GET /Library/VirtualFolders`, cross-checked
|
||||
against each library's `config/root/default/<Library>/options.xml`. The switches are NOT
|
||||
top-level DTO keys: the folder carries `Name`, `CollectionType`, `Locations`, `ItemId`,
|
||||
`PrimaryImageItemId`, `RefreshStatus` and `LibraryOptions`, and every per-library option
|
||||
sits inside `LibraryOptions`. Reading `EnableInternetProviders` off the folder itself
|
||||
answers `None` for all four — the camelCase trap one level deeper, and it looks exactly
|
||||
like an unconfigured library.
|
||||
|
||||
| Library | `CollectionType` | `EnableInternetProviders` | `EnableRealtimeMonitor` |
|
||||
|---|---|---|---|
|
||||
| `Movies` | movies | true (fixed 2026-11-02) | **false** |
|
||||
| `Shows` | tvshows | true (fixed 2026-11-02) | true |
|
||||
| `Music` | music | **false** | true |
|
||||
| `Collections` | boxsets | **false** | false |
|
||||
|
||||
`Music` and `Collections` are silent file indexes in the sense above: a refresh answers
|
||||
`204`, stores nothing, logs nothing. Any metadata or artwork work on either one
|
||||
starts by flipping the switch, never by firing a scan. `Shows` sat in that set
|
||||
until 2026-11-02; the door is `POST /Library/VirtualFolders/LibraryOptions` with
|
||||
the library's own `ItemId` (below), and `scripts/jellyfin-providers.sh verify` is
|
||||
the one command that says which libraries are still in it.
|
||||
|
||||
**The per-library option door on 12.x is `POST /Library/VirtualFolders/LibraryOptions`**
|
||||
— `GET` there answers `405`, which is how the route announces itself. The 10.x doors
|
||||
are gone: `/Library/VirtualFolders/Update` and `/Library/VirtualFolders/Refresh`
|
||||
answer `404` on this build, so every recipe naming them fails outright.
|
||||
|
||||
**A library record can be a phantom.** A per-library option capture taken 2026-11-01
|
||||
carried seven entries — `Shows2`, `Shows3`, `Music2` among them, each with
|
||||
`Locations: []` and a live-looking `ItemId` — while `GET /Library/VirtualFolders` the
|
||||
same day listed four. A numeric suffix on a library name is what a deleted and
|
||||
recreated library leaves behind. The live set is the entries whose `Locations` are
|
||||
non-empty; count those and never the records.
|
||||
|
||||
**`RemoteImages` is the call that separates a provider fault from a library
|
||||
fault.** `GET /Items/{id}/RemoteImages?Type=Primary` returns the provider's real
|
||||
poster list even with every automatic pass switched off — plugin, key and CDN all
|
||||
proven by that one response — so a refresh that stores nothing beside it is a
|
||||
library-switch problem, never a provider problem. It is also the delivery door:
|
||||
`POST /Items/{id}/RemoteImages/Download` saves the chosen image to the internal
|
||||
store on a build where the automatic image pass will not run at all — it is how
|
||||
the Movies library got its art: 2,515 of 2,564 films carry a primary now, and the
|
||||
rest have none in TMDb (shorts like `Astartes`, files whose names match nothing).
|
||||
|
||||
**The Download door's exact wire shape:** the list response carries the address in
|
||||
`Url` — there is no `ImageUrl` field — and the POST takes camelCase query params,
|
||||
`?type=Primary&imageUrl=<Url>`, answering `204`; the stored tag then appears in
|
||||
`ImageTags`. PascalCase params, or an `imageUrl` left empty, answer `500`, which
|
||||
reads as a server fault and is a field-name fault.
|
||||
|
||||
**`RemoteImages` answers empty for an item with no `ProviderIds`** — no provider id,
|
||||
nothing to look up. Art coverage is capped by identification, not by the image door:
|
||||
the items a campaign returns nothing for are the unidentified ones, and no image
|
||||
retry will change that.
|
||||
|
||||
**A `Download` pass reports zero only when it is pinned to one provider.** A TMDb-pinned
|
||||
campaign over `Shows` classified every item in 20 s, downloaded nothing, and concluded that
|
||||
identification was the only lever left. The same door queried with no `Providers=` filter
|
||||
answered for 16 seasons that same day — every offer from `TheTVDB`, which the pinned form
|
||||
never asks — and all 16 are dressed now. A pinned answer proves one provider is empty; the
|
||||
unpinned answer is the provider census, and a TV art loop that pins is simply the wrong tool.
|
||||
Post-fill the artless remainder measures 86 items (80 seasons, 6 series): 61 carry no provider
|
||||
id at all — the unnamed long tail (`3a BBW - Series (2010-11)`, `History Of Africa With Zeinab
|
||||
Badawi`, `(Warhammer TV)`-shaped directories), where the lever is folder names or an identify
|
||||
pass — and 25 hold an id and draw no offer from any provider. No image retry moves either
|
||||
group, which is the half of the old conclusion that survives.
|
||||
|
||||
**Series and season art is fetched; episode stills are not, deliberately.** Every
|
||||
episode query is a fresh provider round-trip — nothing caches per episode — measured
|
||||
at ~11 s per item beside ~2 s per series on the same host, with one hit in five in a
|
||||
sample. Across 42k episodes that is days of provider traffic to dress a fifth of them,
|
||||
and clients display the series and season art anyway. If episode stills are ever
|
||||
ordered, the loop is the same door and its log makes the pass resumable.
|
||||
|
||||
**API shape on 12.x:** `/Items/{id}` needs `UserId` when authenticated by API key — without
|
||||
it the answer is `400 Error processing request`, which reads exactly like an authentication
|
||||
failure, and it bites hardest as a *verification* read directly after a write that succeeded.
|
||||
The `/Items` list route, `?Ids=<id>` included, needs no `UserId`, so verify writes there.
|
||||
`MetadataRefreshMode`
|
||||
accepts `Default` (and `Realtime`) only; `Full` and `ValidationFailure` are
|
||||
rejected outright, and what forces the providers to re-run is
|
||||
`ReplaceAllMetadata=true`.
|
||||
|
||||
## Ports & access
|
||||
|
||||
@@ -71,7 +188,7 @@ Client access paths and the public-exposure policy (auth wall required before
|
||||
any public vhost; Plex-style claim tokens do not exist here — Jellyfin uses its
|
||||
own users/API keys) follow the fleet [[Authentik]]/[[Anubis]] chain pattern.
|
||||
|
||||
## Authentik LDAP authentication (2026-09-25)
|
||||
## Authentik LDAP authentication (live since 2026-09-25, re-verified 2026-11-04)
|
||||
|
||||
Jellyfin has no native external-auth, so it authenticates users through the
|
||||
**LDAP-Auth plugin** (`jellyfin-plugin-ldapauth`) against an [[Authentik]] LDAP
|
||||
@@ -87,17 +204,35 @@ user to verify the password.
|
||||
publishing `389`/`636` on the Dankclaw host for cross-host binds.
|
||||
- Service account `svc-jellyfin-ldap` (DN
|
||||
`cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`).
|
||||
- Access groups `jellyfin_users` (search filter
|
||||
`(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`) and
|
||||
`jellyfin_admins` (admin filter `(cn=jellyfin_admins)`). Add a user to
|
||||
`jellyfin_users` to grant them Jellyfin access; `jellyfin_admins` grants
|
||||
the Jellyfin admin role.
|
||||
- Access group `jellyfin_users`: the plugin's search filter is
|
||||
`(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`, so a user
|
||||
who is not in that group is **invisible** to the plugin and the login fails
|
||||
as "Wrong username or password". Add members with
|
||||
`Group.objects.get(name="jellyfin_users").users.add(user)` in `ak shell`
|
||||
(directory data — the DB owns it, not this repo).
|
||||
`jellyfin_admins` is **inert as configured** — see the admin-filter law.
|
||||
|
||||
**Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth):
|
||||
- Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN
|
||||
`dc=franksdank,dc=cloud`, bind user = the service-account DN, search
|
||||
attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`
|
||||
`CreateUsersFromLdap=true`, all folders enabled.
|
||||
- Admin base `ou=groups,dc=franksdank,dc=cloud`, admin filter
|
||||
`(cn=jellyfin_admins)`, `EnableLdapAdminFilterMemberUid=false`.
|
||||
|
||||
**Law — the admin filter grants nothing unless it can return the user's own
|
||||
DN.** With `EnableLdapAdminFilterMemberUid=false` the plugin searches
|
||||
`LdapAdminBaseDn` with `LdapAdminFilter` and sets the admin permission only when
|
||||
a returned entry's DN equals the bound user's DN
|
||||
(`LDAP-Auth/LDAPAuthenticationProviderPlugin.cs`). A group-OU base returns the
|
||||
*group* entry, whose DN never matches, so **no LDAP user can ever become a
|
||||
Jellyfin admin** — measured 2026-11-04: the search answers
|
||||
`cn=jellyfin_admins,ou=groups,…` for every login. Working shape if group-granted
|
||||
admins are ever wanted: base `dc=franksdank,dc=cloud` +
|
||||
`(memberOf=cn=jellyfin_admins,ou=groups,dc=franksdank,dc=cloud)`. Deliberate
|
||||
today: the server's own admin account stays **local**, which is the break-glass
|
||||
door when authentik or the outpost is down. A *malformed* admin filter is not
|
||||
inert — the plugin throws for every login, so this knob is all-or-nothing.
|
||||
|
||||
**Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is
|
||||
verified by authentik's default authentication flow (standard-password
|
||||
@@ -107,10 +242,23 @@ password and the bind fails with `flow error password: Invalid`. The
|
||||
service-account password is stored only in the Jellyfin plugin config (the
|
||||
`LdapBindPassword` field) — it is never committed to this repo.
|
||||
|
||||
**Verification (2026-09-25):** `ldapsearch` bind + the `jellyfin_users`
|
||||
search both succeed from Dankclaw; a real user (`alex`, in `jellyfin_users`)
|
||||
logged into the web UI with their authentik password and the plugin created
|
||||
the matching Jellyfin user.
|
||||
**Verification (re-taken 2026-11-04, from Dankserver):** the container reaches
|
||||
`192.168.0.143:389`; the service-account bind plus the `jellyfin_users` search
|
||||
return every member; LDAP-Auth 23.0.0.0 loads on Jellyfin 12.4; the `alex` row
|
||||
still carries
|
||||
`AuthenticationProviderId=Jellyfin.Plugin.LDAP_Auth.LdapAuthenticationProviderPlugin`.
|
||||
|
||||
**Law — "the authentik login doesn't work" has two opposite causes; read the
|
||||
Users table before touching authentik.** Copy `config/data/jellyfin.db` with its
|
||||
`-wal`/`-shm` and read it read-only (never sqlite the live file):
|
||||
`Users.AuthenticationProviderId` names the door — `…Users.DefaultAuthentication
|
||||
Provider` = local password, `…LDAP_Auth.LdapAuthenticationProviderPlugin` =
|
||||
authentik. Because the LDAP search is scoped to `jellyfin_users`, a name that
|
||||
exists only as a local account is rejected by the plugin and Jellyfin then
|
||||
silently falls back to the local provider: same UI message, opposite fix. The
|
||||
instrument for "did a bind ever reach authentik" is the **event registry**, never
|
||||
the outpost journal — see [[Authentik Outpost Reference]].
|
||||
|
||||
**Gotcha — the login endpoint changed in Jellyfin 12.1.** The legacy
|
||||
`POST /Users/Auth/Password` route (and its `LegacyEmbyRouteRewriteMiddleware`)
|
||||
was removed; probing it now returns a silent 401 with no plugin log, which
|
||||
@@ -164,9 +312,17 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
||||
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
||||
`community_supported: false`), so it needs **no PIN** — the optional
|
||||
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
||||
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
|
||||
identifies everything. Enabling TVDB as a provider would re-fetch show
|
||||
metadata across the library, so it is a deliberate, separate change.
|
||||
- **TVDB is installed but is not a metadata agent**: TMDb still identifies
|
||||
everything. Enabling TVDB as a provider would re-fetch show metadata across
|
||||
the library, so it is a deliberate, separate change.
|
||||
- **A plugin owns its own config file, under its own root, and omits null
|
||||
settings** — so the element an installer wants to fill may simply not exist,
|
||||
and its root is `<PluginConfiguration>`, not `<Configuration>`. TVDB's file had
|
||||
no `<ApiKey>` element at all, which is why `install` refused with
|
||||
`no place to write <ApiKey>` from October until the anchor fix: the secret sat
|
||||
valid in `provider-keys` the whole time, and `verify`'s `MISSING key` was this
|
||||
fault, not a missing key. The writer now anchors on the file's own root and
|
||||
still refuses by name when a file has none.
|
||||
- **The deployer places the keys before it restarts jellyfin**
|
||||
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
||||
failing when `provider-keys` is absent, so a host without it still deploys.
|
||||
@@ -184,45 +340,148 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
||||
|
||||
## Triggering a scan
|
||||
|
||||
The scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same value is
|
||||
`JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a mode-600 curl
|
||||
config, never a header argument, in the `MediaBrowser` form: 12.4 authenticates
|
||||
only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token`
|
||||
for every key, valid or not. Before believing a 401, prove the key against the
|
||||
server's own table — `select count(*) from ApiKeys where AccessToken = ?` on
|
||||
`jellyfin.db` opened `mode=ro&immutable=1`. The backslashes below matter:
|
||||
Run the driver from the deploy checkout: `scripts/media-refresh-jellyfin.py
|
||||
trigger`, then `… wait`. It calls the library settled only on the rule below — the
|
||||
task seen `Running` (or a run that finished while it watched) first, then
|
||||
`--require-quiet` reads with no scan and no writes — and refuses by name when the
|
||||
scan never started, because a watcher that saw no `Running` would otherwise report
|
||||
the pre-scan library as the fresh one. `… status --after <the trigger's epoch>` is the
|
||||
"has the scan reached my moves" answer, read from the task's `EndTimeUtc` — and that
|
||||
timestamp only advances when a scan *finishes*, so the epoch must be one your trigger
|
||||
could not have followed: a journal mtime is already passed by whatever scan ran before
|
||||
you asked, and bare `status` cannot tell that walk from yours, while `wait` refuses such
|
||||
an epoch by name. The Plex door is worse still — [[Plex]].
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
|
||||
"$(cat ~/.config/secrets/jellyfin)"
|
||||
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
|
||||
echo 'request = "POST"'; } >/tmp/jf.hdr
|
||||
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
|
||||
shred -u /tmp/jf.hdr
|
||||
```
|
||||
It carries every law below. Hand-written, the door is a `curl -K` on a mode-600
|
||||
config: the scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same
|
||||
value is `JELLYFIN_API_KEY` in the ignored `dankserver/.env`) and rides as
|
||||
`Authorization: MediaBrowser Token="…"` — 12.4 refuses a bare `X-Emby-Token` for
|
||||
every key, valid or not — never as a header argument, because
|
||||
`/proc/<pid>/cmdline` is world-readable. `recentMedia` counts
|
||||
`BaseItems.DateModified` rows in `jellyfin.db`, which on this host sits in the
|
||||
quadlet's config bind at `/home/dank/Containers/jellyfin/config/data/`, not the
|
||||
stock `~/.local/share/jellyfin`.
|
||||
|
||||
`204` is acceptance; anything else queued no scan, so re-issue the identical
|
||||
request instead of diagnosing. A `401` from a key that answers `200` on a
|
||||
hand-typed request is this block's quoting, not the server: bash `printf`
|
||||
collapses `\"` to `"`, curl ends the quoted value at that quote, and Jellyfin
|
||||
receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answers
|
||||
204, `\"` answers 401 — both run on this host).
|
||||
204, `\"` answers 401 — both run on this host). Before believing a 401, prove the
|
||||
key against the server's own table — `select count(*) from ApiKeys where
|
||||
AccessToken = ?` on `jellyfin.db` opened plain `mode=ro` (`immutable=1` only as
|
||||
the fallback for a database whose `-shm` cannot be read; it skips the write-ahead
|
||||
log, see [[Media Relayout]]). Reading progress with the same config trick takes
|
||||
**one** URL: a config `url` line *plus* a command-line URL makes curl perform both
|
||||
transfers, and the second dumps the whole `ScheduledTasks` body onto your stdout.
|
||||
|
||||
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
||||
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
||||
later `204` is accepted and then discarded — the status code stops meaning
|
||||
anything. The fix is `systemctl --user restart jellyfin.service` (never
|
||||
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
|
||||
anything. The fix is `systemctl --user restart jellyfin.service`, one POST after
|
||||
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
||||
response code.
|
||||
|
||||
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
|
||||
is the incremental pass, and inotify does not cross a rootless podman bind, so
|
||||
folders created on the host stay invisible to it. The move is
|
||||
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
|
||||
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
|
||||
record's `RefreshStatus` is the progress read.
|
||||
**The door that scans is `POST /Library/Refresh` (204), and `ScheduledTasks` is
|
||||
the progress read.** A task that has never completed answers
|
||||
`LastTaskResult=None, EndTimeUtc=None`, and that pair is what separates "never
|
||||
ran" from "finished a moment ago" — the difference between a door that is dead
|
||||
and one that merely answered `204`.
|
||||
|
||||
**The settle read is `GET /ScheduledTasks`: the row whose `Name` is exactly
|
||||
`Scan Media Library`, and its `State` (`Idle`|`Running`).** `/Library/ScanStatus`
|
||||
is a 404 on 12.4, so a watcher built on the old endpoint reports nothing and a
|
||||
scanning server looks an idle one. Call it settled only after a `Running`
|
||||
sighting followed by two consecutive `Idle`s: sub-tasks leave gaps that a single
|
||||
poll reads as finished, and a census taken in one reports the pre-scan library.
|
||||
|
||||
**The per-library `POST /Items/{id}/Refresh` queues nothing, and the provider
|
||||
switch is not the reason.** Measured twice on 2026-11-02 against `Shows`: before
|
||||
the flip and again after it, the call answered `204`, the journal wrote zero lines
|
||||
over the following ten minutes, and `Scan Media Library` kept its never-run
|
||||
signature; a service restart changes nothing. `POST /Library/Refresh` does run —
|
||||
`204`, then `Running` inside 15 s at 33 % CPU — but it walks every library, so on
|
||||
this build a library-scoped metadata pass has no working door and the global one
|
||||
is the only lever.
|
||||
|
||||
**That endpoint takes its parameters from one source.** The documented query form
|
||||
(empty body) and the JSON body form (no query) each parse; supplying both at once
|
||||
answers `400` with an empty body whatever the values, and `ValidationFailure` is
|
||||
rejected in the body while the query form accepts it.
|
||||
|
||||
**`RefreshStatus` on the `GET /Library/VirtualFolders` record is not a progress
|
||||
read.** It stayed `Idle` through a scan that was verifiably running at 75 %. The
|
||||
record's useful field is `ItemId`, and the live library set is still the entries
|
||||
whose `Locations` are non-empty.
|
||||
|
||||
`inotify` does not cross a rootless podman bind, so folders created on the host
|
||||
stay invisible until a pass like this runs; every pass that writes into a library
|
||||
ends by firing one.
|
||||
|
||||
**`Movies` has no watcher at all.** `EnableRealtimeMonitor` is per library (table in
|
||||
*Data layout*) and it is false for `Movies` and `Collections`. Nothing notices a
|
||||
folder created under `/data/Movies`: it stays invisible until a recursive refresh
|
||||
names it. So every pass that writes into a library ends by firing one, and "Jellyfin
|
||||
does not see the file I just moved in" is the designed answer, not a fault to chase.
|
||||
|
||||
## API-key ledger and rotation
|
||||
|
||||
`/Auth/Keys` is the ledger. `GET` lists every key **with its `AccessToken` in
|
||||
cleartext**, so an admin token in hand prints the whole fleet's keys;
|
||||
`POST /Auth/Keys?app=<name>` mints one and answers `204` **with no body** — the
|
||||
new token is read back from the `GET`, never from the create response.
|
||||
`DELETE /Auth/Keys/{token}` revokes **by the token string**, and a parameter that
|
||||
matches nothing answers `204` too, so `204` there is not proof anything was
|
||||
removed. Discover a route's verb from the `Allow` header of an `OPTIONS` (or
|
||||
`GET`) response — never by firing `DELETE` at a guess.
|
||||
|
||||
- `IsActive` reads `False` on rows that authenticate `200`. It is not a usability
|
||||
signal; only an authenticated request is.
|
||||
- `DateLastActivity` never moves off `0001-01-01` on 12.4, so "is this key in
|
||||
use?" cannot be answered from the ledger. The census is repo + host credential
|
||||
stores + the `Devices` table (whose `DateLastActivity` *is* real).
|
||||
- Rotation order: mint → rewrite **both** stores (`~/.config/secrets/jellyfin`
|
||||
and `JELLYFIN_API_KEY` in the ignored `dankserver/.env`) → prove the new key on
|
||||
`/System/Info` → `DELETE` the old token → prove `401`. Revoking while a store
|
||||
still holds that value breaks the scan door silently.
|
||||
- Blast radius as measured 2026-11-03: no repo script reads `JELLYFIN_API_KEY`;
|
||||
Seerr keeps a key of its own in its own DB (which of its two rows it uses was
|
||||
never confirmed), so revoke by token and never by app name.
|
||||
|
||||
## Big-screen theme (the TV)
|
||||
|
||||
`dankserver/jellyfin/theme/wmc/theme.css` is repo-owned and bind-mounted over the
|
||||
image's **Media Center** theme slot (the `Volume=` line in
|
||||
`dankserver/jellyfin/quadlet/jellyfin.container`). It exists for
|
||||
`danklivingroom`, which renders this server-served web app at 3840×2160 with no
|
||||
client-side scaling, so 14 px text arrives on a panel read from three metres.
|
||||
|
||||
The file is four rules, and its first line is the design:
|
||||
`@import url("../dark/theme.css")` pulls in the **live upstream** dark palette, so
|
||||
the repo never vendors a CSS snapshot and a Jellyfin image bump cannot strand one.
|
||||
That works because jellyfin-web applies a theme as
|
||||
`<link rel="stylesheet" href="themes/<id>/theme.css">` — a relative URL resolved
|
||||
against the CSS file, not the document. The size lives in one number:
|
||||
`--dank-big-screen-zoom` (1.75 ≈ three metres; 1.5 mild, 2.0 projector). `zoom` on
|
||||
the root element is the right lever: the card-grid code measures layout pixels, so
|
||||
fewer, larger columns are computed rather than overflow produced.
|
||||
|
||||
- **Adoption is per device, and it stays that way.** The theme a client renders
|
||||
comes from its own settings (localStorage first), so there is no server-side
|
||||
"everyone gets the big UI" knob. On the screen that wants it:
|
||||
**Display Settings → Theme → Media Center.** That is also the honest label —
|
||||
the dropdown names are compiled into the bundle, so a "Big Screen" entry will
|
||||
never appear; do not go looking for one.
|
||||
- The mount is `:ro` and the app only fetches the file, so the failure mode of a
|
||||
stale `@import` target is slightly wrong colors, never a dead web app.
|
||||
- **Editing the CSS needs no restart.** The bind hands out the host file's live
|
||||
bytes, so a changed zoom number is on screen at the next page load. The planner
|
||||
agrees by construction: only `dankserver/<svc>/quadlet/**` maps to a service, so
|
||||
a `theme/**`-only commit plans nothing — that is correct, not a gap to "fix",
|
||||
because restarting a media server for a font size is the wrong trade. Only the
|
||||
first deploy of the `Volume=` line restarts anything.
|
||||
- **Rollback is one line:** drop the `Volume=` and redeploy; nothing in the
|
||||
database records the theme, so no cleanup follows.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -11,7 +11,8 @@ timestamp: 2026-10-31T00:00:00Z
|
||||
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
||||
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
||||
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
||||
is the NARRATOR, both matched out of the files' own tags. Today one book is split
|
||||
is the NARRATOR, both matched out of the files' own tags. [[Audiobookshelf]]
|
||||
streams the same tree, read-only, and owns nothing in it. Today one book is split
|
||||
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
||||
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
||||
re-files the tree so the property both servers' book scanners assume is true —
|
||||
@@ -36,8 +37,11 @@ python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
|
||||
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
||||
# as in step 5 of the movie Procedure — from inside the container, never over
|
||||
# the host port.
|
||||
# Audiobookshelf stores absolute paths, so the same refresh has to reach its
|
||||
# library too ([[Audiobookshelf]]) — otherwise it keeps rows for folders that
|
||||
# no longer exist.
|
||||
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
||||
--journal /mnt/entropy/.media-campaign/media-relayout-audiobooks.jsonl --apply
|
||||
```
|
||||
|
||||
The group is the identity Plex asserts in its own database — **(narrator, book,
|
||||
@@ -64,7 +68,7 @@ no chapter row names, and a loose file in a folder holding two books all park un
|
||||
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
|
||||
`Various Artists` is not a placeholder — that is a multi-cast read.
|
||||
|
||||
The reviewed plan lives at `/mnt/entropy/media-relayout-audiobooks-plan.tsv`; it
|
||||
The reviewed plan lives at `/mnt/entropy/.media-campaign/media-relayout-audiobooks-plan.tsv`; it
|
||||
applies on its own journal, behind the music writer, and then refreshes the Plex
|
||||
section. Verify it with `--db-kind plex` — Jellyfin holds no rows here, so a
|
||||
Jellyfin pass proves nothing either way.
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Duplicate Prune
|
||||
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
|
||||
tags: [dankserver, media-server, storage, procedure]
|
||||
timestamp: 2026-11-02T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Duplicate Prune
|
||||
|
||||
The movie library holds the same film twice in two different ways — the same file
|
||||
downloaded again, or the same film grabbed a second time at another quality — and
|
||||
[[Entropy Volume]] is permanently full, so both are worth finding. They need
|
||||
different rules, and neither is decided by name similarity alone.
|
||||
|
||||
A series duplicates differently again: one show spread over several folders, so
|
||||
neither the file nor the folder is the unit — see *A series is measured per
|
||||
episode*.
|
||||
|
||||
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
|
||||
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||
|
||||
## What counts
|
||||
|
||||
| Class | Test | Action |
|
||||
|---|---|---|
|
||||
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
|
||||
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
|
||||
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
|
||||
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
|
||||
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |
|
||||
|
||||
## The screen
|
||||
|
||||
The array cannot afford a full-content read, so identity is established in steps
|
||||
and each step only narrows what the next one touches:
|
||||
|
||||
1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
|
||||
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
|
||||
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
|
||||
evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
|
||||
the better part of an hour; the sparse window cost 50 s, a file that merely
|
||||
shares its head and tail cannot survive it, and the destination is a
|
||||
recoverable hold rather than a deletion.
|
||||
4. Durations come from `scripts/media-census.py probe` — the relayout's own
|
||||
census. Do not hand-roll a second probe: it re-seeks the array for an answer
|
||||
the repo tool already produces, and a probe is the one census step that costs
|
||||
hours instead of seconds.
|
||||
|
||||
## Which copy survives
|
||||
|
||||
In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
|
||||
folder (an extra is not a copy of the film, and extra names collide across films);
|
||||
then the shallowest path; then the file whose name equals its `Title (year)`
|
||||
folder; then the largest. For the same-work class the largest is the answer, and
|
||||
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
|
||||
outlive `Film.mkv` for no reason but a space sorting before a dot.
|
||||
|
||||
## A series is measured per episode
|
||||
|
||||
`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
|
||||
executor's own format; it never moves anything itself.
|
||||
|
||||
```bash
|
||||
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||
--hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
|
||||
--meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
|
||||
--ids /mnt/entropy/.media-campaign/tv-tvdb-ids.tsv \
|
||||
--out /mnt/entropy/.media-campaign/tv-dupes-plan.tsv --json /tmp/tv-groups.json
|
||||
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
|
||||
```
|
||||
|
||||
- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
|
||||
name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
|
||||
and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
|
||||
match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
|
||||
`BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
|
||||
stripped to the bare top-level folder name — 616 folders, none carrying two
|
||||
ids, written to `/mnt/entropy/.media-campaign/tv-tvdb-ids.tsv`. Two folders with disjoint ids
|
||||
are two shows however alike they read — *Avatar: The Last Airbender* (1999)
|
||||
beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
|
||||
had also titled *Millennium* — and a pair whose identity cannot be proven is
|
||||
reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
|
||||
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
|
||||
fill an untagged folder and are **dropped** when they contradict its own
|
||||
`[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
|
||||
provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
|
||||
[tvdbid-73838]`; unioned, those folders are one show and the special's bigger
|
||||
files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
|
||||
the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
|
||||
staged file rests on a borrowed id.
|
||||
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
|
||||
2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
|
||||
Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
|
||||
season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
|
||||
the pass would have staged about 340 GB of unique episodes; judged per episode it
|
||||
staged 54 GiB of true copies.
|
||||
- **A file is droppable only when every episode it carries is held elsewhere in the
|
||||
same show at >= its size** (Frank's rule, applied per episode). A byte-identical
|
||||
pair ties and the tie breaks by path so exactly one copy stages; a file holding
|
||||
one unique episode stays whatever else it carries.
|
||||
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
|
||||
poster or a whole season, so key-less names are never staged and are counted in
|
||||
the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
|
||||
Where a packager named files by id, Jellyfin's own table is the witness: a
|
||||
`path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
|
||||
and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
|
||||
rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
|
||||
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
|
||||
`pragma table_info` before trusting the query, because an empty witness silently
|
||||
falls back to name parsing.
|
||||
Seven clips in the library are still named only by number, and where Jellyfin
|
||||
was asked its own `Name` for them is the number too, so no database holds a
|
||||
better title — that residue is naming, not a task.
|
||||
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
|
||||
when both halves are held elsewhere, and it wins the survivor seat for both when
|
||||
it is the largest carrier: the byte law compares files, and a name both servers
|
||||
parse as a range stays indexed as two episodes.
|
||||
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
|
||||
artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
|
||||
survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
|
||||
copies were bare rips, and counting a hold's non-media files is the census that
|
||||
tells you which case you are in.
|
||||
- **The audit gates the plan file.** Every survivor is re-statted and every
|
||||
destination re-checked before `--out` is written; on failure the report prints
|
||||
`AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
|
||||
- **Two folders that look duplicated and share zero episode keys are complementary,
|
||||
not duplicated.** That is how the two pairs the census still refuses to merge were
|
||||
settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
|
||||
beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
|
||||
Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
|
||||
episode keys: 0` in both cases, so a tag would have freed nothing and cost a
|
||||
rename with library consequences. Re-running the whole census with the id map over
|
||||
all 753 top-level folders (606 already tagged) staged **0 files**: the library is
|
||||
duplicate-free to the limit of provable identity, and what remains is naming.
|
||||
|
||||
## Guards
|
||||
|
||||
Each of these is a file that would have been destroyed without it:
|
||||
|
||||
- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
|
||||
the finished file while qB still counts the download incomplete; moving it
|
||||
breaks the re-check and re-downloads the film. Report it, leave it. Both
|
||||
leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
|
||||
not a prune decision.
|
||||
- **Require a year, and keep digits in the title.** Stripping digits merged
|
||||
*Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
|
||||
happened to share a name merged.
|
||||
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
|
||||
is excluded: half a multi-disc release is not a copy of the other half.
|
||||
- Media containers only. Grouping every extension buries the report in subtitle
|
||||
and cover pairs — and a duration guard that appears to fire on 720 groups is
|
||||
counting subtitles, not protecting films.
|
||||
- **A short file named after a film is promotion, not a movie.** Thirty-two files
|
||||
in `Movies` — 31 clips and the `.nfo` filed beside one of them — carried real
|
||||
titles (`The Descent (2005) - SD.mkv`) and were 8–20-second h264 video at
|
||||
640×360, 1024×576 or 1280×720, 49 MiB all together —
|
||||
25 of them in a `_release-group watermarks/` folder and the rest at the library
|
||||
root or inside one film's folder. No duplicate screen ever catches them: nothing
|
||||
shares their size, and the feature-length guard above makes them invisible to the
|
||||
similar-title pass too. They were held as `.AGENTTRASH/20261104-promo-clips`
|
||||
(journal `/mnt/entropy/.media-campaign/promo-hold.jsonl`, which survives) and purged
|
||||
the same evening, once both databases read `REAL: 0` against that journal.
|
||||
The census that finds the next batch is a duration floor, not a hash: a video under
|
||||
a minute on a name that reads like a feature is advertising.
|
||||
|
||||
## The hold
|
||||
|
||||
Nothing is deleted. Every doomed copy moves into a dated hold with the same
|
||||
executor the relayout uses, so any file comes back:
|
||||
|
||||
```bash
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/dup-prune-similar-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl # dry
|
||||
scripts/media-apply.py apply --plan … --root /mnt/entropy --journal … --apply
|
||||
scripts/media-apply.py verify --plan … --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||
scripts/media-apply.py rollback --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||
```
|
||||
|
||||
The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
|
||||
every hold parent), and the planner re-checks the two things that go stale between
|
||||
analysis and apply: a file that has moved since the census, and a group whose
|
||||
keeper is missing — either one drops the whole group rather than half-applying it.
|
||||
|
||||
- **A hold on the same volume frees nothing.** The purge is a separate, approved
|
||||
act and the only thing that moves the free-space number; `df` does not budge
|
||||
when 1 TiB moves sideways.
|
||||
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
|
||||
ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
|
||||
era, not this pass — read the mtimes before blaming the prune.
|
||||
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
|
||||
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`.
|
||||
- The 2026-11-02 TV pass held 220 files (54 GiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
|
||||
journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
|
||||
same day's TVShows detritus pass staged 109 files (552 KiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
|
||||
`tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. All four
|
||||
journals and plans live at the `entropy` root, not `/tmp`.
|
||||
- **Purged 2026-11-04, on Frank's word, after the gate below:** the duplicate
|
||||
prune, the TV duplicate pass, the Ghibli duplicates (6 files — both `Nausicaä`
|
||||
copies among them) and the 2026-11-04 promo-clip hold left the array trash,
|
||||
~1.2 TB freed (the array went 218 GB → 1.4 TB available). The journals survive as
|
||||
the only record of where a purged file came from; rollback after a purge is
|
||||
impossible, which is why the gate runs first.
|
||||
|
||||
### The purge gate: a carrier for every held file
|
||||
|
||||
`REAL: 0` in both databases says only that no row points at a moved path; it says
|
||||
nothing about whether the film survived. So a purge waits on a stronger proof:
|
||||
**every held file has a same-work carrier in the live library at >= its size,
|
||||
re-derived at purge time**, never copied from the plan. The 2026-11-04 purge re-found
|
||||
all 452 held files this way; the first pass of the gate could not place 142 of them,
|
||||
and every one of those was a matcher gap, not a lost film. What the re-derivation has
|
||||
to absorb:
|
||||
|
||||
- **The layout moves under the plan.** The relayout renames folders to
|
||||
`Title (YYYY)` and `[tvdbid-N]/Season N` and renames the keeper files too, so the
|
||||
paths *and* the sizes a plan recorded no longer describe the library. Keys are work
|
||||
identity — normalized title+year for a film, `(S,E)` for a TV pack — never a path.
|
||||
- **Fold accents, then compare.** The library holds `Nausicaä`; a plan, a
|
||||
`find -ipath "*Nausicaa*"`, and an ASCII substring search all miss it. NFKD-fold and
|
||||
drop combining marks, or a real keeper reads absent for two rounds and the file
|
||||
beside it starts to look like an orphan.
|
||||
- **The carrier is often another container, and usually larger.** A held `.mp4` can be
|
||||
kept as the sibling `.mkv`; keep-larger means the keeper is frequently far bigger (a
|
||||
377 MB held episode matched a 501 MB keeper; a 1.6 GiB film stands beside a 34 GiB
|
||||
BDRemux of itself in another folder). "Same folder, same bytes" reports survivors as
|
||||
lost — it is the wrong test.
|
||||
- **An exact twin can sit in a sibling folder**, so "no same-size file in the folder"
|
||||
needs an array-wide name search before it means anything.
|
||||
|
||||
### Running the purge
|
||||
|
||||
`scripts/purge-agenttrash.sh` is the only door — never `rm`. It acts on exactly one
|
||||
object: a root's own `.AGENTTRASH`. A partial purge is therefore **staged, not
|
||||
filtered**: move the approved buckets into a scratch root
|
||||
(`.PURGE-<date>/.AGENTTRASH/`) and point the purger at that root, and park anything
|
||||
not approved at the array root as `.HOLD-<date>-*`, out of its reach. Dry-run first —
|
||||
the plan must print exactly one `plan empty` row, and it is the same row `--execute`
|
||||
acts on — and send `--log` into `.media-campaign/`, because `/tmp` is not a record.
|
||||
Then read `df`: a purge is the only media act that moves the free-space number.
|
||||
|
||||
## After the pass
|
||||
|
||||
Both servers still hold rows for files that are now in the hold, and neither
|
||||
notices: the media bind is the [[Media Metadata View]], and array changes do not
|
||||
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
|
||||
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
|
||||
are owned by [[Jellyfin]] and [[Plex]].
|
||||
|
||||
The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user
|
||||
stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a
|
||||
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
|
||||
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
|
||||
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
|
||||
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
|
||||
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
|
||||
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.
|
||||
|
||||
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
|
||||
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
|
||||
from folders whose whole subtree held no media, so it creates no stale rows and
|
||||
needs no scan. The guards that hold a folder back are the finding: external
|
||||
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
|
||||
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
|
||||
extension the census does not recognise is a reason to stop, not to sweep.
|
||||
|
||||
A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
|
||||
file is a manual match — a human decision about what the folder is — and detritus
|
||||
rules do not overrule one.
|
||||
|
||||
`scripts/media-verify.py` reads differently for a prune than for a relayout:
|
||||
`landed: 0` is the **correct** end state (no server may index the hold), so the
|
||||
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
|
||||
from the database are the pass succeeding.
|
||||
|
||||
One residue is normal, so do not chase it: a section pass rewrites the feature rows
|
||||
first, and rows for clips in a film's `other/` or `extras/` subfolder can still name
|
||||
a held path after the section's `refreshing` flag is back to `0` — and a second
|
||||
section pass does not clear them either, so do not run one hoping for `0`. Beside
|
||||
`REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch
|
||||
the cache, the database, or the hold.
|
||||
|
||||
## Failure signatures
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
|
||||
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
|
||||
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
|
||||
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |
|
||||
|
||||
## Links
|
||||
|
||||
- [[Media Relayout]] — the census tools, the executor, the journal format.
|
||||
- [[Media Metadata View]] — why neither server sees the change without a scan.
|
||||
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
|
||||
- [[Entropy Volume]] — the array that is always full.
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Media Metadata View
|
||||
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
timestamp: 2026-11-02T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Metadata View
|
||||
@@ -20,18 +20,40 @@ The mount point is not a volume and holds nothing while unmounted. It sits insid
|
||||
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
||||
next to the array.
|
||||
|
||||
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
|
||||
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
|
||||
`/var/lib/media-meta` does not exist, and the installed quadlets under
|
||||
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
|
||||
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
|
||||
is the shape git describes, not the shape the servers run. The cutover is an
|
||||
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
|
||||
`ensure_media_view` (`install` then `verify`) before it restarts the container,
|
||||
so it must not be fired through the middle of a library scan. Until that deploy
|
||||
lands there is no upper layer, so no relayout can leave ghost metadata and
|
||||
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
|
||||
view really is a mountpoint.
|
||||
**Deployed on Dankserver — 2026-10-31, both servers.** `plex` binds
|
||||
`/var/lib/media-meta/view:/mnt/entropy:ro` and `jellyfin` binds
|
||||
`/var/lib/media-meta/view:/data:rw`; `findmnt` reports the overlay with
|
||||
`lowerdir=/mnt/entropy upperdir=/var/lib/media-meta/upper`, and each container
|
||||
was proven to resolve a real media file through it — `podman exec <svc> ls -la
|
||||
<its data path>` is the whole proof, because both servers keep the paths their
|
||||
databases recorded. Two service-scoped deploys, each gated by
|
||||
`media-view.sh verify` and the Playwright suite.
|
||||
|
||||
Re-verified 2026-11-01: `mount` shows the overlay on
|
||||
`/var/lib/media-meta/view` (`lowerdir=/mnt/entropy`,
|
||||
`upperdir=/var/lib/media-meta/upper`), and the installed quadlets under
|
||||
`~/.config/containers/systemd/` carry the view binds:
|
||||
`Volume=/var/lib/media-meta/view:/data` for [[Jellyfin]],
|
||||
`Volume=/var/lib/media-meta/view:/mnt/entropy:ro` for [[Plex]]. Prove it in that
|
||||
order: `scripts/media-view.sh verify` (the deploy gate), then
|
||||
`grep -n Volume ~/.config/containers/systemd/{jellyfin,plex}.container`.
|
||||
|
||||
Moving a bind means restarting the server, and **a restart discards that
|
||||
server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
|
||||
`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
|
||||
it. Renames made on the array while the view was unmounted are invisible to both
|
||||
watchers too, so the additive index has to be asked for by hand; the door is an
|
||||
API key, and the row in the failure table below says what to check when it
|
||||
refuses.
|
||||
|
||||
The cutover itself is an ordinary deploy: `deploy.sh dankserver --service
|
||||
jellyfin|plex` runs `ensure_media_view` (`install` then `verify`) before it
|
||||
restarts the container, so it must not be fired through the middle of a library
|
||||
scan. Two consequences are now live rather than theoretical: inotify events for
|
||||
array writes do not cross the merge, so any relayout or prune needs a manual
|
||||
library refresh; and a move can leave ghost sidecars above, so `reap` runs after
|
||||
a relayout — a pass that moves only media files with no sidecars next to them
|
||||
leaves the upper layer untouched, and `reap` correctly finds nothing.
|
||||
|
||||
## Shape
|
||||
|
||||
@@ -112,6 +134,19 @@ the defaults are the host values above.
|
||||
upper layer*, not for changes on the array underneath. After any relayout or
|
||||
externally-written metadata, trigger a library refresh — see
|
||||
[[Media Relayout]] for the refresh procedure.
|
||||
- **`stat` through the view can outlive the file.** Measured 2026-11-02 while a TV
|
||||
dedupe pass moved folders on the raw path: the merged tree still answered `stat`
|
||||
at the old coordinates — link count `0`, and `ls` of the same parent no longer
|
||||
listed the name. Overlayfs keeps a dentry for a lower file that was moved
|
||||
underneath it, so a script that tests existence with `[ -e ]` against the view can
|
||||
believe a file its own `readdir` denies. Prove existence by listing the parent.
|
||||
The media servers walk by `readdir`, so ghosts never reach the library — this bites
|
||||
a tool, never a user, which is why no server symptom accompanies it.
|
||||
- **Tools that move or measure media walk the raw array, never the view** — every
|
||||
relayout, prune and detritus pass, and every journal it writes. The view is for
|
||||
Jellyfin's writes and Plex's reads only. One deliberate exception: `tvshow.nfo`
|
||||
sidecars live in the upper layer, so a tool reading show identity points at the
|
||||
view (`media-tvdupes.py --meta /var/lib/media-meta/view/TVShows`).
|
||||
|
||||
## Failure signatures
|
||||
|
||||
@@ -122,7 +157,11 @@ the defaults are the host values above.
|
||||
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
||||
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
||||
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
||||
| `install` prints `refused: upper base /var/lib/media-meta does not exist (put it on the SSD first)` | A fresh host. `install` deliberately never creates the base — a mistyped `--upper-base` must not build a metadata tree on the wrong filesystem — so `scripts/media-view.sh ensure-base` makes the documented one, which the deployer calls first. |
|
||||
| A media server's scan record reads `"Status": "Aborted"` | Its container restarted (a cutover, an image bump, an OOM) and the walk was discarded; systemd's `Restart=always` restarts the *server*, never the scan. Trigger a refresh — see [[Media Relayout]]. |
|
||||
| The Jellyfin API answers 401 to every token header | Almost always the header **form**, not the key: 12.4 authenticates only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for every key, valid or not — and one collapsed backslash in the curl config produces the same 401. `[[Jellyfin]]` owns the block that works. Before concluding anything, prove the key against the server's own table: `select count(*) from ApiKeys where AccessToken = ?` on `jellyfin.db` opened `mode=ro&immutable=1`. |
|
||||
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
||||
| Stale dentry through the view: `stat` succeeds with link count `0` while the parent's `ls` omits the name | The lower file was moved on the raw path after the view cached its dentry. Nothing is lost; existence through the view is proved by the parent listing. |
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.
|
||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
|
||||
# executor, journal and rollback as [[Media Relayout]] shows, on its own journal.
|
||||
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-music.jsonl --apply
|
||||
--journal /mnt/entropy/.media-campaign/media-relayout-music.jsonl --apply
|
||||
```
|
||||
|
||||
The group is what the tags assert, not where the files sit: **(album-artist tag,
|
||||
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Media Relayout
|
||||
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||
timestamp: 2026-10-30T00:00:00Z
|
||||
timestamp: 2026-11-02T11:00:00Z
|
||||
---
|
||||
|
||||
# Media Relayout
|
||||
@@ -71,12 +71,23 @@ that would be filed wrong, and the plan is not run until it is understood.
|
||||
```bash
|
||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
|
||||
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/.media-campaign/media-relayout.jsonl
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
scripts/media-apply.py rollback --journal /mnt/entropy/.media-campaign/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||
```
|
||||
|
||||
**Every campaign artifact — plan, journal, exceptions file, run log — lives in
|
||||
`/mnt/entropy/.media-campaign/`.** The array root carries libraries and nothing
|
||||
else: a root strewn with `*.jsonl` and `*-plan.tsv` gets browsed as content, and
|
||||
the dot-directory is ignored by both media servers. Relocated 2026-11-02, 48 files,
|
||||
undo with `media-apply.py rollback --journal /mnt/entropy/.media-campaign/media-campaign-relocate.jsonl --root /mnt/entropy --apply`.
|
||||
The holds and trash (`.AGENTTRASH`, `.HOLD-<date>-*`) stay at the root on purpose:
|
||||
the purge machinery is given those paths as `--hold`, and a hold that moves out
|
||||
from under the tool that made it is a hold nobody can find. Name `--journal`
|
||||
explicitly on every run — the tool's default is `media-relayout.jsonl` beside
|
||||
`--root`, so a bare run recreates exactly the root-level log this page forbids.
|
||||
|
||||
An apply also leaves the release folders it emptied standing where they were —
|
||||
hundreds of them at the library root — which is what `sweep` removes, dry-run
|
||||
first, from the same journal.
|
||||
@@ -88,12 +99,17 @@ interrupted apply still replays the batch and resumes with nothing lost. Batch 1
|
||||
is the old behaviour; the only reason to name it is debugging, because on the
|
||||
spinning array it spends most of a large apply's wall time in fsyncs.
|
||||
|
||||
After it lands, both servers need a scan or the change is invisible. [[Plex]] and
|
||||
[[Jellyfin]] own the exact requests; two laws hold at this step: `204` is
|
||||
acceptance and anything else queued nothing — re-issue the identical request
|
||||
rather than diagnosing — and ONE accepted refresh is the whole procedure, because
|
||||
a second while a scan runs moves `Scan Media Library` to `Cancelling` and every
|
||||
later `204` is accepted and then discarded.
|
||||
After it lands, both servers need a scan or the change is invisible. The two
|
||||
drivers are the doors — `scripts/media-refresh-jellyfin.py trigger` then `… wait`,
|
||||
and only after it settles, `scripts/media-refresh-plex.sh trigger --library <key>`
|
||||
with `… status --library <key> --after $(stat -c %Y <journal>)` as the poll. Run
|
||||
them in that order and never together: Jellyfin's only door walks every library,
|
||||
every Plex section forks its own scanner walk, and the array is the bottleneck
|
||||
both servers share. [[Plex]] and [[Jellyfin]] own the laws behind the drivers; two
|
||||
of them decide this step: `204` is acceptance and anything else queued nothing —
|
||||
re-issue the identical request rather than diagnosing — and ONE accepted refresh is
|
||||
the whole procedure, because a second while a scan runs moves `Scan Media Library`
|
||||
to `Cancelling` and every later `204` is accepted and then discarded.
|
||||
|
||||
### Proving the move landed
|
||||
|
||||
@@ -110,7 +126,7 @@ a read beside a running scan can fail as "database disk image is malformed".
|
||||
`immutable` stays only the fallback for a database whose `-shm` cannot be read.
|
||||
|
||||
```bash
|
||||
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
|
||||
scripts/media-verify.py --journal /mnt/entropy/.media-campaign/media-relayout-music.jsonl \
|
||||
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
|
||||
```
|
||||
|
||||
@@ -138,6 +154,8 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
||||
no `Accept` and client-identifier header (see the scan block above for the form
|
||||
that works).
|
||||
|
||||
A duplicate prune is the same census and the same proof, with its own keep rules — films by title and duration, a series per episode: [[Media Duplicate Prune]].
|
||||
|
||||
## TV in the movie library
|
||||
|
||||
A box set that arrives in the movie share stays there until someone moves it:
|
||||
@@ -148,11 +166,11 @@ by series, and writes a plan in the executor's own format:
|
||||
|
||||
```bash
|
||||
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
|
||||
--out /mnt/entropy/media-tv-plan.tsv
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
|
||||
--out /mnt/entropy/.media-campaign/media-tv-plan.tsv
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-tv-plan.tsv --root /mnt/entropy
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-tv-plan.tsv --root /mnt/entropy \
|
||||
--journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl --apply
|
||||
scripts/media-apply.py sweep --journal /mnt/entropy/.media-campaign/media-relayout-tv.jsonl \
|
||||
--root /mnt/entropy --apply
|
||||
```
|
||||
|
||||
@@ -184,162 +202,10 @@ show. A name whose marker is `Part 2` is a film and stays put.
|
||||
|
||||
## The TV library's own relayout
|
||||
|
||||
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
||||
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
||||
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
||||
|
||||
<Series> (<year>) [tvdbid-NNNN]/Season NN/
|
||||
<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
||||
|
||||
It never guesses: every series name, episode number and title it writes traces to
|
||||
an authority, and what has none is parked for a human to read.
|
||||
|
||||
```bash
|
||||
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
||||
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
||||
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_titles.tsv
|
||||
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_enum.tsv
|
||||
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
||||
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
||||
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
||||
# the last one — a duration belongs to the file, not to its name: re-key the
|
||||
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
||||
# seeking the array again. It is also the only column the film arm reads, and
|
||||
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
||||
# all of them covered by a re-key that costs seconds, not an hour).
|
||||
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
||||
# which travels as the `api_key` query parameter. Only a v4 request token is
|
||||
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
||||
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
||||
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
||||
# cache file, so the plan is reproducible offline afterwards: run it once to
|
||||
# fill the cache, and quote the second run.
|
||||
# Paths are only strings to a dry run, so this step can run on the workstation
|
||||
# against scp'd TSVs — which is where the two provider secrets live.
|
||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
||||
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
||||
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
|
||||
--park /mnt/entropy/media-park/TVShows \
|
||||
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
|
||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
||||
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
||||
# applied pass is its unwind record, so a second pass that reuses the name
|
||||
# destroys the only map back.
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
|
||||
```
|
||||
|
||||
Evidence, strongest first — the order the planner believes things:
|
||||
|
||||
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`); a digit
|
||||
run longer than three at that position is NOT a number — `S03E2019` and
|
||||
`S04E47370355` are one box set's airdate fragment and pack index — so such a file
|
||||
is renumbered by tvdb's title or parks, and is never truncated down to fit;
|
||||
- the **`Season NN` folder it sits in and the number its name leads with inside
|
||||
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
||||
- the **provider row**: which episode the file IS, and the numbering of a file
|
||||
whose own name says nothing;
|
||||
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
||||
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
||||
episode title, because a name the server read off a folder is not a show name.
|
||||
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
|
||||
Plex's scanner both read, so a show whose title is ambiguous resolves by number
|
||||
instead of by a human guessing which folder is which. The id rides the folder
|
||||
only — a file name gets 180 characters and the episode title needs the room.
|
||||
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
||||
series' canonical extra folder;
|
||||
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
||||
when the movie provider's title is a run of words the FILE itself states and its
|
||||
year agrees with the file's: the query is only what `title_of` salvaged from a
|
||||
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
||||
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
||||
parks.
|
||||
|
||||
Four refusals carry the design: what the planner will NOT do is the point of it.
|
||||
|
||||
- **Folders resolve a series; they never invent one.** A folder's words may
|
||||
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
||||
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
||||
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
||||
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
||||
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
||||
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
||||
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
||||
the file's own leading number, win.
|
||||
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
|
||||
specials by airdate, so a row can arrive saying `season 2002`; that is not a
|
||||
season and it never becomes a folder. The one authority on such a row is tvdb's
|
||||
episode list read backwards: the title the row carries names the (season,
|
||||
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
|
||||
the file parks with the reason it always carried. `Season 00` IS a legal
|
||||
season — specials live there — so a row that really says 0 files there instead
|
||||
of parking.
|
||||
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
|
||||
numbered by its release group books `S05E01` in a season tvdb does not hold, or
|
||||
books more episodes there than tvdb lists: the slot its number points at is a
|
||||
different episode, and dressing it in that episode's title is a confident lie.
|
||||
The file still renames to the standard shape; the title does not ride, and the
|
||||
series is listed under *seasons whose numbering tvdb contradicts* in the
|
||||
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
|
||||
partial pack is the normal shape. Neither reading holds for a list tvdb
|
||||
cut off at its own page limit: the default episode query answers at most
|
||||
500 rows and states no total, and One Piece, Crayon Shin-chan and
|
||||
Detective Conan each answer exactly 500 while holding over a thousand —
|
||||
a page bounds nothing above itself, so it contradicts nothing.
|
||||
|
||||
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
||||
after the folder it scanned, so a library filed by release folders ends up with
|
||||
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
||||
copying those names verbatim launders folder junk into the contract. The planner
|
||||
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
||||
identity is also what turns a box set split across five release folders into ONE
|
||||
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
||||
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
||||
identity* in the exceptions — that list, not a rename, is the review.
|
||||
|
||||
**The review pile lives outside the library.** Files parked inside
|
||||
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
||||
show's rows then looked like evidence. So a row whose series is the hidden
|
||||
top-level folder the file itself sits in is dropped — and the folder is matched
|
||||
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
||||
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
||||
visible series folder that real files get renamed into. `--park` outside the root
|
||||
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
||||
top-level folder a file's own words may MATCH a registered show, never invent one.
|
||||
|
||||
**The operator's id map is the door for a name tvdb cannot be searched by.**
|
||||
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
|
||||
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
|
||||
tvdb titles a record in the show's own language and the English name rides in
|
||||
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
|
||||
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
|
||||
2023 live action, not to the 1999 record (81797) the pack here is. The map
|
||||
outranks a provider id — a box set's provider id is the server's own guess at a
|
||||
folder name — and it merges the release folders that are one show. Everything it
|
||||
touched is written to the exceptions: every folder it named, every row that
|
||||
matched no folder, every disagreement with a provider id, every id tvdb never
|
||||
answered. A line whose id is not a number stops the plan, because a typo renames
|
||||
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
|
||||
it; a row for a folder the search would resolve hides a live lookup behind a
|
||||
number nobody re-checks.
|
||||
|
||||
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
|
||||
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
|
||||
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
|
||||
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
|
||||
release junk is stripped from the folder's words), `NEAR` (tvdb answered
|
||||
something else — the best hit is printed for a human and nothing applies it),
|
||||
`NONE`. Acceptance never moved across the passes that built this: a hit counts
|
||||
only when tvdb's name equals the query with every non-alphanumeric dropped from
|
||||
both sides, and the record's year still has to fall inside any year span the
|
||||
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
|
||||
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
|
||||
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
|
||||
and the sheet is never fed to the planner.
|
||||
`/mnt/entropy/TVShows` is its own arm: a different authority (tvdb), its own
|
||||
refusals, and its own test for whether a folder really holds two shows —
|
||||
[[Media TV Relayout]]. It shares this page's executor, the one-journal-per-arm
|
||||
law, the sweep and the rollback.
|
||||
|
||||
## The music and audiobook arms
|
||||
|
||||
@@ -401,9 +267,12 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
||||
- **The journal lives BESIDE the library root, never inside it** (default:
|
||||
`media-relayout.jsonl` next to the root): a log inside the library is a file
|
||||
Jellyfin indexes.
|
||||
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
|
||||
duplicate destination stops the run *before the first rename*, so the operator
|
||||
fixes one thing and re-runs instead of untangling a half-applied plan.
|
||||
- **Refusals are all-or-nothing.** An occupied target, a vanished source, a
|
||||
duplicate destination, or a plan row whose verb is not `mkdir`, `keep` or
|
||||
`move`, stops the run *before the first rename*, so the operator fixes one
|
||||
thing and re-runs instead of untangling a half-applied plan. An unread verb is
|
||||
refused by line number: silently dropping one turned a pass meant to rename
|
||||
two series folders into `applied: 0 moved` with exit 0 (2026-11-02).
|
||||
`--force` relaxes only the opinion guards; it never permits overwriting.
|
||||
- **A sweep deletes only what the journal proves the run emptied.** A folder is
|
||||
eligible when a journalled file moved out of it or out of something under it,
|
||||
@@ -430,6 +299,14 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
||||
`--root` set to the share that holds BOTH libraries: the executor refuses any
|
||||
path outside `--root`, and a journal shared with the movie relayout would
|
||||
undo both runs when only one of them is wanted.
|
||||
- **A hold folder outside `--root` forces `--force` on BOTH `apply` and
|
||||
`sweep`.** `sweep` refuses the entire journal when any journalled path sits
|
||||
outside the root (`this journal belongs to another library`) — the same guard
|
||||
the plan hits, and it exists to stop one pass unwinding another's files.
|
||||
`--force` relaxes it and stays safe there: `sweep_targets` marks only
|
||||
ancestors under `--root`, so a hold folder beyond the root can never be
|
||||
selected; it only blocks everything that is. The cleaner shape is the law
|
||||
above — put `--root` on the share that holds both.
|
||||
|
||||
- **Sidecar metadata does not follow a rename, because it lives on another
|
||||
filesystem.** The media servers write `.nfo` and artwork through the
|
||||
@@ -438,6 +315,9 @@ executor, the one-journal-per-arm law, the sweep and the rollback.
|
||||
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
|
||||
that both servers browse and report as content. Whether this host mounts the
|
||||
view at all is stated on that page; `media-apply.py` stays silent when it does not.
|
||||
A rename of folders the server never identified strands nothing: Jellyfin
|
||||
writes a sidecar only for a series it recognised, so `reap` answering
|
||||
`no ghosts` after such a pass is the expected result, not a broken tool.
|
||||
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
|
||||
whiteout in the upper layer, which hides the array's copy without removing it;
|
||||
the file returns when the view is unmounted. Deletions in this pipeline happen
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media TV Relayout
|
||||
description: "Re-file TVShows from scene packs into the shape Jellyfin reads: authority order, refusals, the two-series split test."
|
||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||
timestamp: 2026-11-02T11:00:00Z
|
||||
---
|
||||
|
||||
# Media TV Relayout
|
||||
|
||||
The `/mnt/entropy/TVShows` share arrived as scene packs — release folders whose
|
||||
names are the release, not the show. This arm re-files it into the shape
|
||||
[[Jellyfin]] reads, with tvdb as the name authority. The executor, the journal,
|
||||
the sweep and the rollback belong to [[Media Relayout]]; this page owns the TV
|
||||
authority order, the refusals, and the test that decides a split.
|
||||
|
||||
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
||||
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
||||
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
||||
|
||||
<Series> (<year>) [tvdbid-NNNN]/Season NN/
|
||||
<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
||||
|
||||
It never guesses: every series name, episode number and title it writes traces to
|
||||
an authority, and what has none is parked for a human to read.
|
||||
|
||||
```bash
|
||||
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
||||
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
||||
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_titles.tsv
|
||||
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
||||
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
||||
--out /tmp/tv_enum.tsv
|
||||
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
||||
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
||||
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
||||
# the last one — a duration belongs to the file, not to its name: re-key the
|
||||
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
||||
# seeking the array again. It is also the only column the film arm reads, and
|
||||
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
||||
# all of them covered by a re-key that costs seconds, not an hour).
|
||||
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
||||
# which travels as the `api_key` query parameter. Only a v4 request token is
|
||||
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
||||
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
||||
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
||||
# cache file, so the plan is reproducible offline afterwards: run it once to
|
||||
# fill the cache, and quote the second run.
|
||||
# Paths are only strings to a dry run, so this step can run on the workstation
|
||||
# against scp'd TSVs — which is where the two provider secrets live.
|
||||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
||||
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
||||
--tmdb-cache /mnt/entropy/.media-campaign/media-tv-names.json --tmdb --tvdb \
|
||||
--park /mnt/entropy/NeedsProcessing/TVShows \
|
||||
--out /mnt/entropy/.media-campaign/media-relayout-shows2-plan.tsv \
|
||||
--exceptions /mnt/entropy/.media-campaign/media-relayout-shows2-exceptions.txt
|
||||
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
||||
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
||||
# applied pass is its unwind record, so a second pass that reuses the name
|
||||
# destroys the only map back.
|
||||
scripts/media-apply.py apply --plan /mnt/entropy/.media-campaign/media-relayout-shows2-plan.tsv \
|
||||
--root /mnt/entropy --journal /mnt/entropy/.media-campaign/media-relayout-shows2.jsonl --apply
|
||||
```
|
||||
|
||||
Evidence, strongest first — the order the planner believes things:
|
||||
|
||||
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`); a digit
|
||||
run longer than three at that position is NOT a number — `S03E2019` and
|
||||
`S04E47370355` are one box set's airdate fragment and pack index — so such a file
|
||||
is renumbered by tvdb's title or parks, and is never truncated down to fit;
|
||||
- the **`Season NN` folder it sits in and the number its name leads with inside
|
||||
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
||||
- the **provider row**: which episode the file IS, and the numbering of a file
|
||||
whose own name says nothing;
|
||||
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
||||
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
||||
episode title, because a name the server read off a folder is not a show name.
|
||||
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
|
||||
Plex's scanner both read, so a show whose title is ambiguous resolves by number
|
||||
instead of by a human guessing which folder is which. The id rides the folder
|
||||
only — a file name gets 180 characters and the episode title needs the room.
|
||||
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
||||
series' canonical extra folder;
|
||||
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
||||
when the movie provider's title is a run of words the FILE itself states and its
|
||||
year agrees with the file's: the query is only what `title_of` salvaged from a
|
||||
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
||||
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
||||
parks.
|
||||
|
||||
Four refusals carry the design: what the planner will NOT do is the point of it.
|
||||
|
||||
- **Folders resolve a series; they never invent one.** A folder's words may
|
||||
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
||||
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
||||
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
||||
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
||||
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
||||
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
||||
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
||||
the file's own leading number, win.
|
||||
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
|
||||
specials by airdate, so a row can arrive saying `season 2002`; that is not a
|
||||
season and it never becomes a folder. The one authority on such a row is tvdb's
|
||||
episode list read backwards: the title the row carries names the (season,
|
||||
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
|
||||
the file parks with the reason it always carried. `Season 00` IS a legal
|
||||
season — specials live there — so a row that really says 0 files there instead
|
||||
of parking.
|
||||
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
|
||||
numbered by its release group books `S05E01` in a season tvdb does not hold, or
|
||||
books more episodes there than tvdb lists: the slot its number points at is a
|
||||
different episode, and dressing it in that episode's title is a confident lie.
|
||||
The file still renames to the standard shape; the title does not ride, and the
|
||||
series is listed under *seasons whose numbering tvdb contradicts* in the
|
||||
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
|
||||
partial pack is the normal shape. Neither reading holds for a list tvdb
|
||||
cut off at its own page limit: the default episode query answers at most
|
||||
500 rows and states no total, and One Piece, Crayon Shin-chan and
|
||||
Detective Conan each answer exactly 500 while holding over a thousand —
|
||||
a page bounds nothing above itself, so it contradicts nothing.
|
||||
|
||||
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
||||
after the folder it scanned, so a library filed by release folders ends up with
|
||||
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
||||
copying those names verbatim launders folder junk into the contract. The planner
|
||||
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
||||
identity is also what turns a box set split across five release folders into ONE
|
||||
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
||||
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
||||
identity* in the exceptions — that list, not a rename, is the review.
|
||||
|
||||
**The review pile lives outside the library.** Files parked inside
|
||||
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
||||
show's rows then looked like evidence. So a row whose series is the hidden
|
||||
top-level folder the file itself sits in is dropped — and the folder is matched
|
||||
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
||||
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
||||
visible series folder that real files get renamed into. `--park` outside the root
|
||||
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
||||
top-level folder a file's own words may MATCH a registered show, never invent one.
|
||||
|
||||
**The operator's id map is the door for a name tvdb cannot be searched by.**
|
||||
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
|
||||
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
|
||||
tvdb titles a record in the show's own language and the English name rides in
|
||||
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
|
||||
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
|
||||
2023 live action, not to the 1999 record (81797) the pack here is. The map
|
||||
outranks a provider id — a box set's provider id is the server's own guess at a
|
||||
folder name — and it merges the release folders that are one show. Everything it
|
||||
touched is written to the exceptions: every folder it named, every row that
|
||||
matched no folder, every disagreement with a provider id, every id tvdb never
|
||||
answered. A line whose id is not a number stops the plan, because a typo renames
|
||||
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
|
||||
it; a row for a folder the search would resolve hides a live lookup behind a
|
||||
number nobody re-checks.
|
||||
|
||||
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
|
||||
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
|
||||
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
|
||||
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
|
||||
release junk is stripped from the folder's words), `NEAR` (tvdb answered
|
||||
something else — the best hit is printed for a human and nothing applies it),
|
||||
`NONE`. Acceptance never moved across the passes that built this: a hit counts
|
||||
only when tvdb's name equals the query with every non-alphanumeric dropped from
|
||||
both sides, and the record's year still has to fall inside any year span the
|
||||
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
|
||||
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
|
||||
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
|
||||
and the sheet is never fed to the planner.
|
||||
|
||||
**A parked folder merges into the row that owns the show.** The folder that never
|
||||
got an id usually holds episodes the identified folder for that show lacks, and
|
||||
the server shows the audience neither: the parked copy is invisible and the owner
|
||||
reads as a short series. One `media-apply.py` plan per show, built from the
|
||||
server's own rows:
|
||||
|
||||
- the **owner is the folder whose `Series` row carries the tvdb id**; the parked
|
||||
side is the folder with the same words and no id (or the same id under a second
|
||||
row — see the law below);
|
||||
- the **comparison is episode identity, not filename**: which (season, episode)
|
||||
the provider row gives each file, read straight out of the server. Release
|
||||
folders renumber — one park carried `S01E01-11` of a show tvdb books as seasons
|
||||
2 and 3, and every episode runs the same eleven minutes, so neither the marker
|
||||
nor the duration says what one query into the rows says;
|
||||
- **a season the owner has no file in moves whole**: `mkdir` the `Season NN`,
|
||||
then one row per file with its name unchanged — the folder shape is the
|
||||
contract, the file name is not;
|
||||
- **a subtitle pack is re-keyed, never just moved**: a sidecar attaches only when
|
||||
its stem equals the video's, so each `.srt` has to land on the OWNER's stem with
|
||||
the language suffix kept (`… - The Cat and the Claw (1).en.srt`). Moved under
|
||||
its release-group name it stays a folder of 88 subtitles no player ever offers;
|
||||
- **the same episode on both sides is the operator's call**, never a plan row:
|
||||
which copy survives is a quality judgment — the parked copy is usually the
|
||||
bigger one, and a re-cut is not a worse one — and the parked file often carries
|
||||
no title to align by. Those pairs are listed and left alone.
|
||||
|
||||
The two sides speak different path dialects, since the server records the
|
||||
container coordinate ([[Jellyfin]]) and the journal speaks the array; a join that
|
||||
forgets the translation returns zero rows and looks like an empty library.
|
||||
|
||||
## Laws of this arm
|
||||
|
||||
- **A folder that names two series is not evidence of two series — the files
|
||||
answer.** `ffprobe -show_entries format_tag=title` names the series each file
|
||||
belongs to. Measured 2026-11-02 on `COW and CHICKEN … Complete Animated TV
|
||||
Series`, a bundle whose name pairs *Cow and Chicken* with *I Am Weasel*: 103
|
||||
of 104 files carry a `Cow and Chicken - SxxEyy - <title>` tag and none carries
|
||||
an *I Am Weasel* one. The 7-minute duration class is a half-episode segment,
|
||||
not a second show, and a file holding both segments needs a remux, which this
|
||||
executor will never do. Split only when the tags (or a provider episode list)
|
||||
name both series per file; everything else is a rename.
|
||||
- **When the numbering is right and only the name is wrong, rename the folder
|
||||
and nothing else.** Tags whose `Sxx` matches the `Season NN` folder holding
|
||||
them, per-season counts equal, and no duplicate `SxxEyy` key anywhere prove
|
||||
the layout is already correct. Renumbering files under that evidence buys
|
||||
nothing and can silently mislabel an episode.
|
||||
- **One tvdb id can carry two `Series` rows, so the id-less folder list proves
|
||||
nothing about a merge.** A parked folder whose words parse to a registered show
|
||||
gets identified on its own and keeps its own folder: the count of id-less
|
||||
folders drops, and the library now offers the same show twice. Prove a merge by
|
||||
the episode rows standing under the owner folder, never by that list.
|
||||
- **Move library files only while `Scan Media Library` is Idle.** A rename that
|
||||
lands under a running scan is a row the server read a moment ago pointing at a
|
||||
path that no longer exists. `ScheduledTasks` is the read ([[Jellyfin]]); wait
|
||||
for a `Running` sighting and then two consecutive `Idle` polls, because a gap
|
||||
between sub-tasks reads as finished.
|
||||
|
||||
## See Also
|
||||
|
||||
[[Media Relayout]] — the executor, the journal, the sweep and the rollback this
|
||||
arm shares. [[Jellyfin]] — the reader of the resulting shape. [[Plex]] — the
|
||||
second reader. [[Media Metadata View]] — where a rename's sidecars strand, or
|
||||
do not.
|
||||
|
||||
## Citations
|
||||
|
||||
- Jellyfin Shows documentation — series, season and episode folder/file naming:
|
||||
<https://jellyfin.org/docs/general/server/media/shows>
|
||||
@@ -27,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
||||
| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — live since the 2026-10-31 cutover, so the PMS database's recorded paths keep resolving with zero re-scan |
|
||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||
@@ -40,7 +40,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
||||
The migration moved zero bytes: the container bind-mounts the native data
|
||||
root where it already lived, and the media archive re-mounts at
|
||||
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||
(the array today; the overlay view in git has never been deployed), so every
|
||||
(the array until the 2026-10-31 view cutover, the overlay since), so every
|
||||
absolute library path in the
|
||||
PMS database keeps resolving. The one metadata change: rootless podman
|
||||
cannot map the real host uid 997, so the fleet model applies — container
|
||||
@@ -86,45 +86,77 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
||||
4. **Cold container boot is slow.** PMS pre-binds 32400 and empty-replies
|
||||
while the HTTP layer finishes loading; the first boot of a years-old
|
||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||
600 s before failing; later boots are much faster (warm page cache).
|
||||
600 s before failing; later boots are much faster (warm page cache). A
|
||||
`systemctl --user start` issued after an `unmask` is a job in the manager,
|
||||
not in the shell that issued it: it lands minutes later and survives that
|
||||
client dying, so a wrapper script that looks hung is usually only waiting —
|
||||
read `is-active` and `podman ps` before declaring a fault, and give every
|
||||
step of such a script its own `timeout`.
|
||||
5. **`32400` answers TLS only to remote clients.** PMS requires secure
|
||||
connections, so `curl http://192.168.0.235:32400/identity` from any other
|
||||
machine closes without a single byte (curl exit 52) — that reads as a dead
|
||||
server beside a healthy one, and it is how a post-cutover health check gets
|
||||
misread. Probe with `curl -k https://192.168.0.235:32400/identity`. Two
|
||||
consequences: `plex.franksdank.cloud` works **only** because its `transport
|
||||
http` block carries `tls_insecure_skip_verify` (that option makes Caddy speak
|
||||
TLS to PMS's self-signed cert — delete it and the vhost 502s), and the scan
|
||||
block below rides `http://127.0.0.1` **inside** the container, where insecure
|
||||
connections are allowed.
|
||||
|
||||
## Triggering a scan
|
||||
|
||||
A rename on disk is invisible until the section is rescanned, and a moved file is
|
||||
a `Couldn't find the file to stream` 404 until that scan passes it. Run this from
|
||||
the host shell through `podman exec`, never from a neighbour container: PMS's
|
||||
HTTPD closes without a single byte on any request that arrives from another
|
||||
container's netns (10.89.x.x reads as "empty reply", not as a 401). The token
|
||||
rides stdin, never argv, where any process on the host could read it out of
|
||||
`/proc/<pid>/cmdline`; it lives in `Preferences.xml` inside the RW data bind.
|
||||
a `Couldn't find the file to stream` 404 until that scan passes it. Run the
|
||||
driver from the host shell; the section keys are `1` Movies, `3` TV Shows,
|
||||
`5` Audiobooks, `6` Music, `11` VODs & Live Events:
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
|
||||
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
|
||||
cat >/tmp/pxscan.sh <<'EOS'
|
||||
read -r T
|
||||
cfg=$(mktemp)
|
||||
mk() { { printf 'header = "X-Plex-Token: %s"\nheader = "Accept: application/json"\n' "$T"
|
||||
printf 'url = "%s"\n' "$1"; } > "$cfg"; }
|
||||
mk "http://127.0.0.1:32400/library/sections"
|
||||
for k in $(curl -s -K "$cfg" | grep -o '"key":"[0-9]*"' | cut -d\" -f4); do
|
||||
mk "http://127.0.0.1:32400/library/sections/$k/refresh"
|
||||
printf 'request = "POST"\n' >> "$cfg"
|
||||
echo "plex section $k: $(curl -s -o /dev/null -w '%{http_code}' -K "$cfg")"
|
||||
done
|
||||
shred -u "$cfg"
|
||||
EOS
|
||||
podman cp /tmp/pxscan.sh plex:/tmp/pxscan.sh
|
||||
podman exec -i plex sh /tmp/pxscan.sh </tmp/px.tok
|
||||
shred -u /tmp/px.tok
|
||||
scripts/media-refresh-plex.sh trigger --library 1
|
||||
python3 scripts/media-verify.py --db-kind plex \
|
||||
--journal /mnt/entropy/.media-campaign/<journal>.jsonl # the real proof
|
||||
```
|
||||
|
||||
Every section answers `204` — or `200` with a Directory body when
|
||||
`Accept: application/json` is sent. Inside the quoted `printf` format a
|
||||
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
|
||||
one glued line and one stray backslash is a config file curl rejects, and the
|
||||
failure prints as `option -K: is unknown`, never as a bad config line.
|
||||
Exactly ONE section: a refresh costs a full walk of that tree, and refreshing all
|
||||
five for a rename that touched one buys nothing. `trigger` answers
|
||||
`refresh accepted (204)` — `200` when a body is requested — and `000` means PMS is
|
||||
still warming, which the driver retries out loud; a service restart is never the
|
||||
fix.
|
||||
|
||||
**`status` is a census, not a waiter: no epoch makes it watch a scan.** It reads
|
||||
`section_locations.scanned_at` — the epoch of the last scan PMS has *recorded* for the
|
||||
section, the same value `GET /library/sections` prints as `scannedAt` (verified
|
||||
2026-11-04) — and compares it against `--after`. Measured twice on 2026-11-04: a chain
|
||||
fell through its wait loop in the same second its own trigger was accepted, and a direct
|
||||
read 30 minutes later still answered `settled` off that one stamp
|
||||
(`2026-11-04T19:46:59Z`, for a request accepted at `19:46:58Z`) while the walk ran. A
|
||||
journal's mtime fails, and so does the trigger's own epoch — the request satisfies the
|
||||
very door it was asked to wait for. The failure direction is a false *success*, never a
|
||||
false alarm, which is what makes it worth the warning: ask `status` whether a scan got
|
||||
recorded, and never let a chain depend on it.
|
||||
|
||||
**The proof that a scan did its work is the journal replayed against the database:**
|
||||
`scripts/media-verify.py --db-kind plex --journal <journal>` names every row that still
|
||||
points at a source path (`still at src`) and every entry whose file is physically still
|
||||
there (`REAL`). Both at zero is what gates a purge, and it needs no settle signal at all
|
||||
— a scan can clear rows but cannot invent one. Jellyfin's door has no such trap:
|
||||
`scripts/media-refresh-jellyfin.py wait` watches the scan task itself and refuses by
|
||||
name when no task reaches `Running`.
|
||||
|
||||
The driver ships its runner into the container with `podman cp` rather than
|
||||
quoting a command through `podman exec sh -c`, because PMS's HTTPD closes without
|
||||
a single byte on a request that arrives from another netns — the host address and
|
||||
the `10.89.x.x` container address both read as "empty reply", never as a 401 —
|
||||
and because a three-layer `ssh → podman exec → sh -c` quote chain dies on a parse
|
||||
error long before it reaches the server. The request therefore rides plain
|
||||
`http://127.0.0.1:32400` from inside the container, where insecure connections
|
||||
are allowed (gotcha 5). The credential is `PlexOnlineToken` in the container's
|
||||
`Preferences.xml` — NOT `PlexHTTPToken`, which a claimed server need not carry —
|
||||
read inside the container into a mode-600 curl config there, so it never enters
|
||||
argv (`/proc/<pid>/cmdline` is world-readable) and never a URL (proxies log
|
||||
URLs). Hand-writing that config has one more trap: inside a quoted `printf`
|
||||
format a backslash-newline is LITERAL, so a two-line string becomes one glued
|
||||
line plus a stray backslash, and curl answers `option -K: is unknown` — never
|
||||
"bad config line".
|
||||
|
||||
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
||||
replays the run's journal against the database — the only read that
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Outpost, proxy-provider, OAuth2, LDAP and flow configuration for t
|
||||
tags:
|
||||
- infrastructure
|
||||
- authentik
|
||||
timestamp: 2026-10-19
|
||||
timestamp: 2026-11-03
|
||||
---
|
||||
|
||||
# Authentik Outpost Reference
|
||||
@@ -56,7 +56,7 @@ correctly — the outpost matches on hostname.
|
||||
|
||||
## Provisioning rules for proxy providers
|
||||
|
||||
Two provisioning rules, each verified live by an outage:
|
||||
Three provisioning rules, each verified live by an outage or a dead login:
|
||||
|
||||
1. **All three flows are required.** A ProxyProvider created via raw API/ORM
|
||||
with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
|
||||
@@ -80,6 +80,16 @@ Two provisioning rules, each verified live by an outage:
|
||||
the app. No error is logged anywhere — the SSO "works" and the app never
|
||||
appears. Create the Application with `provider_id` set (the UI does this
|
||||
automatically; raw API/ORM drives must do it explicitly).
|
||||
3. **A raw ORM create skips the OAuth defaults.** The REST serializer calls
|
||||
`ProxyProvider.set_oauth_defaults()`; `ProxyProvider.objects.create()` does not. The row
|
||||
then looks fully configured — external_host, internal_host, application bound, outpost
|
||||
claimed — while `redirect_uris` is empty (those URIs are *derived* from `external_host` by
|
||||
`_get_callback_url`) and `grant_types` is `[]`. Both kill the guard before any credential,
|
||||
in this order: authentik renders its own **"Redirect URI Error"** page, and once the URIs
|
||||
are filled by hand the next wall is a bodiless **HTTP 400** out of `/application/o/authorize/`
|
||||
(both measured on the Audiobookshelf guard provider, 2026-11-03). Census and repair in one idempotent pass, all
|
||||
providers at once: `scripts/authentik-proxy-provider-defaults.sh`; `--check` prints the
|
||||
census and exits 1 on drift, and a provider that already matches is left alone.
|
||||
|
||||
## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
|
||||
|
||||
@@ -138,36 +148,46 @@ Read from the live database (read-only `SELECT`s against the
|
||||
| Object | Value |
|
||||
|---|---|
|
||||
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
|
||||
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
|
||||
| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — an earlier revision's "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
|
||||
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — an earlier revision recorded `intent=verification`; the live value is `app_password` |
|
||||
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` — **verified by a live bind 2026-11-04** (the bind and the `memberOf` search both succeed, so the `ou=users` path is right) |
|
||||
| Service account | `svc-jellyfin-ldap` (user pk 8). It has a usable password and authenticates through the default flow — that password is held by Jellyfin in the plugin config (`LdapBindPassword`) and is what the bind actually uses (live bind 2026-11-04) |
|
||||
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — and **unused by Jellyfin**, which has no token field (see the rule below) |
|
||||
| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
|
||||
| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
|
||||
| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (an earlier revision said slug `jellyfin`, which does not exist) |
|
||||
|
||||
Hard-won rules (the first is now contested by the live database):
|
||||
Hard-won rules:
|
||||
|
||||
- **Token intent: contested, do not trust either value blindly.** An earlier
|
||||
revision asserted that a bind token with `intent=app_password` is rejected by
|
||||
the flow's password stage with `Invalid password` and only
|
||||
`intent=verification` passes. The live token for `svc-jellyfin-ldap` is
|
||||
`app_password` (`authentik_core_token`, 2026-10-18), so either the rule is
|
||||
wrong or the current Jellyfin bind path is not that token. Nothing observable
|
||||
settles it today: the LDAP outpost's retained log holds only its 5-minute
|
||||
`initialized direct binder` / `Update providers` config cycles and no bind
|
||||
activity *(unverified 2026-10-18: no bind was performed — reproducing it
|
||||
requires the bind secret)*. Re-verify with a real bind before changing the
|
||||
token's intent either way.
|
||||
- **Token intent is moot for Jellyfin.** The earlier argument (a bind token with
|
||||
`intent=app_password` is rejected by the flow's password stage, only
|
||||
`intent=verification` passes) never applied here: LDAP-Auth sends the service
|
||||
account's **password** from `LdapBindPassword`, not a token. A live bind plus a
|
||||
`memberOf=cn=jellyfin_users` search (11 entries) succeeded from Dankserver on
|
||||
2026-11-04. Leave the token row alone.
|
||||
- **Search needs `search_full_directory`.** Without the RBAC grant the
|
||||
outpost returns only the bind user itself — a `(memberOf=…)` filter then
|
||||
matches zero entries and the plugin logs `Found no users matching X`.
|
||||
The permission is object-level on the provider; grant it via an RBAC role
|
||||
and retest.
|
||||
- **The outpost log never shows a bind.** Its retained journal is only the
|
||||
5-minute `initialized direct binder` / `Update providers` config cycle; the
|
||||
absence of a bind line is not evidence of anything. Instruments that do answer:
|
||||
`ldapsearch` with the plugin's own DN, password and filter, and the authentik
|
||||
event registry (`login` / `login_failed` with `client_ip` from the podman
|
||||
gateway, since the published port SNATs).
|
||||
- `ak shell` import path for the provider model in 2026.8:
|
||||
`from authentik.providers.ldap.models import LDAPProvider`.
|
||||
- (unverified 2026-10-18: importing it outside a configured Django shell fails
|
||||
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
|
||||
auth service for this check)*
|
||||
`from authentik.providers.ldap.models import LDAPProvider` (verified inside a
|
||||
live `ak shell`, 2026-11-04 — it only fails outside a configured Django shell).
|
||||
- **`ak shell` field names, 2026.8:** `Event` is keyed by `created` and `action`
|
||||
— there is no `timestamp` and no `type`; `Outpost` has `providers` but **no
|
||||
`apps`**. Introspect first: `[f.name for f in Model._meta.get_fields()]`.
|
||||
- **Never pipe `ak shell` inside a heredoc.** `podman exec -i … ak shell | grep
|
||||
-v '^{'` binds the heredoc to `grep` (the redirection belongs to the LAST
|
||||
element of the pipeline), so the shell gets no stdin and prints nothing at all.
|
||||
And `2>/dev/null` on `ak shell` eats the traceback, which looks identical to "no
|
||||
rows". Clean pattern: have the script write its results to a file and read that
|
||||
file back —
|
||||
`podman exec -i authentik-server ak shell >/dev/null 2>/dev/null <<'PY' … open("/tmp/x.txt","w") … PY`
|
||||
then `podman exec authentik-server cat /tmp/x.txt`.
|
||||
|
||||
## Self-serve user-settings flow (password change)
|
||||
|
||||
|
||||
@@ -145,18 +145,17 @@ relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
|
||||
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
|
||||
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
|
||||
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
|
||||
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
|
||||
single-platform digest**
|
||||
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
|
||||
(`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26`, matching the
|
||||
running image in `podman ps`, 2026-10-18). The digest in this page before 2026-10-18
|
||||
(`…fa33bc8f…`) is superseded. The reason for pinning a digest at all: podman
|
||||
5.7.0 refuses the plain tag because it resolves to a *nested* manifest list
|
||||
(index → per-platform manifest lists) — `unexpectedly received a manifest list
|
||||
instead of a manifest for a single image` — first hit on `:2026.8.0`, still the
|
||||
rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
|
||||
<image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") |
|
||||
.digest'` and pin that digest.
|
||||
`Restart=always`, container up, re-checked 2026-11-04). **The image pin has one
|
||||
owner: the `Image=` line of
|
||||
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container`** — never restate
|
||||
the digest here, because every pin bump turns this page into a lie (the digest
|
||||
cited here went stale twice: `…fa33bc8f…`, then `…2e7b259e…`). The reason a
|
||||
digest is pinned at all: podman 5.7.0 refuses the plain tag because it resolves
|
||||
to a *nested* manifest list (index → per-platform manifest lists) —
|
||||
`unexpectedly received a manifest list instead of a manifest for a single image`
|
||||
— first hit on `:2026.8.0` and still the rule. Resolve once with
|
||||
`podman manifest inspect <image> | jq -r '.manifests[] |
|
||||
select(.platform.architecture=="amd64") | .digest'` and pin that digest.
|
||||
|
||||
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
|
||||
(`PublishPort=389:3389`, `636:6636`
|
||||
@@ -393,9 +392,10 @@ downstream —
|
||||
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
|
||||
every human account).
|
||||
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
|
||||
debo, freya, mitch, eric). The pks in this page are confirmed against
|
||||
`authentik_core_user` on 2026-10-18: ridley 13, kassie 14, alex 15,
|
||||
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
|
||||
debo, freya, mitch, eric, jason). The pks were confirmed against
|
||||
`authentik_core_user` on 2026-10-18 and the table re-read 2026-11-04: ridley
|
||||
13, kassie 14, alex 15, artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21,
|
||||
mitch 22, eric 23, **jason 24 (created 2026-11-04)** —
|
||||
and every one carries the `<user>@franksdank.cloud` address the template
|
||||
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
|
||||
- (unverified 2026-10-18)*. Template: `username` +
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Caddy is the single reverse proxy for both hosts."
|
||||
tags:
|
||||
- infrastructure
|
||||
- caddy
|
||||
timestamp: 2026-08-29
|
||||
timestamp: 2026-11-03
|
||||
---
|
||||
|
||||
# Caddy
|
||||
@@ -90,7 +90,111 @@ pihole.franksdank.cloud {
|
||||
|
||||
After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.
|
||||
|
||||
Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile:14-24,283-423`). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), and the media/*arr vhosts. Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik (`dankclaw/Caddyfile:122-132`). See [[Homepage]] and [[Authentik]].
|
||||
Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`, `notebook.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile`: the header comment and the two internal listeners). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), the media/*arr vhosts, and `audiobooks.` — Audiobookshelf enforces its own auth (local accounts plus authentik OIDC) and must answer native app clients, podcast RSS and OPDS, which the chain structurally cannot deliver: it answers a client that solves no JavaScript proof-of-work with challenge HTML wherever that client expects JSON or a socket payload ([[Audiobookshelf]]). Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik. See [[Homepage]] and [[Authentik]].
|
||||
|
||||
## Guard a new vhost
|
||||
|
||||
Order is the whole procedure: the IdP row has to exist **before** the route flips, or
|
||||
authenticated traffic reaches the outpost with no matching provider and the user gets
|
||||
authentik's error page instead of the app.
|
||||
|
||||
1. **Provider first.** A proxy provider plus an application, claimed by the embedded
|
||||
outpost. No blueprint owns this state — it lives only in the IdP's database, which is
|
||||
why it is easy to forget:
|
||||
|
||||
```
|
||||
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
|
||||
from authentik.providers.proxy.models import ProxyProvider
|
||||
from authentik.core.models import Application
|
||||
from authentik.outposts.models import Outpost
|
||||
NAME, EXT = "example", "https://example.franksdank.cloud/"
|
||||
pp = ProxyProvider.objects.filter(name=f"{NAME}-proxy").first()
|
||||
if pp is None:
|
||||
pp = ProxyProvider.objects.create(name=f"{NAME}-proxy", external_host=EXT,
|
||||
internal_host="http://caddy:8017")
|
||||
if not Application.objects.filter(provider=pp).exists():
|
||||
Application.objects.create(name=f"{NAME.title()} Guard", slug=f"{NAME}-guard",
|
||||
provider=pp)
|
||||
o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
|
||||
o.providers.add(pp) # an unclaimed provider is invisible to the outpost
|
||||
print(pp.name, pp.external_host, o.providers.count())
|
||||
PY
|
||||
```
|
||||
|
||||
`external_host` is the exact vhost URL **with the trailing slash**: the embedded
|
||||
outpost routes on the ORIGINAL `Host` header alone, so anything else matches zero
|
||||
providers. Never print a provider's fields wholesale — a proxy provider carries a
|
||||
client secret, and it lands in whatever transcript the command runs in.
|
||||
|
||||
`ProxyProvider.objects.create()` is not the API: the serializer's
|
||||
`set_oauth_defaults()` never runs, so the row lands with `redirect_uris` and
|
||||
`grant_types` empty, and the guard fails *before* any credential — authentik's
|
||||
"Redirect URI Error" page, then a bodiless HTTP 400 once the URIs are filled
|
||||
by hand. Run `scripts/authentik-proxy-provider-defaults.sh` after the snippet
|
||||
(idempotent, covers every provider; `--check` is the census); law and failure
|
||||
signatures in [[Authentik Outpost Reference]].
|
||||
2. **Then the route.** The public site block proxies the whole request to `anubis:8923`
|
||||
with `header_up X-Real-Ip {remote_host}`, and the post-auth `http://:8017` listener
|
||||
gains a `@<name> host <vhost>` handler pointing at the app — copy `romm`/`files`,
|
||||
which are the same shape including a Dankserver LAN-port upstream.
|
||||
An app that derives its own origin from the request needs the scheme pinned:
|
||||
Audiobookshelf checks every OIDC callback against `x-forwarded-proto://Host`; this
|
||||
listener is fed over plain http so Caddy's own `X-Forwarded-Proto` is `http`, and the
|
||||
app answers a 400 "Invalid callback URL - must be same-origin" for its own https
|
||||
callback (measured 2026-11-03, while this vhost was still guarded — it is exempt
|
||||
now, and the same pin sits on its edge block instead).
|
||||
The edge for a guarded name is TLS-only, so `header_up X-Forwarded-Proto
|
||||
https` is the truthful value ([[Audiobookshelf]]).
|
||||
3. **Then the name and the audit.** A `dankclaw/pihole/addn-hosts` line (the `pihole`
|
||||
seam fails the build if a Caddyfile vhost has no record), a Porkbun A record if the
|
||||
name must resolve off-LAN ([[LAN-DNS]]), and the subdomain added to `GUARDED` in
|
||||
`tests/e2e/specs/14-guard.spec.ts` with an app-specific leak marker — that spec is
|
||||
the deploy gate's proof that the guard, not the app, answers an anonymous client.
|
||||
|
||||
## Unguard a vhost
|
||||
|
||||
The same three artifacts, in the opposite order. Getting it wrong takes the app
|
||||
down for browsers while looking healthy:
|
||||
|
||||
1. **Route first.** Move the public site block off `anubis:8923` to a direct
|
||||
`reverse_proxy` (keeping any origin-pinning `header_up` the app needs), delete
|
||||
the `@<name>` handler from the `http://:8017` listener, and deploy. Traffic
|
||||
stops entering the auth listener at `:8016`, so the IdP row goes unused — and
|
||||
nothing breaks.
|
||||
2. **Then the IdP rows**, once the route is verified:
|
||||
|
||||
```
|
||||
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
|
||||
from authentik.providers.proxy.models import ProxyProvider
|
||||
from authentik.outposts.models import Outpost
|
||||
pp = ProxyProvider.objects.filter(name="audiobooks-proxy").first()
|
||||
if pp:
|
||||
o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
|
||||
o.providers.remove(pp)
|
||||
app = pp.application # the "<Name> Guard" application row
|
||||
if app:
|
||||
app.delete()
|
||||
pp.delete()
|
||||
print("providers now claimed:", Outpost.objects.get(
|
||||
managed="goauthentik.io/outposts/embedded").providers.count())
|
||||
PY
|
||||
```
|
||||
|
||||
Deleting the provider **before** the route flips is the wrong order: the
|
||||
outpost then matches zero providers for a Host that still arrives, and every
|
||||
browser gets authentik's error page instead of the app.
|
||||
3. **Then the audit lists.** Drop the subdomain from `GUARDED` in
|
||||
`tests/e2e/specs/14-guard.spec.ts` (and its leak marker), and replace it
|
||||
there with an exemption assertion — the deploy gate must keep proving the
|
||||
exemption is deliberate, the way `plex` does: app bytes to an anonymous
|
||||
client, zero challenge HTML, and the app's own `401` on an unauthenticated
|
||||
API call rather than the gate's `200`.
|
||||
|
||||
What the exemption costs, said plainly: the app's login endpoint becomes
|
||||
internet-visible with no rate limiter in front of it unless the edge carries one
|
||||
(this Caddy build has no `rate_limit` module) or the app ships a lockout (ABS
|
||||
2.37.2 does not). That is the same exposure `jelly.`, `plex.` and the *arr
|
||||
vhosts already live with; the mitigation is the credentials.
|
||||
|
||||
## Operations
|
||||
|
||||
@@ -113,7 +217,10 @@ Certificates and ACME state live in the caddy data volume (`data/caddy/`). Back
|
||||
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit, and the repo has no `dankclaw/compose.yaml` left to re-add it to (`ls`, 2026-10-18).
|
||||
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
|
||||
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
|
||||
- A public vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]]. The deliberate exception is an **internal-only** name: `hermes.franksdank.cloud` and `hermesadmin.franksdank.cloud` are in `addn-hosts` and the Caddyfile but have no public record, which is what keeps them LAN-only — DNS-01 issuance needs no public A record (`dankclaw/Caddyfile:173-179`).
|
||||
- **Exposure has exactly two gates: the router's 80/443 forward to Caddy, and Caddy's vhost list.** Caddy matches on SNI/Host, so *every* name in the Caddyfile already answers a WAN client that sends that name — a public A record buys **discoverability, not access** (measured 2026-11-04). A published name needs a Pi-hole `addn-hosts` line, a Porkbun A record and the Caddyfile block ([[LAN-DNS]]); DNS-01 issuance needs no public record, so an **internal-only** name is one with no A record. Posture measured 2026-11-04: the whole authentik-guarded set is published — `www.`, `search.`, `romm.`, `files.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `notebook.` — while the names with no public record are the self-authenticating exempt ones (`pihole.`, `hermes.`, `hermesadmin.`, `prowlarr.`, `radarr.`, `lidarr.`, `readarr.`, `qbittorrent.`) and the legacy redirects (`dankclaw.`, `dankserver.`). Every A record is a copy of the WAN address, which is dynamic — republish after an IP change.
|
||||
- **A missing DNS record is not a gate: the names are public anyway.** Caddy issues one certificate per name and every issuance is published in Certificate Transparency logs — `crt.sh` for `%.franksdank.cloud` lists the live *and* retired set (`sonarr.`, `comfyui.`, `chat.`, `jellyfin.`, the whole *arr stack), so "LAN-only because there is no A record" is fiction and a stranger can send the Host. A household-only service needs a **source gate in its vhost**, which is what the *arr stack and qBittorrent now carry: a LAN client proxies through, anything off-LAN gets `404`. Keep those five gated even though they have no A record.
|
||||
- **The LAN gate is `remote_ip`, and it reads the true client.** The G36 DNATs the 80/443 forward straight to Caddy with no proxy hop, so `@lan remote_ip 127.0.0.1/32 ::1/128 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12` sees either a LAN address or the internet address — never the router's. Verify the split without waiting for a WAN client: a LAN request must reach the upstream (a dead upstream still proves it passed the gate, because the answer is its `502`, not the gate's `404`), and an external fetcher must get the `404`.
|
||||
- **A new record looks like it failed for up to the SOA's negative TTL.** The zone's negative cache is 1800 s, so a resolver that asked for the name *before* the create keeps answering NXDOMAIN (1.1.1.1 did, while Quad9 and the authoritative NS had the record seconds after the create) — check `dig @<authoritative NS>` before re-creating anything.
|
||||
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]]. *(unverified 2026-10-18: hairpin behaviour is only observable from a WAN-side client; nothing on the LAN or in this repo can prove it)*
|
||||
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
|
||||
- `transport http {versions h2c 1.1}` on a vhost makes Caddy send an h2c
|
||||
|
||||
@@ -28,11 +28,12 @@ descriptions and a `docs` link to `/docs#models`.
|
||||
|
||||
**Sections + WIP split (2026-09-21):** both pages now group their panels
|
||||
under section headings rendered by shared `renderSection()`/`renderPanel()`
|
||||
components in `app.js`. The user page orders **AI, Web Apps, Media Library
|
||||
Apps**; the admin page orders **Media Library Apps, Observability, AI,
|
||||
Infrastructure**. The WIP badge split into **WIP: OPEN** (work in progress)
|
||||
and **WIP: CLOSED** (work finished by decision — e.g. a card that is
|
||||
deliberately unprobeable) via the `wip` value (`"open"` / `"closed"`).
|
||||
components in `app.js`. The user page orders **Web Apps, Media Library
|
||||
Apps, AI, Work in progress** (the three not-open-yet apps moved under their
|
||||
own heading, below AI, on 2026-11-04); the admin page orders **Media Library
|
||||
Apps, Observability, AI, Infrastructure**. The WIP badge split into **WIP:
|
||||
OPEN** (work in progress) and **WIP: CLOSED** (work finished by decision — e.g.
|
||||
a card that is deliberately unprobeable) via the `wip` value.
|
||||
Tautulli (Plex statistics, `192.168.0.235:8181`) joined the admin page's
|
||||
Media Library Apps section.
|
||||
|
||||
@@ -314,9 +315,10 @@ flowchart TD
|
||||
## Service classification
|
||||
|
||||
**User services** (root page), in section order:
|
||||
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-dragoon, frank-archon, frank-zealot — status + docs link, no Open link).
|
||||
- **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
|
||||
- **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
|
||||
- **Media Library Apps** — Plex, Jellyfin, Audiobookshelf (moved up from the admin page 2026-11-03, once Authentik SSO made it a household app).
|
||||
- **AI** — OpenWebUI, the three LLM fleet cards (frank-dragoon, frank-archon, frank-zealot) and `frank-tts`, the cloned-voice endpoint (moved up from the admin page 2026-11-04 so the household sees whether the voice door is up). All four are status + docs link, no Open link; the fused **Model Information** panel renders under this heading.
|
||||
- **Work in progress** — Calibre-Web, RomM, Seerr. WIP badge, no Open link, no household account yet: the ask-Frank door. See the docs page's *Three doors that are not open to the household yet*.
|
||||
|
||||
**Admin services** (`/admin/`), in section order:
|
||||
- **Media Library Apps** — the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr), qBittorrent, and Tautulli — live on FrankBook-Pro and Dankserver, operator-only: no vhosts (Tautulli included), direct IP:port access only. See [[FrankBook-Pro]].
|
||||
@@ -324,6 +326,12 @@ flowchart TD
|
||||
- **AI** — Hermes, ComfyUI.
|
||||
- **Infrastructure** — Caddy, dnsmasq, Gluetun, and the low-level backends (mariadb, valkey, Postgres). The backends show status only. They get a link only where a UI is meaningful.
|
||||
|
||||
**The registry is the authority.** `shared/homepage/services.json` decides which
|
||||
page a card renders on (`category`), whether it shows a WIP badge instead of live
|
||||
status (`wip`), and whether it gets an Open link (`showLink` + `url`). The two
|
||||
lists above are a locator only; when they disagree with the registry, the page is
|
||||
the bug — fix the page.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
@@ -426,15 +434,22 @@ agent knowledge base (`~/.pi/agent/docs/caddy/caddyfile.md`).
|
||||
|
||||
- `category` — `user` or `admin`. Selects the page.
|
||||
- `section` — the heading the panel renders under. Must be one of the page's
|
||||
`SECTION_ORDER` entries (user: AI, Web Apps, Media Library Apps; admin:
|
||||
Media Library Apps, Observability, AI, Infrastructure), or the panel is
|
||||
silently dropped from the page.
|
||||
`SECTION_ORDER` entries (user: Web Apps, Media Library Apps, AI, Work in
|
||||
progress; admin: Media Library Apps, Observability, AI, Infrastructure), or
|
||||
the panel is silently dropped from the page.
|
||||
- `url` — the "Open" link (the Caddy site).
|
||||
- `docs` — the per-service docs page (`/docs/<svc>/`).
|
||||
- `probe` — the backend `host:port` the checker probes.
|
||||
- `method` — `tcp` or `http`. The probe type for this service (Q10).
|
||||
- `wip` — `false`, or `"open"` / `"closed"`. When set, the badge shows
|
||||
**WIP: OPEN** or **WIP: CLOSED** and the checker skips the probe.
|
||||
- `wip` — `false`, or `"open"` / `"closed"` / `"down"`. When set, the badge shows
|
||||
**WIP: OPEN** / **WIP: CLOSED** / **WIP: DOWN** and the checker skips the probe.
|
||||
- `model` — present only on Model Information cards. Such a card renders in the
|
||||
fused `renderModelInfo()` panel (`mi-*` styles) instead of the app grid, and its
|
||||
`description` is unused. `unit` is the StarCraft identity: art
|
||||
`/img/unit-<unit>.jpg`, accent `.mi-<unit>`, voice lines from `/sfx/`. A card
|
||||
with no `unit` is a tool (today: `frank-tts`) — art falls back to the card's own
|
||||
`art`, the accent is `.mi-tool`, and a `tagline` with no `sfx` renders as plain
|
||||
text rather than a play button.
|
||||
|
||||
**A card must be probeable from the monitor's host, or it must be `wip`.** The
|
||||
checker runs on Dankclaw, so its reach is: that host's own loopback, its own
|
||||
|
||||
@@ -197,23 +197,18 @@ and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip.
|
||||
**Post-boot gates.** 0 failed units, `systemd` running. 32 IOMMU groups — 31
|
||||
`identity` + 1 `DMA`, and that one belongs to the Genesys Logic SD card reader
|
||||
(`c2:00.0`, a kernel quirk, not a fleet device). The iGPU (`c5:00.0`) sits in group
|
||||
20 `identity` with ATS capability; interrupt remapping is enabled. frank-dragoon's
|
||||
boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5
|
||||
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
|
||||
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
|
||||
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
|
||||
slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
|
||||
boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
|
||||
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working)
|
||||
see [[frank-dragoon]].)
|
||||
20 `identity` with ATS capability; interrupt remapping is enabled. The flip moved
|
||||
frank-dragoon's boot ledger by nothing measurable — its shape and every number in it are
|
||||
owned by [[frank-dragoon]] — and Gitea, the Caddy edge and Grafana all answered 200.
|
||||
|
||||
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
|
||||
Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized
|
||||
amdxdna_accel_driver 0.7.0 … on minor 0`, IOMMU group 26 `identity`, PASID cap width
|
||||
10 (`Exec- Priv-`). `/dev/accel/accel0` is `crw-rw---- root:render 261,0` and `dank`
|
||||
is in `render`, so a quadlet needs `AddDevice=/dev/accel/accel0` +
|
||||
`GroupAdd=keep-groups`. **No NPU endpoint is deployed** — stays the
|
||||
history of the retired endpoint; what changed is only its precondition.
|
||||
`GroupAdd=keep-groups`. The flip brought the device back, not a service: no NPU chat
|
||||
endpoint was deployed then. The XDNA2 lane returned to work a day later as frank-dragoon's
|
||||
small-model lane ([[frank-dragoon]]) — the shape and its laws are owned there.
|
||||
|
||||
**The `pp_dpm_fclk` pin (level 7, 2000 MHz) went in on 2026-10-19**, with the first thing that needed it: the
|
||||
vendor's root host unit, installed and enabled by `frank-dragoon`'s deployer while its NPU lane is named. Pinning
|
||||
|
||||
@@ -20,7 +20,7 @@ timestamp: 2026-10-19
|
||||
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
|
||||
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
|
||||
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
|
||||
PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
|
||||
PRO R9700s. Capacity contract **4 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
|
||||
is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws).
|
||||
A payload swap is a `MODEL_FILE` line plus a re-download — see Eras.
|
||||
|
||||
@@ -525,6 +525,21 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
|
||||
the disk tier silently never opens. The tier does survive a payload swap when arch and
|
||||
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
|
||||
- **A radiance crash reads as a clean exit unless you know where to look, and the engine never
|
||||
says why.** The unit's `ExecStart` ends in `exec podman wait`, so `systemctl show -p
|
||||
ExecMainStatus` reports **0** after a container death — that is `podman wait`'s own status, not
|
||||
the engine's — and `Result=success` beside `NRestarts=1` is the *normal* fingerprint of a
|
||||
crash. The container's real code is the bare digit `podman wait` prints to stdout
|
||||
(`serve-frank-archon.sh[NNNN]: 1`). The engine's fatal path writes **nothing** to its log: no
|
||||
stderr line, no message before the exit and no core. The only copy of the reason is the error
|
||||
the client received — `the engine stopped on an error and is not serving (<errno>)`, where the
|
||||
parenthesis is the failing call's errno text (`invalid argument` = EINVAL, 2026-11-04). So
|
||||
grep the *client* logs before concluding the journal is empty: omp keeps the provider error in
|
||||
`~/.omp/logs/omp.<date>.<pid>.log` (`agent turn ended with provider error`), and OpenWebUI
|
||||
shows it as the chat's error toast. Seen once (2026-11-04, restart counter 1 after 3d20h up):
|
||||
three concurrent triple-digit-thousand-token agent turns prefilled at once, prefill rate
|
||||
collapsed, all in-flight streams failed at the same instant, then a silent teardown — kernel
|
||||
logged only the KFD SVM workqueue churning, with no OOM, GPU hang, MCE or swap pressure.
|
||||
|
||||
### Dead ends — do not re-try
|
||||
|
||||
|
||||
@@ -107,6 +107,12 @@ job.sh watch # refreshing status table
|
||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
||||
every failure disappeared once the script was a file.
|
||||
- **A script fed over stdin owns that stdin.** `ssh host 'bash -s' <<'EOS'` hands
|
||||
the script itself to stdin, so any child that reads stdin — `podman exec -i`,
|
||||
`ssh -t`, a bare `read` — swallows everything after it and the remainder
|
||||
vanishes without a word. It reads exactly like a hang: the commands before the
|
||||
child print, the ones after never existed. Drop `-i`, or give every child
|
||||
`< /dev/null`.
|
||||
|
||||
## Verify before trusting it (3am law)
|
||||
|
||||
|
||||
@@ -44,6 +44,7 @@ is a locator: one row, one clause, no numbers.
|
||||
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Audiobookshelf]] | Audiobookshelf on Dankserver: quadlet shape, array bind, SSD state, API bootstrap, authentik SSO, public vhost, laws. |
|
||||
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
|
||||
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
|
||||
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
||||
@@ -52,9 +53,11 @@ is a locator: one row, one clause, no numbers.
|
||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
||||
| [[Media Duplicate Prune]] | Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge. |
|
||||
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
||||
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||
| [[Media TV Relayout]] | Re-file TVShows from scene packs into the shape Jellyfin reads: authority order, refusals, the two-series split test. |
|
||||
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
|
||||
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
|
||||
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
|
||||
|
||||
@@ -10,7 +10,7 @@ const CATEGORY = window.location.pathname.indexOf('/admin') !== -1 ? 'admin' : '
|
||||
// same renderSection()/renderPanel() below. SECTION_ORDER fixes each page's
|
||||
// section order; a section with no services renders nothing.
|
||||
const SECTION_ORDER = {
|
||||
user: ['Web Apps', 'Media Library Apps', 'AI'],
|
||||
user: ['Web Apps', 'Media Library Apps', 'AI', 'Work in progress'],
|
||||
admin: ['Media Library Apps', 'Observability', 'AI', 'Infrastructure'],
|
||||
};
|
||||
|
||||
@@ -79,15 +79,17 @@ function voiceBtn(text, files) {
|
||||
|
||||
function miCard(svc, state) {
|
||||
const m = svc.model;
|
||||
const art = svc.art || '/img/unit-' + m.unit + '.jpg';
|
||||
const parts = ['<div class="mi-top">' +
|
||||
'<img class="mi-art" src="/img/unit-' + esc(m.unit) + '.jpg" alt="' + esc(m.unit) + '">' +
|
||||
'<img class="mi-art" src="' + esc(art) + '" alt="' + esc(m.unit || svc.name) + '">' +
|
||||
'<div class="mi-head"><div class="mi-id"><span class="badge ' + state + '">' + BADGE_LABEL[state] + '</span>' +
|
||||
'<span class="mi-name">' + esc(svc.name) + '</span></div>' +
|
||||
'<div class="mi-role">' + esc(m.role) + '</div>' +
|
||||
(m.hf ? '<a class="mi-hf" href="' + esc(m.hf) +
|
||||
'" target="_blank" rel="noopener">model on huggingface ↗</a>' : '') +
|
||||
'</div></div>',
|
||||
voiceBtn(m.tagline, m.sfx)
|
||||
(m.sfx && m.sfx.length ? voiceBtn(m.tagline, m.sfx)
|
||||
: m.tagline ? '<div class="mi-tag">' + esc(m.tagline) + '</div>' : '')
|
||||
];
|
||||
if (m.equiv) parts.push('<div class="mi-equiv">' + esc(m.equiv) + '</div>');
|
||||
if (m.spec) parts.push('<div class="mi-spec">' + esc(m.spec) + '</div>');
|
||||
@@ -99,7 +101,7 @@ function miCard(svc, state) {
|
||||
if (m.warn) parts.push('<details class="mi-warn"><summary>⚠ privacy note</summary>' +
|
||||
'<div class="warn-body">' + esc(m.warn) + '</div></details>');
|
||||
const card = document.createElement('article');
|
||||
card.className = 'mi-card mi-' + esc(m.unit);
|
||||
card.className = 'mi-card mi-' + esc(m.unit || 'tool');
|
||||
card.innerHTML = parts.join('');
|
||||
return card;
|
||||
}
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"host": "dankserver",
|
||||
"services": [
|
||||
{
|
||||
"name": "Homepage",
|
||||
"url": "/",
|
||||
"description": "This page. Password-protected service index.",
|
||||
"docs": "serverAppConfigs-docs/Infrastructure/Homepage.md"
|
||||
},
|
||||
{
|
||||
"name": "RomM",
|
||||
"url": "http://romm.franksdank.cloud",
|
||||
"description": "Retro game manager and player for the ROM collection.",
|
||||
"docs": "serverAppConfigs-docs/Dankserver/RomM.md"
|
||||
},
|
||||
{
|
||||
"name": "Copyparty",
|
||||
"url": "http://files.franksdank.cloud",
|
||||
"description": "High-speed file sharing and streaming server.",
|
||||
"docs": "serverAppConfigs-docs/Dankserver/Copyparty.md"
|
||||
},
|
||||
{
|
||||
"name": "Calibre-Web",
|
||||
"url": "http://books.franksdank.cloud",
|
||||
"description": "E-book library browser and reader.",
|
||||
"docs": "serverAppConfigs-docs/Dankserver/Calibre-Web.md"
|
||||
},
|
||||
{
|
||||
"name": "Plex",
|
||||
"url": "http://plex.franksdank.cloud/web/index.html",
|
||||
"description": "Media server (existing host service, proxied).",
|
||||
"docs": "serverAppConfigs-docs/Dankserver/Dankserver Stack.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -17,191 +17,458 @@
|
||||
</header>
|
||||
<main class="docs">
|
||||
|
||||
<section id="getting-in">
|
||||
<h1>Getting in</h1>
|
||||
<p>Everything here lives behind two doors. The first is <strong>Anubis</strong>, a
|
||||
quiet bot-check: the first time you visit any guarded page in a session you see a
|
||||
short "making sure you're not a bot" screen. It runs once. The second door is
|
||||
<strong>Authentik</strong>, the family single sign-on: log in once and every
|
||||
guarded service (this homepage, search, chat, media requests, and friends) opens
|
||||
for you without a second password.</p>
|
||||
<ul>
|
||||
<li>Need an account? Ask Frank. Accounts are issued by hand; there is no signup.</li>
|
||||
<li>Your clearance lives in a cookie. Clearing cookies means re-clearing both doors.</li>
|
||||
<li>If a page asks you to log in repeatedly, that is a server-side problem — tell Frank, don't brute-force it.</li>
|
||||
</ul>
|
||||
<h1>The household guide</h1>
|
||||
<p class="sub">Everything running on the computers in this house, what it is for,
|
||||
and which door to knock on. Written for people who did not build it.</p>
|
||||
|
||||
<section id="start">
|
||||
<h2>Start here</h2>
|
||||
<p class="kicker">Three steps and you are in</p>
|
||||
<p><strong>1. Open the front door:</strong>
|
||||
<a href="https://www.franksdank.cloud">https://www.franksdank.cloud</a><br>
|
||||
It works from anywhere — phone data, a hotel, a friend's house. The first page you
|
||||
see is a robot check: your browser does a few seconds of arithmetic so the internet's
|
||||
scanners can't wander in. Then type the family username and password Frank gave you.
|
||||
There is no signup button on purpose.</p>
|
||||
<p><strong>2. Pick what you want to do:</strong></p>
|
||||
<table>
|
||||
<tr><th>I want to…</th><th>Open</th><th>Needs its own login?</th></tr>
|
||||
<tr><td>watch a film or a show</td><td><a href="https://jelly.franksdank.cloud">Jellyfin</a></td><td>no — your franksdank.cloud login</td></tr>
|
||||
<tr><td>watch it on a TV that already has Plex</td><td><a href="https://plex.franksdank.cloud/web">Plex</a></td><td>yes — a Plex account, Frank invites it</td></tr>
|
||||
<tr><td>listen to an audiobook</td><td><a href="https://audiobooks.franksdank.cloud/audiobookshelf">Audiobookshelf</a></td><td>no — the family password</td></tr>
|
||||
<tr><td>ask a computer a question, privately</td><td><a href="https://ai.franksdank.cloud">AI chat</a></td><td>no</td></tr>
|
||||
<tr><td>search the web without being followed</td><td><a href="https://search.franksdank.cloud">Search</a></td><td>no</td></tr>
|
||||
<tr><td>read a news article without the junk</td><td><a href="https://marreta.franksdank.cloud">Marreta</a></td><td>no</td></tr>
|
||||
<tr><td>grab a file off the big disk</td><td><a href="https://files.franksdank.cloud">Files</a></td><td>no (read-only)</td></tr>
|
||||
<tr><td>ebooks, retro games, request a film</td><td>not open to the household yet — <a href="#notopen">see below</a></td><td>—</td></tr>
|
||||
</table>
|
||||
<p><strong>3. On your phone,</strong> open the front door in Safari or Chrome and choose
|
||||
<em>Add to Home Screen</em>. You get an icon that behaves like an app. Details in
|
||||
<a href="#phone">On your phone</a>.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/homepage.webp" alt="The homepage: one tile per service, each with a short description">
|
||||
<figcaption>The front door itself: one tile per thing the house runs, and a line about what
|
||||
each is for. The badges say whether it's awake.</figcaption>
|
||||
</figure>
|
||||
<div class="toc">
|
||||
<a href="#doors">The two doors</a>
|
||||
<a href="#phone">On your phone</a>
|
||||
<a href="#jellyfin">Jellyfin</a>
|
||||
<a href="#plex">Plex</a>
|
||||
<a href="#audiobookshelf">Audiobookshelf</a>
|
||||
<a href="#notopen">Not open yet</a>
|
||||
<a href="#files">Files</a>
|
||||
<a href="#search">Search</a>
|
||||
<a href="#marreta">Read an article</a>
|
||||
<a href="#ai">AI chat</a>
|
||||
<a href="#gitea">Gitea</a>
|
||||
<a href="#models">The models</a>
|
||||
<a href="#trouble">When something misbehaves</a>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="troubleshooting">
|
||||
<h1>When something looks broken</h1>
|
||||
<ul>
|
||||
<li><strong>A status card says DOWN.</strong> Reload the page once; the checker
|
||||
re-probes every few minutes. If it stays down, that service really is unhappy — tell Frank.</li>
|
||||
<li><strong>"status unavailable" on the homepage.</strong> The page couldn't read
|
||||
its status file. Hard-reload (Ctrl/Cmd-Shift-R).</li>
|
||||
<li><strong>Connection timeout on any device.</strong> Our names only resolve
|
||||
on the home network (or the VPN). If you are on guest Wi-Fi, a phone hotspot,
|
||||
or out of the house, the pages will not load — that is the security working.
|
||||
Connect to the house network and retry.</li>
|
||||
<li><strong>The bot-check spins forever.</strong> Let it finish; heavy JS blocking
|
||||
(privacy extensions) can starve it. Pause the blocker, reload, resume.</li>
|
||||
</ul>
|
||||
<section id="doors">
|
||||
<h2>The two doors</h2>
|
||||
<p>Almost everything here is behind two checks, one after the other.</p>
|
||||
<ul>
|
||||
<li><strong>Anubis</strong> is the bouncer. It makes a browser prove it is a browser
|
||||
(that robot-check page) and stops the automatic scanners that comb the internet for
|
||||
open servers. It never asks for a password.</li>
|
||||
<li><strong>Authentik</strong> is the front desk. It asks who you are, once per device
|
||||
per few days, and after that you drift from app to app without being stopped again.</li>
|
||||
</ul>
|
||||
<p>One account opens: the homepage, search, Marreta, the AI chat, Files, Gitea, and Jellyfin
|
||||
(there it is typed as username + password, not the automatic sign-in).</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/door-authentik.webp" alt="The sign-in screen: an identification field, then a password field">
|
||||
<figcaption>The front desk. Your username first, then your password. Nothing here signs
|
||||
you up for anything; if you don't have an account yet, ask Frank.</figcaption>
|
||||
</figure>
|
||||
<p class="key-warn">Two doors keep their own accounts and will not take the family
|
||||
password: <strong>Plex</strong> (a Plex account, its own e-mail
|
||||
login) and the e-book door. Frank creates those by hand. <strong>Jellyfin does take the
|
||||
family login</strong> — your franksdank.cloud username and password.</p>
|
||||
<p class="foot-note">Want your own password? It lives at
|
||||
<a href="https://auth.franksdank.cloud/if/flow/default-user-settings-flow/">account settings</a>
|
||||
— the same button you get at the bottom of the homepage.</p>
|
||||
</section>
|
||||
|
||||
<section id="media">
|
||||
<h1>Media & files</h1>
|
||||
<ul>
|
||||
<li><strong>Movies & TV:</strong> request with <a href="https://seerr.franksdank.cloud">Seerr</a>;
|
||||
watch with <a href="https://plex.franksdank.cloud/web/index.html">Plex</a> or
|
||||
<a href="https://jelly.franksdank.cloud">Jellyfin</a>. Seerr is the front door —
|
||||
requests ride the VPN tunnel from there.</li>
|
||||
<li><strong>Books:</strong> <a href="https://books.franksdank.cloud">Calibre-Web</a> — the e-book library, with its own login.</li>
|
||||
<li><strong>Games:</strong> <a href="https://romm.franksdank.cloud">RomM</a> — ROM library and emulator collection.</li>
|
||||
<li><strong>Raw files:</strong> <a href="https://files.franksdank.cloud">Copyparty</a> — direct downloads and shares.</li>
|
||||
<li><strong>Search:</strong> <a href="https://search.franksdank.cloud">SearXNG</a> — metasearch, VPN-routed, no AI snoop. <details class="key-warn"><summary>⚠ privacy note</summary><div class="warn-body">Searches are obfuscated, not anonymous: SearXNG hides you from the upstream engines and the VPN tunnel hides the traffic from the LAN — but the upstream search API keys are tied to Frank’s identity. Don’t treat it as Tor.</div></details></li>
|
||||
</ul>
|
||||
<section id="phone">
|
||||
<h2>On your phone</h2>
|
||||
<p><strong>Get the icon.</strong> In the browser: <em>Share</em> (iPhone) or the
|
||||
<em>⋮</em> menu (Android) → <em>Add to Home Screen</em>. Same trick on a tablet.
|
||||
There is no app store anything to install for the household doors; they are web pages
|
||||
wearing a good coat.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/phone-homepage.webp" alt="The homepage on a phone, one row per service">
|
||||
<figcaption>The front door on a 5-inch screen. Same tiles, same links, no app.</figcaption>
|
||||
</figure>
|
||||
<p><strong>Apps that do have a real client</strong> (better than the web page on a phone):</p>
|
||||
<table>
|
||||
<tr><th>What</th><th>Where</th><th>Server address to type</th></tr>
|
||||
<tr><td>Jellyfin</td><td>Google Play & the App Store (also Fire TV, Roku, most smart TVs)</td><td><code>https://jelly.franksdank.cloud</code></td></tr>
|
||||
<tr><td>Plex</td><td>on nearly every TV ever sold</td><td>nothing — sign in to your Plex account</td></tr>
|
||||
<tr><td>Audiobookshelf</td><td>Google Play (Android, solid). The iPhone build is a beta with a limited number of seats — use the browser, or SoundLeaf / ShelfPlayer</td><td><code>https://audiobooks.franksdank.cloud/audiobookshelf</code></td></tr>
|
||||
</table>
|
||||
<p><strong>The one address that only works at home.</strong> The raw file shelf also speaks
|
||||
WebDAV on <code>http://192.168.0.235:3923/</code> — that's what a real file-manager app
|
||||
mounts. Two catches, both measured: it is <em>http</em>, not https (the phone will warn you;
|
||||
it is fine on the house Wi-Fi), and it answers only on the house network. From elsewhere the
|
||||
public door hands a file app the robot-check page instead of files, because a file app can't
|
||||
solve it. Off-site, use the browser.</p>
|
||||
<ul>
|
||||
<li>iPhone: the built-in Files app does <em>not</em> do WebDAV. <em>FileBrowser</em> or
|
||||
<em>Owlfiles</em> do.</li>
|
||||
<li>Android: <em>Cx File Explorer</em>, <em>Material Files</em>, <em>Solid Explorer</em> all do.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="chat">
|
||||
<h1>Talking to the AI</h1>
|
||||
<p>The everyday front door is <a href="https://ai.franksdank.cloud">Open WebUI</a>,
|
||||
the family chat app. Open the address, click <strong>Continue with authentik</strong>,
|
||||
and use the same single sign-on as everything else — no separate password. Your
|
||||
conversations are saved to your account, so history follows you between devices.</p>
|
||||
<img src="img/openwebui-chat.png" alt="Open WebUI chat screen with a question and the AI's answer">
|
||||
<p>Below the text box, the model box on the right chooses which model answers you.
|
||||
Family accounts get <code>frank-zealot</code> — the fleet's fast all-rounder,
|
||||
which also sees images: drag a photo or screenshot straight into the chat box.
|
||||
Frank's account can additionally reach the bigger models.</p>
|
||||
<img src="img/openwebui-model-picker.png" alt="The Open WebUI welcome screen with the family model selected in the model box">
|
||||
<p>Open WebUI keeps your chat history; the raw model endpoints (used by developer
|
||||
tools, see <a href="#api-access">API access</a>) do not.</p>
|
||||
<section id="jellyfin">
|
||||
<h2>Jellyfin — films and TV</h2>
|
||||
<p><a href="https://jelly.franksdank.cloud">jelly.franksdank.cloud</a> — free, no
|
||||
account of any company's, and the posters and episode guides are all filled in.</p>
|
||||
<h3>Getting in</h3>
|
||||
<ul>
|
||||
<li>Pick your name. There is no list of names on the login screen — type yours.</li>
|
||||
<li>Type your <strong>franksdank.cloud password</strong> — the same one that opens the
|
||||
homepage, search and the AI chat. There is no separate Jellyfin password.</li>
|
||||
<li>Refused, but the same password opens the homepage? Your name hasn't been given library
|
||||
access yet — ask Frank, it's a one-line change on his side.</li>
|
||||
<li>No name yet? Ask Frank to make one. There is no self-service signup, on purpose.</li>
|
||||
</ul>
|
||||
<figure>
|
||||
<img src="/img/onboarding/jellyfin-login.webp" alt="The Jellyfin login screen: a username field and a password field">
|
||||
<figcaption>The empty login. Username, password, <em>Sign in</em>. Nothing to register.</figcaption>
|
||||
</figure>
|
||||
<h3>The first ten minutes</h3>
|
||||
<ul>
|
||||
<li>Home shows <em>Continue watching</em> and what's new. Press a poster; it starts.</li>
|
||||
<li>The <strong>⋮</strong> menu on a poster is where the useful things live: mark
|
||||
watched, add to a playlist, download it to the device.</li>
|
||||
<li>On a TV the app asks for a <em>Server address</em>: <code>https://jelly.franksdank.cloud</code>.</li>
|
||||
<li>Settings → theme: the big-screen look is installed for the living-room box.
|
||||
Themes are per device, so change it only on the one you're sitting in front of.</li>
|
||||
</ul>
|
||||
<h3>If it misbehaves</h3>
|
||||
<ul>
|
||||
<li>It rejects a password that works everywhere else? Your name isn't on the library-access
|
||||
list yet — ask Frank.</li>
|
||||
<li>A film with a blank poster is missing artwork, not missing files — Frank can refresh it.</li>
|
||||
<li>Buffering away from home usually means the pipe out of the house, not the server.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="plex">
|
||||
<h2>Plex — the same shelf, different shop window</h2>
|
||||
<p><a href="https://plex.franksdank.cloud/web">plex.franksdank.cloud/web</a> bounces you to
|
||||
Plex's own sign-in, and that is correct: Plex is closed-source and insists on a Plex account.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/plex-login.webp" alt="Plex's own sign-in page on plex.tv">
|
||||
<figcaption>Plex keeps its own front desk. You need a Plex account (an e-mail address is
|
||||
enough); Frank invites it into the library.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Jellyfin and Plex read <strong>the same shelf of files</strong>. The same films are in
|
||||
both; the artwork, watch-state and profiles are separate.</li>
|
||||
<li>Choose Plex when your TV already has the Plex app on it, or you want Plex's trailers
|
||||
and music. Otherwise Jellyfin is the one the household uses.</li>
|
||||
<li>Need a Plex account of your own? Ask Frank to invite it; he can also tell you which of
|
||||
the two has the thing you're looking for.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="audiobookshelf">
|
||||
<h2>Audiobookshelf — audiobooks and podcasts</h2>
|
||||
<p><a href="https://audiobooks.franksdank.cloud/audiobookshelf">audiobooks.franksdank.cloud/audiobookshelf</a>
|
||||
— the family password works here.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/audiobookshelf-login.webp" alt="The Audiobookshelf login page with username and password fields">
|
||||
<figcaption>Two ways in: the password Frank gave you, or <em>Sign in with Authentik</em>,
|
||||
which is the same family account.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Your place in every book follows you between phone, laptop and the kitchen tablet.</li>
|
||||
<li>On the phone the Android app is the best version: it will do <em>downloads for offline
|
||||
listening</em>. A web page can't play offline; if you need it for a flight, use an app.</li>
|
||||
<li>iPhone: the official build is a TestFlight beta with a small number of seats, so either
|
||||
use the browser + <em>Add to Home Screen</em>, or a third-party client
|
||||
(<em>SoundLeaf</em>, <em>ShelfPlayer</em>).</li>
|
||||
<li>Adding the server in any app: <code>https://audiobooks.franksdank.cloud/audiobookshelf</code>.</li>
|
||||
</ul>
|
||||
<figure>
|
||||
<img src="/img/onboarding/audiobookshelf-library.webp" alt="An empty audiobook library with a scan button">
|
||||
<figcaption>An empty library. Books appear when Frank scans, and then they're yours to keep playing.</figcaption>
|
||||
</figure>
|
||||
</section>
|
||||
|
||||
<section id="notopen">
|
||||
<h2>Three doors that are not open to the household yet</h2>
|
||||
<ul>
|
||||
<li><strong>Calibre-Web</strong> (e-books) —
|
||||
<a href="https://books.franksdank.cloud">books.franksdank.cloud</a>. Its own login, and
|
||||
Frank hasn't decided who gets in. Ask.</li>
|
||||
<li><strong>RomM</strong> (retro games, plays them in the browser) —
|
||||
<a href="https://romm.franksdank.cloud">romm.franksdank.cloud</a>. Needs its own account,
|
||||
and it is not open to family users yet. Ask.</li>
|
||||
<li><strong>Seerr</strong> (the "would you watch this?" request box) —
|
||||
<a href="https://seerr.franksdank.cloud">seerr.franksdank.cloud</a>. Not open yet: tell
|
||||
Frank what you want to watch and he'll add it.</li>
|
||||
</ul>
|
||||
<figure>
|
||||
<img src="/img/onboarding/calibre-login.webp" alt="The Calibre-Web login page">
|
||||
<figcaption>What a closed door looks like: a login that wants credentials of its own.</figcaption>
|
||||
</figure>
|
||||
</section>
|
||||
|
||||
<section id="files">
|
||||
<h2>Files — the raw shelf</h2>
|
||||
<p><a href="https://files.franksdank.cloud">files.franksdank.cloud</a> is the honest view of
|
||||
the big disk: folders, dates, sizes, no cleverness. <strong>Read-only</strong> for the
|
||||
household — you can take things out, nothing you do here deletes anything.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/copyparty.webp" alt="A folder listing with sizes, dates and a download link on each row">
|
||||
<figcaption>Each row has a download link; the <em>zip</em> link at the bottom takes the
|
||||
whole folder as one archive. <em>control-panel</em> and the <em>up</em> link navigate.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Good for: a photo you want the original of, a document, a game ROM, a song to put on another device.</li>
|
||||
<li>Bad for: watching video (the player is the wrong tool — use Jellyfin) and for
|
||||
saving your own work (it can't write).</li>
|
||||
<li>Works in any browser, on any machine, from anywhere.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="search">
|
||||
<h2>Search — SearXNG</h2>
|
||||
<p><a href="https://search.franksdank.cloud">search.franksdank.cloud</a> asks a lot of search
|
||||
engines at once and merges the answers, then leaves no trail behind. Outbound requests ride
|
||||
the house VPN.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/search-results.webp" alt="Search results merged from many engines, with engine checkboxes on the left">
|
||||
<figcaption>One question, many engines. The boxes on the left choose which engines answer.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Make it your browser's search engine: browser settings → search → add a custom
|
||||
engine with <code>https://search.franksdank.cloud/search?q=%s</code>.</li>
|
||||
<li>On a phone there is no app; bookmark it or put the homepage on your home screen.</li>
|
||||
</ul>
|
||||
<figure>
|
||||
<img src="/img/onboarding/phone-search.webp" alt="A search result page on a phone">
|
||||
<figcaption>Same results, thumb-sized.</figcaption>
|
||||
</figure>
|
||||
<details class="key-warn">
|
||||
<summary>The privacy catch worth knowing</summary>
|
||||
<div class="warn-body">
|
||||
<p>This hides your question from Google, Bing and the rest — it does not hide it from
|
||||
<em>this house</em>. Queries are logged on this server so Frank can tell a broken search
|
||||
from a slow one. And because results are fetched from here, the search engines see one
|
||||
house address, not your phone: your habits don't get stitched to your face. If you need a
|
||||
genuinely anonymous search, that's Tor or a paid service, not this.</p>
|
||||
</div>
|
||||
</details>
|
||||
</section>
|
||||
|
||||
<section id="marreta">
|
||||
<h2>Read an article without the junk — Marreta</h2>
|
||||
<p><a href="https://marreta.franksdank.cloud">marreta.franksdank.cloud</a>. Paste a link, get
|
||||
the article: no cookie banner, no video autoplay, no sixteen sidebars. It also runs an RSS
|
||||
reader once Frank feeds it news.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/marreta.webp" alt="Marreta's empty home page with a URL box and an Add as app hint">
|
||||
<figcaption>Paste the link. The app itself offers <em>Add as app</em> in its menu, which is
|
||||
the cleanest way to keep it on a phone.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Some pages with hard paywalls still come back thin. Nothing in the house can fix that.</li>
|
||||
<li>A white screen after pasting? Pause your ad blocker for this site — the app warns
|
||||
about exactly that.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="ai">
|
||||
<h2>AI chat — Open WebUI</h2>
|
||||
<p><a href="https://ai.franksdank.cloud">ai.franksdank.cloud</a>. The family password, or
|
||||
<em>Continue with authentik</em> on the login card, which is the same thing.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/openwebui-login.webp" alt="The Open WebUI welcome card with a Continue with authentik button">
|
||||
<figcaption>Press <em>Continue with authentik</em> and the family login does the rest.</figcaption>
|
||||
</figure>
|
||||
<ul>
|
||||
<li>Nothing you type here goes to a company's server. It goes to the computers in this house.</li>
|
||||
<li>Install it as an app: in Chrome or Edge, the menu → <em>Install site as an app</em>;
|
||||
in Safari, share → <em>Add to Home Screen</em>.</li>
|
||||
<li>Conversations are kept per account, so your history is yours and Frank's is Frank's.</li>
|
||||
</ul>
|
||||
<h3>Which model do I pick?</h3>
|
||||
<p>You are offered all three, and the app opens on <code>frank-dragoon</code>. Leave it there
|
||||
unless you have a reason. Short version (longer version in <a href="#models">The models</a>):</p>
|
||||
<ul>
|
||||
<li><code>frank-dragoon</code> — fast, sees pictures, doesn't lecture. Good default.</li>
|
||||
<li><code>frank-archon</code> — the big careful one on the two-card server. Slower per
|
||||
answer, best at long and difficult work. Polite by construction.</li>
|
||||
<li><code>frank-zealot</code> — small, quick, holds a very long conversation. Also
|
||||
doesn't lecture.</li>
|
||||
</ul>
|
||||
<p class="key-warn">Two of the three will answer almost anything, including adult and dangerous
|
||||
material, without a warning. That is deliberate for Frank's work, not a toy for a child:
|
||||
conversations stay on the house machines and are not visible to Frank's phone. If that isn't
|
||||
the setup you want for a kid, keep them off this door and ask Frank about a filtered one.</p>
|
||||
|
||||
</section>
|
||||
|
||||
<section id="gitea">
|
||||
<h2>Gitea — Frank's workshop</h2>
|
||||
<p><a href="https://gitea.franksdank.cloud">gitea.franksdank.cloud</a> holds the code that runs
|
||||
this house: the configs, the deploy scripts, the tests. It has its own accounts (the top right
|
||||
offers <em>Register</em>, or <em>Sign in with OpenID</em> to ride the family login), but it is a
|
||||
workshop, not a household door. You very likely don't need it.</p>
|
||||
<figure>
|
||||
<img src="/img/onboarding/gitea-signin.webp" alt="The Gitea sign-in form with a password box and a Sign in with OpenID button">
|
||||
<figcaption>If you ever do need it: <em>Register</em> for its own account, or
|
||||
<em>Sign in with OpenID</em> for the family one.</figcaption>
|
||||
</figure>
|
||||
</section>
|
||||
|
||||
<section id="models">
|
||||
<h1>The running models</h1>
|
||||
<p>Three models serve the family, named for the Protoss units of StarCraft.
|
||||
The homepage "Model Information" panel shows whether each one is up right now —
|
||||
and plays the unit's voice if you click its quoted line.</p>
|
||||
<h2>The models</h2>
|
||||
<p class="kicker">What's actually answering you</p>
|
||||
<p>Three machines in this house do the thinking. None of them phone a company. Each is a
|
||||
different trade-off of speed, memory and how much it can hold in its head at once; the chat
|
||||
app above is just a window onto them.</p>
|
||||
<table>
|
||||
<tr><th>Name</th><th>Runs on</th><th>Good at</th><th>Notes</th></tr>
|
||||
<tr>
|
||||
<td><code>frank-dragoon</code></td>
|
||||
<td>the Strix Halo machine's built-in graphics</td>
|
||||
<td>fast answers, long chats, reading pictures, tools</td>
|
||||
<td>two conversations at a time; abliterated, so no refusals</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>frank-archon</code></td>
|
||||
<td>the media server, two big graphics cards</td>
|
||||
<td>the hardest work: long code, careful reasoning, big documents</td>
|
||||
<td>four conversations at a time; keeps its guardrails</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>frank-zealot</code></td>
|
||||
<td>the RTX 3090 in the corner</td>
|
||||
<td>very long conversations, cheap and quick; it is what Frank's own coding agent runs on</td>
|
||||
<td>three conversations at 200,000 tokens each; no refusals</td>
|
||||
</tr>
|
||||
</table>
|
||||
<p class="foot-note">The exact weights: <code>frank-dragoon</code> is a halogen conversion of
|
||||
Qwen3.8-Flash-Next v2, abliterated
|
||||
(<a href="https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated">Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated</a>);
|
||||
<code>frank-archon</code> is the official Qwen3.8-Flash-Next re-packed for that engine
|
||||
(<a href="https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe">StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe</a>);
|
||||
<code>frank-zealot</code> is Ternary-Bonsai-2-9B in a 2-bit packing
|
||||
(<a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-9B-gguf">prism-ml/Ternary-Bonsai-2-9B-gguf</a>)
|
||||
with a small adapter fitted that removes its refusal reflex. All three accept pictures. Ask
|
||||
Frank before pointing a script at one of them directly; the address is on the homepage.</p>
|
||||
|
||||
<p class="foot-note">Naming: the Open WebUI picker and the raw API show
|
||||
exactly the Protoss names used on this page (renamed 2026-10-09). The name you
|
||||
read here is the name you type into a config file.</p>
|
||||
|
||||
<h2>frank-zealot — the fast utility model</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf" target="_blank" rel="noopener">Ternary-Bonsai-2-27B</a> (PQ2_0 ternary weights) on the RTX 3090 — roughly 77 tokens per second.</li>
|
||||
<li><strong>Equivalent:</strong> last year's frontier level (2025).</li>
|
||||
<li><strong>Capacity:</strong> 200k-token context, two submissions at a time; sees images.</li>
|
||||
<li><strong>Inputs:</strong> text, image.</li>
|
||||
<li><strong>Best for:</strong> code editor autocomplete, agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence, and anything with a picture. This is the model Open WebUI offers to family accounts.</li>
|
||||
<li><strong>Compaction:</strong> if your agent harness supports setting a compaction model, set this one. A compaction job severely slows the bigger models for far longer than it slows this one.</li>
|
||||
<li><strong>Supervision:</strong> requires tight human supervision.</li>
|
||||
<li><strong>Warning:</strong> <details class="key-warn"><summary>⚠ censored base model</summary><div class="warn-body">Base-alignment (censored) model: refusals are possible on sensitive topics, and any model can still be steered into risky output. Uncensored work belongs on frank-dragoon. Take this warning seriously — don’t ruin this server for everybody.</div></details></li>
|
||||
</ul>
|
||||
|
||||
<h2>frank-dragoon — deep reasoning</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> <a href="https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated" target="_blank" rel="noopener">Qwen3.8-Flash-Next v2 native abliterated</a> (halogen flash-server, ~62 GiB) on the Strix Halo iGPU — the base checkpoint itself, converted to the engine’s native format and censorship-removed, thinking before it answers.</li>
|
||||
<li><strong>Equivalent:</strong> Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier intelligence).</li>
|
||||
<li><strong>Capacity:</strong> 200k-token context, two submissions at a time; sees images.</li>
|
||||
<li><strong>Inputs:</strong> text, image.</li>
|
||||
<li><strong>Best for:</strong> complex agentic tasks, programming, system administration, and "claw"-style personal agents.</li>
|
||||
<li><strong>Supervision:</strong> safe for autonomous, low-supervision use.</li>
|
||||
<li><strong>Warning:</strong> <details class="key-warn"><summary>⚠ uncensored model</summary><div class="warn-body">Uncensored model: output can include adult content, content that describes harmful, hateful, or illegal acts, content concerning “sensitive offensive cybersecurity topics”, and other output that may carry severe legal risk. Take this warning seriously — don’t ruin this server for everybody.</div></details></li>
|
||||
|
||||
<h2>frank-archon — the big one</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> our own build — a GPTQ-w4 quant of <a href="https://huggingface.co/frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700" target="_blank" rel="noopener">Swift1.5-Qwen3.8-Flash-Next</a>, quantized and tuned in-house (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — the biggest brain on the network and the fastest runtime.</li>
|
||||
<li><strong>Equivalent:</strong> Claude Opus 4.8 / OpenAI-Terra (Summer 2026 frontier intelligence).</li>
|
||||
<li><strong>Capacity:</strong> 200k-token context, three submissions at a time; sees images.</li>
|
||||
<li><strong>Inputs:</strong> text, image.</li>
|
||||
<li><strong>Best for:</strong> software engineering, system administration, and instruction tasks. <strong>⚠ Ask Frank for permission before use</strong> — this is his primary model; he uses it more than anything else on the fleet.</li>
|
||||
</ul>
|
||||
|
||||
<p class="foot-note">Which one to pick? Small utility work, autocomplete, RAG,
|
||||
programming, and personal agents: <code>frank-dragoon</code>. Software
|
||||
engineering, sysadmin, and instruction tasks: <code>frank-archon</code> (ask
|
||||
Frank first). Every name on this page is the id the Open WebUI picker and
|
||||
the raw API answer to.</p>
|
||||
<h3>And the voices</h3>
|
||||
<p>Two speech models live in this house, and they are different animals. One of them
|
||||
speaks as a <em>specific person</em>; the other is a pack of stock voices that ships inside
|
||||
the tools you already use.</p>
|
||||
<table>
|
||||
<tr><th>Name</th><th>Runs on</th><th>Good at</th><th>Notes</th></tr>
|
||||
<tr>
|
||||
<td><code>frank-tts</code></td>
|
||||
<td>the Strix Halo machine's built-in graphics, sharing the card with <code>frank-dragoon</code></td>
|
||||
<td>cloning: it speaks as one particular voice, chosen by name</td>
|
||||
<td>five voices today — <code>dr.kern</code>, <code>garrickhagon</code>,
|
||||
<code>jeffhays</code>, <code>vaati</code>, <code>wilwheaton</code>; ask for one by name and
|
||||
you get it. No key, but it is a house-hardware door: it answers on the house network only</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Kokoro</strong></td>
|
||||
<td>your own browser, or Frank's own terminal — no server in the loop</td>
|
||||
<td>instant plain narration, quick and cheap</td>
|
||||
<td>about two dozen preset voices with coded names (<code>af_heart</code>,
|
||||
<code>bm_george</code>…). It cannot clone, and it will never answer to the five names
|
||||
above</td>
|
||||
</tr>
|
||||
</table>
|
||||
<ul>
|
||||
<li><strong>The AI chat's speaker button is Kokoro.</strong> That is why it offers a
|
||||
different list of names, and why it starts talking a moment before anything on the server
|
||||
could. It speaks in your browser, on your machine.</li>
|
||||
<li><strong>The podcast maker uses the clones.</strong> An episode's two hosts are
|
||||
<code>wilwheaton</code> and <code>vaati</code>, rendered by <code>frank-tts</code>. That door
|
||||
is Frank's for now.</li>
|
||||
<li><strong>Frank's coding agent narrates through the clones too</strong> — the default
|
||||
voice is <code>wilwheaton</code>.</li>
|
||||
<li><strong>Want your own voice in the list?</strong> Ask Frank. It needs about twenty
|
||||
seconds of you talking plainly, once; the voice then carries your name.</li>
|
||||
</ul>
|
||||
<p class="foot-note">The exact weights: <code>frank-tts</code> is a Qwen3 talker
|
||||
(1.7 billion parameters, 4-bit) plus its 12 Hz audio codec, driven by
|
||||
<code>qwentts.cpp</code>; Kokoro is a much smaller 82-million-parameter model whose voices are
|
||||
baked in. A cloned voice is not a setting — it is a directory of reference audio on the
|
||||
server, which is why adding one is a fifteen-minute job and not a checkbox. Programmers on the
|
||||
house network: <code>POST http://192.168.0.143:7772/v1/audio/speech</code> with
|
||||
<code>{"input": "…", "voice": "vaati"}</code> gets you WAV or MP3, and
|
||||
<a href="http://192.168.0.143:7772/v1/audio/voices"><code>/v1/audio/voices</code></a> answers
|
||||
the live list, so this page can never lie to you for long.</p>
|
||||
</section>
|
||||
|
||||
<section id="api-access">
|
||||
<h1>API access — the AI on your own laptop</h1>
|
||||
<p>Developer assistants like <strong>Opencode</strong> and <strong>Pi</strong> are
|
||||
chat programs that run on your own computer and can read your files, run commands,
|
||||
and edit code while they talk. They are all built to speak the same standard
|
||||
language as the big commercial products (the "OpenAI-compatible" API), and our
|
||||
server speaks it too. You give the program three things — an address, a personal
|
||||
key, and a model name — and your laptop thinks through our hardware.</p>
|
||||
|
||||
<h2>Step 1 — get your personal key</h2>
|
||||
<ol>
|
||||
<li>Open <a href="https://ai.franksdank.cloud">ai.franksdank.cloud</a> and log in.</li>
|
||||
<li>Click your circle (bottom-left corner) → <strong>Settings</strong> → <strong>Account</strong>.</li>
|
||||
<li>Scroll to <strong>API keys</strong> → press <strong>Show</strong> (it flips to
|
||||
<strong>Hide</strong>) — the key appears as a row of dots.</li>
|
||||
<li>Press the <strong>copy icon</strong> beside the key (the eye previews it;
|
||||
the <strong>⋯</strong> menu rotates it later). The key looks like <code>sk-3f2a…</code>.</li>
|
||||
</ol>
|
||||
<img src="img/openwebui-api-keys.png" alt="The Open WebUI Account settings page, scrolled to the API keys section with the masked key, preview, copy and rotate controls">
|
||||
<p class="key-warn">The key is a password. Anyone who has it can use our AI as
|
||||
you. Paste it only into the program's config file — never into a shared chat,
|
||||
a screenshot, or a git repository.</p>
|
||||
|
||||
<h2>Step 2 — the three details every tool asks for</h2>
|
||||
<ul>
|
||||
<li><strong>Base URL:</strong> <code>https://ai.franksdank.cloud/api/v1</code></li>
|
||||
<li><strong>Model:</strong> <code>frank-zealot</code> — the family utility model (this is the id the API expects)</li>
|
||||
<li><strong>API key:</strong> the <code>sk-…</code> from step 1 — sent as the header <code>Authorization: Bearer sk-…</code></li>
|
||||
</ul>
|
||||
<p>Proof that it works, from any terminal (replace the key):</p>
|
||||
<pre><code>curl https://ai.franksdank.cloud/api/v1/chat/completions \
|
||||
<h2>Let your own tools use it</h2>
|
||||
<p class="kicker">For the household's programmers</p>
|
||||
<p>Frank can hand you an API key. It works on one door
|
||||
— <code>https://ai.franksdank.cloud/api/v1</code> — and speaks the plain OpenAI
|
||||
dialect, so anything that can talk to a hosted model can talk to the house.</p>
|
||||
<pre><code>curl https://ai.franksdank.cloud/api/v1/chat/completions \
|
||||
-H "Authorization: Bearer PASTE-YOUR-KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "frank-zealot", "messages": [{"role": "user", "content": "Hello!"}]}'</code></pre>
|
||||
|
||||
<h2>Step 3a — Opencode</h2>
|
||||
<p>Install once (<code>npm install -g opencode-ai</code>), then create the file
|
||||
<code>~/.config/opencode/opencode.json</code> with:</p>
|
||||
<pre><code>{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"frank": {
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "Frank fleet",
|
||||
"options": {
|
||||
"baseURL": "https://ai.franksdank.cloud/api/v1",
|
||||
"apiKey": "PASTE-YOUR-KEY"
|
||||
},
|
||||
"models": {
|
||||
"frank-zealot": { "name": "frank-zealot" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}</code></pre>
|
||||
<p>Start <code>opencode</code> and pick the model with <code>/model</code> —
|
||||
it appears as <code>frank/frank-zealot</code>.</p>
|
||||
|
||||
<h2>Step 3b — Pi</h2>
|
||||
<p>Create the file <code>~/.pi/agent/models.json</code> with:</p>
|
||||
<pre><code>{
|
||||
-d '{
|
||||
"model": "frank-dragoon",
|
||||
"messages": [{"role": "user", "content": "What is on the shelf?"}]
|
||||
}'</code></pre>
|
||||
<p>For <a href="https://github.com/badlogic/pi-mono">Pi</a>, drop this in
|
||||
<code>models.json</code>:</p>
|
||||
<pre><code>{
|
||||
"providers": {
|
||||
"frank": {
|
||||
"baseUrl": "https://ai.franksdank.cloud/api/v1",
|
||||
"api": "openai-completions",
|
||||
"apiKey": "PASTE-YOUR-KEY",
|
||||
"models": [
|
||||
{ "id": "frank-zealot" }
|
||||
{ "id": "frank-dragoon" }
|
||||
]
|
||||
}
|
||||
}
|
||||
}</code></pre>
|
||||
<p>Then start Pi with <code>pi --model frank/frank-zealot</code>.</p>
|
||||
<p>Then start Pi with <code>pi --model frank/frank-dragoon</code>.</p>
|
||||
<p class="foot-note">The key is for the house AI only — it is not your Authentik password,
|
||||
and losing it costs nothing beyond asking Frank to rotate it. Want a different model than the
|
||||
one you were given? Ask Frank to widen the key.</p>
|
||||
</section>
|
||||
|
||||
<p class="foot-note">The key works only for the family AI — it is not your
|
||||
Authentik password, and losing it costs nothing beyond asking Frank to rotate it.
|
||||
Want a bigger model (<code>frank-archon</code>, <code>frank-dragoon</code>)? Ask Frank to raise your access.</p>
|
||||
<section id="trouble">
|
||||
<h2>When something misbehaves</h2>
|
||||
<ul>
|
||||
<li><strong>The robot check page keeps coming back.</strong> Your browser is blocking the
|
||||
scripts it needs (or an extension is). Allow scripts for that site, or try another browser.
|
||||
Still looping? Ask Frank — the guard can be told you're a person.</li>
|
||||
<li><strong>"Invalid password" on the front desk.</strong> Right password, wrong door: the
|
||||
family login does not open Plex or the e-book door. Jellyfin does take it, but only for the
|
||||
names Frank has given library access.</li>
|
||||
<li><strong>The AI is slow.</strong> Each model only holds a couple of conversations at once
|
||||
(two, three or four, depending which). It is a queue, not a crash — or Frank is
|
||||
mid-conversation; the big one is on the same machine as the films.</li>
|
||||
<li><strong>The AI invents something confidently.</strong> All three are small models living
|
||||
on house hardware; they will make things up rather than say "I don't know". Check anything
|
||||
that matters.</li>
|
||||
<li><strong>A film is missing.</strong> It is probably not on the shelf yet, or it landed
|
||||
yesterday and the library hasn't been told. Say so and Frank will scan.</li>
|
||||
<li><strong>A file app can't reach the shelf from outside the house.</strong> Expected: the
|
||||
public door is browser-shaped. Use the browser, or mount the home address on the house Wi-Fi.</li>
|
||||
<li><strong>Something is genuinely down.</strong> The homepage badge says
|
||||
<span style="color: var(--down)">DOWN</span> — tell Frank which one. Screenshots help.</li>
|
||||
</ul>
|
||||
<p class="foot-note">Anything missing, wrong, or written like a manual instead of a person:
|
||||
that's a bug in this page. Tell Frank, or tell the machine that wrote it.</p>
|
||||
</section>
|
||||
|
||||
</main>
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
<svg fill="#82612C" role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><title>Audiobookshelf</title><path d="M12 0A12 12 0 0 0 0 12a12 12 0 0 0 12 12 12 12 0 0 0 12-12A12 12 0 0 0 12 0Zm-.023.402A11.598 11.598 0 0 1 23.575 12a11.598 11.598 0 0 1-11.598 11.598A11.598 11.598 0 0 1 .378 12 11.598 11.598 0 0 1 11.977.402Zm0 1.776a7.093 7.093 0 0 0-7.092 7.093v1.536a6.395 6.395 0 0 0-.439.33.35.35 0 0 0-.126.27v1.84a.36.36 0 0 0 .126.272c.22.182.722.564 1.504.956v.179c0 .483.31.873.694.873.384 0 .694-.392.694-.873v-4.415c0-.483-.31-.873-.694-.873-.369 0-.67.359-.694.812h-.002v-.91a6.027 6.027 0 1 1 12.054.003v.91c-.025-.454-.326-.813-.695-.813-.384 0-.694.391-.694.873v4.415c0 .483.31.873.694.873.384 0 .695-.392.695-.873v-.179a7.964 7.964 0 0 0 1.503-.956.35.35 0 0 0 .126-.272v-1.843a.342.342 0 0 0-.124-.27 5.932 5.932 0 0 0-.438-.329V9.271a7.093 7.093 0 0 0-7.092-7.093zm-3.34 5.548a.84.84 0 0 0-.84.84v9.405c0 .464.376.84.84.84h.866a.84.84 0 0 0 .84-.84V8.566a.84.84 0 0 0-.84-.84Zm2.905 0a.84.84 0 0 0-.84.84v9.405c0 .464.377.84.84.84h.867a.84.84 0 0 0 .84-.84V8.566a.84.84 0 0 0-.84-.84zm2.908 0a.84.84 0 0 0-.84.84v9.405c0 .464.376.84.84.84h.867a.84.84 0 0 0 .84-.84V8.566a.84.84 0 0 0-.84-.84zM8.112 9.983h1.915v.2H8.112Zm2.906 0h1.915v.2h-1.915Zm2.908 0h1.915v.2h-1.915zm-7.58 9.119a.633.633 0 0 0 0 1.265h11.26a.632.632 0 0 0 0-1.265z"/></svg>
|
||||
|
After Width: | Height: | Size: 1.3 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 4.6 KiB |
|
After Width: | Height: | Size: 4.0 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 87 KiB |
|
After Width: | Height: | Size: 10 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 6.3 KiB |
|
After Width: | Height: | Size: 19 KiB |
|
After Width: | Height: | Size: 3.2 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 71 KiB |
|
After Width: | Height: | Size: 14 KiB |
|
After Width: | Height: | Size: 43 KiB |
@@ -57,8 +57,8 @@
|
||||
{
|
||||
"name": "Calibre-Web",
|
||||
"category": "user",
|
||||
"section": "Media Library Apps",
|
||||
"description": "E-book library. Work in progress: not open to family users yet.",
|
||||
"section": "Work in progress",
|
||||
"description": "E-book library. Not open to family users yet — ask Frank.",
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"probe": "192.168.0.235:8083",
|
||||
@@ -70,8 +70,8 @@
|
||||
{
|
||||
"name": "RomM",
|
||||
"category": "user",
|
||||
"section": "Media Library Apps",
|
||||
"description": "ROM and emulator library. Work in progress: not open to family users yet.",
|
||||
"section": "Work in progress",
|
||||
"description": "ROM and emulator library. Not open to family users yet — ask Frank.",
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"probe": "192.168.0.235:8085",
|
||||
@@ -110,20 +110,20 @@
|
||||
"name": "Jellyfin",
|
||||
"category": "user",
|
||||
"section": "Media Library Apps",
|
||||
"description": "Open source media server. Work in progress: not open to family users yet.",
|
||||
"url": "",
|
||||
"description": "Open source media server for the film and TV library. Sign in with your franksdank.cloud username and password — the same login as the rest of the house; ask Frank if yours isn't in yet.",
|
||||
"url": "http://jelly.franksdank.cloud",
|
||||
"docs": "",
|
||||
"probe": "192.168.0.235:8096",
|
||||
"method": "http",
|
||||
"wip": "down",
|
||||
"showLink": false,
|
||||
"wip": false,
|
||||
"showLink": true,
|
||||
"art": "/img/apps/jellyfin.svg"
|
||||
},
|
||||
{
|
||||
"name": "Seerr",
|
||||
"category": "user",
|
||||
"section": "Media Library Apps",
|
||||
"description": "Download requests. Work in progress: not open to family users yet.",
|
||||
"section": "Work in progress",
|
||||
"description": "Download requests. Not open to family users yet — ask Frank.",
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"probe": "127.0.0.1:5055",
|
||||
@@ -136,7 +136,7 @@
|
||||
"name": "Marreta",
|
||||
"category": "user",
|
||||
"section": "Web Apps",
|
||||
"description": "Clean, ad-free article reader. Paste any URL, get just the article.",
|
||||
"description": "Paywall sledgehammer: hand it an article URL and it will attempt to smash its paywall. Note: does not work on all sites.",
|
||||
"url": "http://marreta.franksdank.cloud",
|
||||
"docs": "",
|
||||
"probe": "127.0.0.1:8082",
|
||||
@@ -314,7 +314,7 @@
|
||||
"zealot-honor.ogg"
|
||||
],
|
||||
"equiv": "Equivalent to last year's frontier level (2025).",
|
||||
"spec": "Ternary-Bonsai-2-9B (PQ2_0 ternary weights, MTP head in the file) on an NVIDIA RTX 3090 — ~167 tokens/s; 262k-token context, three submissions at a time; sees images.",
|
||||
"spec": "Ternary-Bonsai-2-9B (PQ2_0 ternary weights, MTP head in the file) with the uncensoring adapter fitted at load, on an NVIDIA RTX 3090 — 200k-token conversations, three at a time; sees images.",
|
||||
"inputs": [
|
||||
"text",
|
||||
"image"
|
||||
@@ -327,7 +327,7 @@
|
||||
]
|
||||
},
|
||||
"supervision": "Requires tight human supervision.",
|
||||
"warn": "Base-alignment (censored) model: refusals are possible on sensitive topics, and any model can still be steered into risky output. Take this warning seriously — don’t ruin this server for everybody.",
|
||||
"warn": "Uncensored model — a refusal-removal adapter is fitted at load, so refusals are not a safeguard here: output can include adult content, content that describes harmful, hateful, or illegal acts, and other output that may carry severe legal risk. Take this warning seriously — don’t ruin this server for everybody.",
|
||||
"compaction": "If your agent harness lets you set a compaction model, set this one: a compaction job on the bigger models blocks them for far longer."
|
||||
},
|
||||
"url": "",
|
||||
@@ -350,7 +350,7 @@
|
||||
"dragoon-tagline.ogg"
|
||||
],
|
||||
"equiv": "Equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier intelligence).",
|
||||
"spec": "Qwen3.8-Flash-Next v2 native abliterated (halogen flash-server, ~62 GiB) on the Strix Halo iGPU — 200k-token context, two submissions at a time; sees images.",
|
||||
"spec": "Qwen3.8-Flash-Next v2 native abliterated (halogen flash-server, ~62 GiB) on the Strix Halo iGPU — 200k-token conversations, two at a time; sees images.",
|
||||
"inputs": [
|
||||
"text",
|
||||
"image"
|
||||
@@ -378,14 +378,14 @@
|
||||
"section": "AI",
|
||||
"model": {
|
||||
"unit": "archon",
|
||||
"role": "Frank's strongest and fastest model runtime",
|
||||
"role": "Frank's strongest model",
|
||||
"hf": "https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe",
|
||||
"tagline": "...Power Overwhelming...",
|
||||
"sfx": [
|
||||
"archon-power.ogg"
|
||||
],
|
||||
"equiv": "Equivalent to Claude Opus 4.8 / OpenAI-Terra (Summer 2026 frontier intelligence).",
|
||||
"spec": "The engine author's published quant: 4-bit codebook experts + int8 trunk of Qwen3.8-Flash-Next (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
|
||||
"spec": "The engine author's published quant: 4-bit codebook experts + int8 trunk of the official (censored) Qwen3.8-Flash-Next (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token conversations, four at a time; sees images.",
|
||||
"inputs": [
|
||||
"text",
|
||||
"image"
|
||||
@@ -511,6 +511,19 @@
|
||||
"showLink": true,
|
||||
"art": "/img/apps/mono-tt.svg"
|
||||
},
|
||||
{
|
||||
"name": "Audiobookshelf",
|
||||
"category": "user",
|
||||
"section": "Media Library Apps",
|
||||
"description": "Audiobook server for the library: sign in with your Authentik account and every listener keeps their own position.",
|
||||
"url": "http://audiobooks.franksdank.cloud/audiobookshelf/",
|
||||
"docs": "",
|
||||
"probe": "192.168.0.235:8000",
|
||||
"method": "http",
|
||||
"wip": false,
|
||||
"showLink": true,
|
||||
"art": "/img/apps/audiobookshelf.svg"
|
||||
},
|
||||
{
|
||||
"name": "Postgres",
|
||||
"category": "admin",
|
||||
@@ -539,11 +552,20 @@
|
||||
},
|
||||
{
|
||||
"name": "frank-tts",
|
||||
"category": "admin",
|
||||
"category": "user",
|
||||
"section": "AI",
|
||||
"description": "The fleet's cloned-voice speech endpoint: POST /v1/audio/speech with voice=<name>, audio back in that voice. Published on 7772; the engine behind it (frank-tts-engine) is unpublished on purpose. It is also the harness speech role.",
|
||||
"model": {
|
||||
"role": "The fleet's voice — it speaks, it does not chat",
|
||||
"hf": "https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF",
|
||||
"tagline": "Your library, read aloud in a voice you know.",
|
||||
"spec": "Qwen3-TTS 1.7 B base talker (Q4_K_M) with its 12 Hz tokenizer, run by qwentts.cpp on the gfx1151 iGPU. The frank-tts door answers POST /v1/audio/speech with voice=<name> and hands back audio in that voice.",
|
||||
"inputs": ["text"],
|
||||
"use": "Narration: Open Notebook episodes, the operator harness's spoken output, and anything that should sound like a person instead of a text-to-speech robot.",
|
||||
"permission": "Ask Frank for a voice — each pack is built from that person's own recording.",
|
||||
"warn": "Cloned voices. Every pack on this endpoint is a replica of a real person's voice. Audio coming out of it is not that person speaking and must never be passed off as them."
|
||||
},
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"docs": "/docs#models",
|
||||
"probe": "127.0.0.1:7772",
|
||||
"method": "http",
|
||||
"wip": false,
|
||||
|
||||
@@ -183,12 +183,23 @@ body::after {
|
||||
.docs details.key-warn[open] summary::after { content: " \25BE"; }
|
||||
.docs details.key-warn .warn-body { margin-top: 4px; }
|
||||
.docs .foot-note { color: var(--dim); }
|
||||
.docs h3 { color: var(--head); font-size: 1.05rem; margin-top: 16px; }
|
||||
.docs figure { margin: 14px 0; }
|
||||
.docs figcaption { color: var(--dim); font-size: .85rem; }
|
||||
.docs table { border-collapse: collapse; width: 100%; margin: 10px 0; }
|
||||
.docs th, .docs td { border: 1px solid var(--border); padding: 6px 8px; text-align: left; vertical-align: top; }
|
||||
.docs th { color: var(--head); }
|
||||
.docs .toc { display: grid; grid-template-columns: repeat(auto-fill, minmax(150px, 1fr)); gap: 4px; margin: 12px 0; }
|
||||
.docs .toc a { display: block; border: 1px solid var(--border); padding: 6px 8px; text-decoration: none; }
|
||||
.docs .kicker { color: var(--dim); text-transform: uppercase; letter-spacing: .12em; font-size: .72rem; }
|
||||
html[data-bare="1"] .docs h1, html[data-bare="1"] .docs h2 { color: #000; border-color: #000; }
|
||||
html[data-bare="1"] .docs a, html[data-bare="1"] .docs code { color: #000; }
|
||||
html[data-bare="1"] .docs pre { background: none; border-color: #000; }
|
||||
html[data-bare="1"] .docs pre code { color: #000; }
|
||||
html[data-bare="1"] .docs img { border-color: #000; }
|
||||
html[data-bare="1"] .docs .key-warn { color: #000; }
|
||||
html[data-bare="1"] .docs h3, html[data-bare="1"] .docs th { color: #000; }
|
||||
html[data-bare="1"] .docs th, html[data-bare="1"] .docs td, html[data-bare="1"] .docs .toc a { border-color: #000; }
|
||||
|
||||
/* Bare mode (Q27): strip to default browser rendering. */
|
||||
html[data-bare="1"] body { background: #fff; color: #000; font-family: serif; text-shadow: none; font-size: 16px; }
|
||||
@@ -252,6 +263,7 @@ html[data-bare="1"] .top { border-image: none; }
|
||||
.mi-zealot { --mi: #4fb4ff; } /* Zealot: Khai's blue blades */
|
||||
.mi-archon { --mi: #a8ecff; } /* Archon: pure psychic light */
|
||||
.mi-dragoon { --mi: #5fe0d0; } /* Dragoon: phase-crystal teal */
|
||||
.mi-tool { --mi: #ffc86b; } /* frank-tts: amber — a tool, not a unit */
|
||||
.mi-top { display: flex; align-items: center; gap: 10px; }
|
||||
.mi-art {
|
||||
width: 56px;
|
||||
@@ -283,6 +295,7 @@ html[data-bare="1"] .top { border-image: none; }
|
||||
.mi-voice:hover { text-shadow: 0 0 8px var(--mi); }
|
||||
.mi-voice.mi-speaking { animation: mi-pulse 0.7s ease-in-out infinite; }
|
||||
@keyframes mi-pulse { 50% { text-shadow: 0 0 12px var(--mi); } }
|
||||
.mi-tag { color: var(--mi); margin: 4px 0; }
|
||||
.mi-equiv { color: var(--head); font-style: italic; margin: 4px 0; }
|
||||
.mi-spec { color: var(--fg); margin: 4px 0; }
|
||||
.mi-inputs { color: var(--dim); }
|
||||
|
||||
@@ -75,8 +75,10 @@ PY
|
||||
}
|
||||
|
||||
@test "exempt vhosts never route through anubis (auth, plex, own-auth apps)" {
|
||||
# The IdP login flow and git smart-HTTP clients cannot solve challenges;
|
||||
# these vhosts bypass the chain by construction.
|
||||
# The IdP login flow, git smart-HTTP clients, and any non-browser client (the
|
||||
# Audiobookshelf phone app, podcast RSS readers) cannot solve a challenge, so
|
||||
# services that enforce their own complete auth bypass the chain by
|
||||
# construction — the gate would answer their API calls with HTML.
|
||||
local block
|
||||
block=$(awk '/^auth\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||
echo "$block" | grep -q 'reverse_proxy authentik-server:9000'
|
||||
@@ -84,4 +86,11 @@ PY
|
||||
block=$(awk '/^plex\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||
echo "$block" | grep -q '192\.168\.0\.235:32400'
|
||||
! echo "$block" | grep -q anubis
|
||||
block=$(awk '/^audiobooks\.franksdank\.cloud/,/^}/' "$CADDY")
|
||||
echo "$block" | grep -q '192\.168\.0\.235:8000'
|
||||
! echo "$block" | grep -q anubis
|
||||
# The exemption is total: the post-auth listener must not still carry an
|
||||
# audiobooks route. A leftover handle there is a second owner for one host,
|
||||
# and its X-Forwarded-Proto pin would silently disagree with the edge.
|
||||
! grep -q '@audiobooks host audiobooks.franksdank.cloud' "$CADDY"
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: scripts/audiobookshelf-init.sh — the Audiobookshelf bootstrapper.
|
||||
#
|
||||
# A freshly deployed Audiobookshelf answers /healthcheck 200 while it has no
|
||||
# user, no library and no scan, so "healthy" proves nothing about usefulness.
|
||||
# This seam pins what the bootstrapper must make true, against a stub that
|
||||
# speaks the same API as the pinned image (POST /init + POST /login at the
|
||||
# root, bearer-authenticated /api/*, library count only on /items):
|
||||
# 1. an unregistered server gains the account, the library and a scan, and the
|
||||
# credential lands mode 600 with both keys;
|
||||
# 2. a second run creates nothing new (exactly one /init, one library) and
|
||||
# still fires the incremental scan;
|
||||
# 3. neither the bearer token nor the password ever rides a URL — a secret in
|
||||
# a URL or an argv is world-readable in /proc/<pid>/cmdline, which is how
|
||||
# an HF token leaked into a transcript on 2026-09-16;
|
||||
# 4. a server that cannot be reached is refused by URL with exit 1, never
|
||||
# reported as success (a green deploy over a dead API is a lie);
|
||||
# 5. ABS_SCAN outside 0/1 is a usage error raised before anything is touched;
|
||||
# 6. the deployer runs the bootstrapper only after the health gate answered.
|
||||
# 7. the SSO arm sends the base path: ABS builds its redirect_uri as
|
||||
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback, and the
|
||||
# value must be the path the app is mounted under — never the literal
|
||||
# "undefined" a fresh database leaves there.
|
||||
|
||||
load helpers
|
||||
|
||||
setup() {
|
||||
INIT="$REPO_ROOT/scripts/audiobookshelf-init.sh"
|
||||
TMP="$(mktemp -d)"
|
||||
PORT=18937
|
||||
cat > "$TMP/stub.py" <<'PY'
|
||||
import json, sys
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
port, logf = int(sys.argv[1]), sys.argv[2]
|
||||
state = {"init": False, "user": None, "pw": None, "libs": [], "oidc": None}
|
||||
|
||||
|
||||
def log(line):
|
||||
with open(logf, "a") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
def norm(p):
|
||||
# ABS mounts every route under ROUTER_BASE_PATH. The stub accepts the API at the
|
||||
# root or under that prefix, so a test can still assert which one the client used.
|
||||
return p[len("/audiobookshelf"):] if p.startswith("/audiobookshelf") else p
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def send(self, code, body, as_json=False):
|
||||
data = json.dumps(body).encode() if as_json else str(body).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json" if as_json else "text/plain")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def body(self):
|
||||
n = int(self.headers.get("Content-Length") or 0)
|
||||
return json.loads(self.rfile.read(n) or b"{}")
|
||||
|
||||
def authed(self):
|
||||
return self.headers.get("Authorization") == "Bearer stub-token"
|
||||
|
||||
def do_GET(self):
|
||||
path = norm(self.path.split("?")[0])
|
||||
log("GET %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||
if path == "/status":
|
||||
methods = ["local", "openid"] if state["oidc"] else ["local"]
|
||||
return self.send(200, {"app": "audiobookshelf", "isInit": state["init"],
|
||||
"authMethods": methods}, True)
|
||||
if not self.authed():
|
||||
return self.send(401, "Unauthorized")
|
||||
if path == "/api/libraries":
|
||||
return self.send(200, {"libraries": state["libs"]}, True)
|
||||
if path == "/api/auth-settings":
|
||||
return self.send(200, state["oidc"] or {}, True)
|
||||
if path.endswith("/items"):
|
||||
return self.send(200, {"total": 42, "results": []}, True)
|
||||
return self.send(404, "nope")
|
||||
|
||||
def do_POST(self):
|
||||
path = norm(self.path.split("?")[0])
|
||||
payload = self.body()
|
||||
log("POST %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||
if path == "/init":
|
||||
if state["init"]:
|
||||
return self.send(500, "already initialized")
|
||||
root = payload.get("newRoot", {})
|
||||
state["init"] = True
|
||||
state["user"] = root.get("username")
|
||||
state["pw"] = root.get("password")
|
||||
log("INIT %s" % root.get("username"))
|
||||
return self.send(200, "OK")
|
||||
if path == "/login":
|
||||
if not state["init"]:
|
||||
return self.send(401, "Unauthorized")
|
||||
if (payload.get("username"), payload.get("password")) != (state["user"], state["pw"]):
|
||||
return self.send(401, "Unauthorized")
|
||||
return self.send(200, {"user": {"username": state["user"], "type": "root",
|
||||
"id": "u1", "token": "stub-token"}}, True)
|
||||
if not self.authed():
|
||||
return self.send(401, "Unauthorized")
|
||||
if path == "/api/libraries":
|
||||
lib = {"id": "lib-1", "name": payload.get("name"), "folders": payload.get("folders", [])}
|
||||
state["libs"].append(lib)
|
||||
log("LIBRARY %s %s" % (lib["name"], lib["folders"]))
|
||||
return self.send(200, lib, True)
|
||||
if path.endswith("/scan"):
|
||||
log("SCAN")
|
||||
return self.send(200, "OK")
|
||||
return self.send(404, "nope")
|
||||
|
||||
def do_PATCH(self):
|
||||
path = norm(self.path.split("?")[0])
|
||||
payload = self.body()
|
||||
log("PATCH %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||
if not self.authed():
|
||||
return self.send(401, "Unauthorized")
|
||||
if path == "/api/auth-settings":
|
||||
# Stored and echoed verbatim: the test asserts the wire, not the stub's
|
||||
# opinion about which keys are patchable.
|
||||
state["oidc"] = payload
|
||||
for k in sorted(payload):
|
||||
log("OIDCFG %s=%r" % (k, payload[k]))
|
||||
return self.send(200, payload, True)
|
||||
return self.send(404, "nope")
|
||||
|
||||
|
||||
ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()
|
||||
PY
|
||||
python3 "$TMP/stub.py" "$PORT" "$TMP/requests.log" > "$TMP/stub.out" 2>&1 &
|
||||
STUB_PID=$!
|
||||
for _ in $(seq 1 50); do
|
||||
curl -s -o /dev/null "http://127.0.0.1:$PORT/status" && break
|
||||
sleep 0.1
|
||||
done
|
||||
}
|
||||
|
||||
teardown() {
|
||||
[ -n "${STUB_PID:-}" ] && kill "$STUB_PID" 2>/dev/null || true
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
|
||||
boot() {
|
||||
env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" \
|
||||
ABS_ADMIN_USER=frank ABS_LIBRARY=Audiobooks ABS_FOLDER=/audiobooks \
|
||||
ABS_SCAN="${SCAN:-1}" bash "$INIT"
|
||||
}
|
||||
|
||||
@test "an unregistered server gains the account, the library and the scan" {
|
||||
run boot
|
||||
[ "$status" -eq 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qx "credential: created $TMP/cred"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^account: created (frank)"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^token: obtained (root)"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^library: created Audiobooks (lib-1)"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^items: 42"
|
||||
# The state roots of this service hold an account credential, so the file
|
||||
# that carries it must never be group- or world-readable.
|
||||
[ "$(stat -c %a "$TMP/cred")" = "600" ]
|
||||
grep -qx 'ABS_ADMIN_USER=frank' "$TMP/cred"
|
||||
grep -Eq '^ABS_ADMIN_PASSWORD=[0-9a-f]{24}$' "$TMP/cred"
|
||||
# The folder the library indexes is the bind inside the container, not the
|
||||
# host path — a library pointed at /mnt/entropy would answer 0 items and
|
||||
# look like an empty audiobook collection.
|
||||
grep -q "LIBRARY Audiobooks \[{'fullPath': '/audiobooks'}\]" "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "a second run creates nothing new and still fires the incremental scan" {
|
||||
boot > /dev/null
|
||||
[ -e "$TMP/requests.log" ]
|
||||
run boot
|
||||
[ "$status" -eq 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^account: already registered"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^library: exists Audiobooks (lib-1)"
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^scan: accepted"
|
||||
# No second credential, no second account, no duplicate library: the deployer
|
||||
# may run this on every re-deploy of the service.
|
||||
! printf '%s\n' "${lines[@]}" | grep -q "credential: created"
|
||||
[ "$(grep -c '^INIT ' "$TMP/requests.log")" -eq 1 ]
|
||||
[ "$(grep -c '^LIBRARY ' "$TMP/requests.log")" -eq 1 ]
|
||||
[ "$(grep -c '^SCAN$' "$TMP/requests.log")" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "the bearer token and the password never ride a URL" {
|
||||
boot
|
||||
# The stub logs the request line (method + path). A secret showing up there
|
||||
# means it travelled as a query parameter — visible in access logs, shell
|
||||
# history and /proc/<pid>/cmdline.
|
||||
! awk '{print $2}' "$TMP/requests.log" | grep -q 'stub-token'
|
||||
pw=$(awk -F= '/^ABS_ADMIN_PASSWORD=/{print $2}' "$TMP/cred")
|
||||
[ -n "$pw" ]
|
||||
! awk '{print $2}' "$TMP/requests.log" | grep -q -- "$pw"
|
||||
# and the authenticated routes only answered once the header was present
|
||||
grep -q '^GET /api/libraries auth=yes$' "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "a server that does not answer is refused by URL, not reported as success" {
|
||||
run env ABS_URL="http://127.0.0.1:1" ABS_SECRET="$TMP/cred" bash "$INIT"
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *refused:* ]]
|
||||
[[ "$output" == *"127.0.0.1:1"* ]]
|
||||
}
|
||||
|
||||
@test "ABS_SCAN outside 0/1 is a usage error raised before anything is touched" {
|
||||
run env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" ABS_SCAN=2 \
|
||||
bash "$INIT"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *ABS_SCAN* ]]
|
||||
[ ! -e "$TMP/cred" ]
|
||||
}
|
||||
|
||||
@test "the deployer bootstraps only after the health gate has answered" {
|
||||
body=$(awk '/^deploy_audiobookshelf\(\)/,/^}/' "$REPO_ROOT/scripts/deploy-remote.sh")
|
||||
[ -n "$body" ]
|
||||
printf '%s\n' "$body" | grep -q \
|
||||
'wait_for_health audiobookshelf http://127\.0\.0\.1:8000/healthcheck'
|
||||
printf '%s\n' "$body" | grep -q 'scripts/audiobookshelf-init\.sh'
|
||||
printf '%s\n' "$body" | awk '
|
||||
/wait_for_health audiobookshelf/ { health = NR }
|
||||
/audiobookshelf-init\.sh/ { init = NR }
|
||||
END { exit !(health && init && init > health) }'
|
||||
}
|
||||
|
||||
@test "the SSO arm sends the base path as the redirect subfolder" {
|
||||
# What authentik has registered must equal what ABS builds:
|
||||
# <origin><authOpenIDSubfolderForRedirectURLs>/auth/openid/callback. A database
|
||||
# created before SSO leaves that key unset and ABS then sends the literal
|
||||
# "/undefined/auth/openid/callback" — measured 2026-11-02, where the live login died
|
||||
# on authentik's "Redirect URI Error" page. So the value has to reach the wire, and
|
||||
# it has to be the path the app is served under.
|
||||
umask 077
|
||||
cat > "$TMP/cred" <<'CRED'
|
||||
ABS_ADMIN_USER=frank
|
||||
ABS_ADMIN_PASSWORD=0123456789abcdef0123456789
|
||||
ABS_OIDC_CLIENT_SECRET=0123456789abcdef0123456789abcdef0123456789abcdef0123456789
|
||||
CRED
|
||||
run env ABS_URL="http://127.0.0.1:$PORT/audiobookshelf" ABS_SECRET="$TMP/cred" \
|
||||
ABS_SCAN=0 bash "$INIT"
|
||||
[ "$status" -eq 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -q "^oidc: armed"
|
||||
grep -q "^OIDCFG authOpenIDSubfolderForRedirectURLs='/audiobookshelf'$" "$TMP/requests.log"
|
||||
# every route the bootstrapper calls rides under the same prefix it armed with
|
||||
grep -q "^POST /audiobookshelf/init auth=no$" "$TMP/requests.log"
|
||||
grep -q "^GET /audiobookshelf/api/auth-settings auth=yes$" "$TMP/requests.log"
|
||||
|
||||
# A root-mounted deployment sends the EMPTY subfolder (the value the 2.17.4 upgrade
|
||||
# uses), not whatever the last URL happened to carry.
|
||||
run env ABS_URL="http://127.0.0.1:$PORT" ABS_SECRET="$TMP/cred" ABS_SCAN=0 bash "$INIT"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(grep "^OIDCFG authOpenIDSubfolderForRedirectURLs=" "$TMP/requests.log" | tail -1)" \
|
||||
= "OIDCFG authOpenIDSubfolderForRedirectURLs=''" ]
|
||||
}
|
||||
|
||||
@test "the app and the IdP provider script build the same redirect path" {
|
||||
# The provider row is written by scripts/authentik-audiobookshelf-sso.sh from
|
||||
# ABS_BASE; the app builds its redirect_uri from the base path the deployer hands
|
||||
# audiobookshelf-init.sh, which comes from the unit. Drift between the two is a
|
||||
# redirect_uri mismatch — the login dies at the IdP before any credential is issued.
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
mounted=$(awk -F= '/^Environment=ROUTER_BASE_PATH=/{print $3}' "$unit")
|
||||
[ -n "$mounted" ]
|
||||
provider=$(grep -m1 '^BASE=' "$REPO_ROOT/scripts/authentik-audiobookshelf-sso.sh" \
|
||||
| grep -oE 'https?://[^"}]+')
|
||||
bootstrapper=$(grep -m1 '^ABS_URL=' "$INIT" | grep -oE 'https?://[^"}]+')
|
||||
# Origins legitimately differ (LAN address vs loopback); the PATH must not.
|
||||
[ "${provider#http://192.168.0.235:8000}/auth/openid/callback" \
|
||||
= "${bootstrapper#http://127.0.0.1:8000}/auth/openid/callback" ]
|
||||
[ "${bootstrapper#http://127.0.0.1:8000}" = "$mounted" ]
|
||||
}
|
||||
@@ -127,3 +127,68 @@ load helpers
|
||||
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
|
||||
done
|
||||
}
|
||||
|
||||
@test "audiobookshelf unit exists, is boot-persistent and self-healing" {
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
[ -f "$unit" ] || return 1
|
||||
grep -q '^WantedBy=default.target$' "$unit" || return 1
|
||||
grep -q '^Restart=always$' "$unit" || return 1
|
||||
grep -q '^RestartSec=' "$unit" || return 1
|
||||
}
|
||||
|
||||
@test "audiobookshelf keeps every byte it writes on the SSD" {
|
||||
# The point of the unit: its library database, accounts, progress and cached
|
||||
# covers are all server state, and server state belongs on the NVMe. A bind
|
||||
# of /mnt/entropy that is not :ro would silently put writes on the spindle.
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
grep -q '^Volume=/home/dank/Containers/audiobookshelf/config:/config$' "$unit" || return 1
|
||||
grep -q '^Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata$' "$unit" || return 1
|
||||
grep -q '^Volume=/mnt/entropy/Audiobooks:/audiobooks:ro$' "$unit" || return 1
|
||||
# The overlay view is reserved for the two media servers that write sidecars
|
||||
# (tests/media_overlay.bats pins that set); a reader must not join it.
|
||||
! grep -q 'media-meta/view' "$unit" || return 1
|
||||
}
|
||||
|
||||
@test "audiobookshelf refuses to start against an unmounted array" {
|
||||
# Without the gate podman invents an empty bind source and the first scan
|
||||
# records an empty library as truth.
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
grep -q '^ExecStartPre=.*/wait-for-mount.sh /mnt/entropy$' "$unit" || return 1
|
||||
}
|
||||
|
||||
@test "audiobookshelf health check names a binary its image actually ships" {
|
||||
# The image is Alpine: wget yes, curl no. A curl HealthCmd reads the
|
||||
# container as unhealthy while it is serving perfectly.
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
grep -q '^HealthCmd=wget ' "$unit" || return 1
|
||||
! grep -q '^HealthCmd=curl' "$unit" || return 1
|
||||
grep -q '^PublishPort=8000:8000$' "$unit" || return 1
|
||||
}
|
||||
|
||||
@test "the audiobookshelf deployer creates both state roots before it starts" {
|
||||
body="$(sed -n '/^deploy_audiobookshelf()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||
mkdir_line="$(printf '%s\n' "$body" | grep -n 'mkdir -p' | head -1 | cut -d: -f1)"
|
||||
start_line="$(printf '%s\n' "$body" | grep -n 'systemctl --user restart' | head -1 | cut -d: -f1)"
|
||||
[ -n "$mkdir_line" ] && [ -n "$start_line" ] || return 1
|
||||
[ "$mkdir_line" -lt "$start_line" ] || return 1
|
||||
printf '%s\n' "$body" | grep -q 'audiobookshelf/config' || return 1
|
||||
printf '%s\n' "$body" | grep -q 'audiobookshelf/metadata' || return 1
|
||||
printf '%s\n' "$body" | grep -q 'wait_for_health audiobookshelf' || return 1
|
||||
}
|
||||
|
||||
@test "audiobookshelf sets its port and base path with the names the image reads" {
|
||||
# Measured 2026-11-02 on the first deploy: ABS_PORT / ABS_CONFIG_DIR are the npm
|
||||
# package's names and this image ignores them, so the server came up on :80
|
||||
# under base path /audiobookshelf and the deploy gate timed out beside a unit
|
||||
# that was active (running). PORT and ROUTER_BASE_PATH are what the image's own
|
||||
# ENV block and the server source actually use.
|
||||
# The base path is not a free choice either: /app/client/dist/index.html inside the
|
||||
# pinned image carries routerBasePath:"/audiobookshelf", so a server mounted at / hands
|
||||
# out a page whose every script request falls through to the SPA fallback and comes back
|
||||
# text/html — the UI never leaves "Loading…" while the API stays healthy (measured
|
||||
# 2026-11-02, which is exactly how it hid behind an API-only verification).
|
||||
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||
grep -q '^Environment=PORT=8000$' "$unit" || return 1
|
||||
grep -q '^Environment=ROUTER_BASE_PATH=/audiobookshelf$' "$unit" || return 1
|
||||
! grep -qE '^Environment=ABS_' "$unit" || return 1
|
||||
}
|
||||
@@ -28,15 +28,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the retired ik chat template plans nothing, and its archive is still here" {
|
||||
@test "the retired ik chat template plans nothing" {
|
||||
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
||||
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
||||
# llama.cpp frank-zealot runtime serves the template the GGUF itself
|
||||
# carries, which is the one with reasoning_effort, preserve_thinking and
|
||||
# the tool-call grammar. Editing the archive must deploy nothing -- and the
|
||||
# archive must still be present, so this cannot pass by the file having
|
||||
# vanished.
|
||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp" ]
|
||||
# the tool-call grammar. The planner's rule is a PATH rule, so an edit at
|
||||
# that path deploys nothing no matter where the bytes currently sit: the seam
|
||||
# deliberately does not assert the archive exists. Trash is a purgeable hold
|
||||
# (untracked in #111, and the deletion policy says the purger deletes it), so
|
||||
# pinning it fails every fresh checkout and proves nothing about planning.
|
||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
|
||||
@@ -35,14 +35,20 @@ export default defineConfig({
|
||||
// 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
|
||||
// spec until now: the 17-hermes "uncovered vhost broke silently" class).
|
||||
// 22- joins Prometheus's app-is-really-there proof (its own API), closing
|
||||
// the "not a login wall but maybe a 404" hole 20- closed for Grafana.
|
||||
testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
|
||||
// the "not a login wall but maybe a 404" hole 20- closed for Grafana. 24-
|
||||
// joins 2026-11-03 (Audiobookshelf's DEFAULT listener grant) and belongs to
|
||||
// this project precisely because the standard identity is in no Audiobookshelf
|
||||
// group — the admin project could not observe the default.
|
||||
testMatch: /(10|12|13|16|17|18|20|21|22|24)-.*\.spec\.ts/,
|
||||
use: { storageState: ".auth/qa-e2e.json" },
|
||||
dependencies: ["setup"],
|
||||
},
|
||||
{
|
||||
name: "admin",
|
||||
testMatch: /11-.*\.spec\.ts/,
|
||||
// 23- joins 2026-11-02 (Audiobookshelf SSO): it needs BOTH halves of the admin
|
||||
// fixture — a live authentik session to ride the OIDC round trip, and the
|
||||
// qa-e2e-admin group membership that grants site admin in the app.
|
||||
testMatch: /(11|23)-.*\.spec\.ts/,
|
||||
use: { storageState: ".auth/qa-e2e-admin.json" },
|
||||
dependencies: ["setup"],
|
||||
},
|
||||
|
||||
@@ -6,9 +6,10 @@ import { expect, test } from "@playwright/test";
|
||||
// navigation would skip. These tests drive the browser as an UNAUTHENTICATED
|
||||
// user (the guest project has no storage state) and navigate directly to each
|
||||
// guarded vhost's root path: the guard must present authentik's flow UI in
|
||||
// place and must never leak app content. plex is the documented exemption —
|
||||
// PMS enforces its own auth — so it must be reachable unauthenticated with
|
||||
// zero challenge pages, proving the exemption is deliberate, not accidental.
|
||||
// place and must never leak app content. plex and audiobooks are the documented
|
||||
// exemptions — PMS and Audiobookshelf enforce their own auth — so they must be
|
||||
// reachable unauthenticated with zero challenge pages, proving the exemption is
|
||||
// deliberate, not accidental.
|
||||
|
||||
const BASE = "https://www.franksdank.cloud";
|
||||
|
||||
@@ -17,7 +18,9 @@ const BASE = "https://www.franksdank.cloud";
|
||||
// added here or the guard-chain audit is incomplete.
|
||||
const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
|
||||
|
||||
// Markers that prove app content leaked past the guard to an anonymous user.
|
||||
// Markers that prove app content leaked past the guard to an anonymous user. Each
|
||||
// marker must be a string the APP renders that the gate itself could never print:
|
||||
// authentik's gate never renders an app's UI text, because it IS the IdP.
|
||||
const LEAK_MARKERS: Record<string, string[]> = {
|
||||
"search.franksdank.cloud": ["SearXNG preferences"],
|
||||
"romm.franksdank.cloud": ["RomM"],
|
||||
@@ -98,3 +101,54 @@ test("plex is exempt from the guard chain and serves PMS content unauthenticated
|
||||
const resp = await page.request.get("https://plex.franksdank.cloud/");
|
||||
expect(resp.status(), "PMS must answer directly, not error").toBeLessThan(500);
|
||||
});
|
||||
|
||||
// Audiobookshelf is exempt by construction (2026-11-03). The phone app is a
|
||||
// non-browser client and solves no JS proof-of-work, so behind the guard chain it
|
||||
// was answered with challenge HTML wherever the socket `init` payload belongs —
|
||||
// every remote sign-in died as a TypeError on a non-object, and the container's
|
||||
// request log never saw a byte of it. ABS enforces its own auth (local accounts
|
||||
// plus authentik OIDC), exactly like PMS above. Asserted from the CLIENT side: the
|
||||
// endpoints an app must call before it holds any credential are answered by the
|
||||
// app, and the app's own 401 — not a 200 of HTML — is what an anonymous API call gets.
|
||||
test("audiobookshelf answers an unauthenticated app client with app bytes, not gate bytes", async ({
|
||||
request,
|
||||
}) => {
|
||||
const ABS = "https://audiobooks.franksdank.cloud";
|
||||
|
||||
// The probe every Audiobookshelf client sends first. The app answers it at the
|
||||
// ROOT whatever ROUTER_BASE_PATH says, so this is the vhost's own behaviour.
|
||||
const ping = await request.get(`${ABS}/ping`, { timeout: 30_000 });
|
||||
expect(ping.status(), "/ping must be answered by the app").toBe(200);
|
||||
expect(await ping.json(), "/ping must carry the app's own payload").toEqual({ success: true });
|
||||
|
||||
// The socket.io handshake. Engine.IO answers this in a machine format; the gate
|
||||
// answered it in HTML, which is the defect this exemption exists to kill.
|
||||
const sock = await request.get(`${ABS}/audiobookshelf/socket.io/?EIO=4&transport=polling`, {
|
||||
timeout: 30_000,
|
||||
});
|
||||
expect(
|
||||
await sock.text(),
|
||||
"the socket handshake was answered by the gate, not Engine.IO",
|
||||
).not.toContain("Making sure you're not a bot");
|
||||
expect(
|
||||
sock.headers()["content-type"] ?? "",
|
||||
"the socket handshake must not arrive as HTML",
|
||||
).not.toMatch(/text\/html/);
|
||||
|
||||
// An API call with no credential: the app refuses it. A gate would answer 200
|
||||
// with HTML, so the status code alone distinguishes the two owners.
|
||||
const api = await request.get(`${ABS}/audiobookshelf/api/libraries`, { timeout: 30_000 });
|
||||
expect(api.status(), "the app's own auth gate must own unauthenticated API calls").toBe(401);
|
||||
});
|
||||
|
||||
test("audiobookshelf login renders the app's own sign-in to an anonymous browser", async ({ page }) => {
|
||||
test.setTimeout(120_000);
|
||||
await page.goto("https://audiobooks.franksdank.cloud/audiobookshelf/login");
|
||||
expect(new URL(page.url()).hostname).toBe("audiobooks.franksdank.cloud");
|
||||
|
||||
const body = await page.locator("body").innerText();
|
||||
expect(body, "the bot challenge still fronts this vhost").not.toContain("Making sure you're not a bot");
|
||||
expect(await page.locator('input[name="uidField"]').count(), "the IdP gate form must not front the app").toBe(0);
|
||||
// The app answers, and its login screen carries SSO beside the local form.
|
||||
await expect(page.getByText("Sign in with Authentik")).toBeVisible({ timeout: 90_000 });
|
||||
});
|
||||
@@ -0,0 +1,95 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
// Audiobookshelf: the SSO round trip must land a session with a real permission level.
|
||||
//
|
||||
// The service answers on a LAN address with no DNS name and no Caddy vhost, so it is in no
|
||||
// guarded-vhost list and no other spec ever fetched a byte of it — every deploy gate passed
|
||||
// while its login was dead. Two defect classes, both caught live on 2026-11-02:
|
||||
//
|
||||
// 1. The bundle must be BUILT for /audiobookshelf (ROUTER_BASE_PATH in the quadlet). Built
|
||||
// for "/" the container still answers /healthcheck and still serves index.html, so the
|
||||
// deploy health gate is green while the browser sees a blank page: every <script>
|
||||
// resolves against the root, misses the static allowlist, and falls through to the SPA
|
||||
// catch-all that returns index.html as text/html. Hence the MIME assertion below — the
|
||||
// one thing that distinguishes a booted SPA from a served HTML stub.
|
||||
// 2. The authentik provider row must carry grant_types AND the built-in OpenID scope
|
||||
// mappings (provisioned by scripts/authentik-audiobookshelf-sso.sh). A blueprint
|
||||
// re-import reverts hand-made provider rows silently, and the wire errors are generic:
|
||||
// oauthlib's "The request is otherwise malformed" for an empty grant list, then
|
||||
// "insufficient_scope" from ABS's /userinfo call for a short scope. Only the level
|
||||
// assertion below — that the abs_groups claim granted site admin — sees either.
|
||||
const HOST = "192.168.0.235:8000";
|
||||
const ABS = `http://${HOST}/audiobookshelf`;
|
||||
|
||||
test("SSO round trip boots the SPA and grants a real Audiobookshelf session", async ({
|
||||
page,
|
||||
}) => {
|
||||
test.setTimeout(180_000);
|
||||
|
||||
const scripts: { url: string; type: string; status: number }[] = [];
|
||||
page.on("response", (res) => {
|
||||
const u = new URL(res.url());
|
||||
if (u.host === HOST && u.pathname.endsWith(".js")) {
|
||||
scripts.push({ url: u.pathname, type: res.headers()["content-type"] ?? "", status: res.status() });
|
||||
}
|
||||
});
|
||||
|
||||
await page.goto(`${ABS}/login`, { waitUntil: "domcontentloaded" });
|
||||
const sso = page.getByText("Sign in with Authentik");
|
||||
await expect(sso).toBeVisible();
|
||||
await sso.click();
|
||||
|
||||
// The admin project's storageState carries a live authentik session, so this is a silent
|
||||
// round trip: authorize -> code -> callback -> token. Still on /login means the IdP
|
||||
// refused the authorize request or ABS rejected the code exchange.
|
||||
await expect
|
||||
.poll(() => !new URL(page.url()).pathname.endsWith("/login"), { timeout: 90_000 })
|
||||
.toBe(true);
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const u = new URL(page.url());
|
||||
return u.host === HOST && u.pathname.startsWith("/audiobookshelf");
|
||||
},
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
.toBe(true);
|
||||
|
||||
// Defect 1 gate: nothing the app loaded may be HTML dressed as a script. The render proof
|
||||
// is the SSO button above — it only exists once the bundle has executed, so no DOM-shape
|
||||
// guess is needed (the shell's own mount markup says nothing about whether it booted).
|
||||
// Only responses that carried a body can be judged: the OIDC bounce re-validates the entry
|
||||
// script (Cache-Control: max-age=0 + ETag) and a 304 carries no Content-Type by spec.
|
||||
const bodies = scripts.filter((s) => s.status === 200);
|
||||
expect(bodies.length, "the SPA loaded no JavaScript").toBeGreaterThan(2);
|
||||
for (const s of bodies) {
|
||||
expect(s.type, `${s.url} arrived as ${s.type}`).toContain("javascript");
|
||||
}
|
||||
|
||||
// The token the app landed on is the credential its own API must accept. Give the SPA a
|
||||
// moment to finish the exchange: the URL leaves /login the instant authentik bounces back
|
||||
// through the callback route, which is BEFORE the app has stored what it exchanged.
|
||||
const grabToken = () =>
|
||||
page.evaluate(() => {
|
||||
for (let i = 0; i < localStorage.length; i++) {
|
||||
const v = localStorage.getItem(localStorage.key(i) as string) || "";
|
||||
if (/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(v)) return v;
|
||||
}
|
||||
const c = document.cookie.match(/(?:^|;\s*)abs_token=([^;]+)/);
|
||||
return c ? decodeURIComponent(c[1]) : "";
|
||||
});
|
||||
await expect.poll(grabToken, { timeout: 60_000 }).not.toBe("");
|
||||
const token = await grabToken();
|
||||
|
||||
const me = await page.request.get(`${ABS}/api/me`, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
});
|
||||
expect(me.status()).toBe(200);
|
||||
const user = await me.json();
|
||||
expect(user.username).toBe("qa-e2e-admin");
|
||||
// Site role, derived ONLY from the abs_groups claim: it says "admin" if and only if the
|
||||
// scope mapping reached ABS with audiobookshelf_admins in the list. A short token scope
|
||||
// (missing OpenID mappings) never gets this far — the exchange dies before it.
|
||||
expect(user.type).toBe("admin");
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
// Audiobookshelf grants the LISTENER role to anyone the IdP lets sign in (2026-11-03).
|
||||
//
|
||||
// The role used to come from membership of `audiobookshelf_users` — a group nobody was ever
|
||||
// added to. The household accounts exist in authentik, and every one of them was denied at ABS
|
||||
// AFTER a successful authentication, which reads as a broken login rather than a policy decision.
|
||||
// The scope mapping now answers `abs_groups: ["user"]` for every signer and reserves `admin` for
|
||||
// `audiobookshelf_admins`, so the IdP is the only gate to the library and onboarding a listener
|
||||
// is exactly "create the account".
|
||||
//
|
||||
// This is the counterpart to 23-: that spec proves the admin escalation, this one proves the
|
||||
// DEFAULT grant, from an identity in neither group. Against the old mapping it is red — ABS
|
||||
// refuses the sign-in and the SPA never leaves /login.
|
||||
const HOST = "192.168.0.235:8000";
|
||||
const ABS = `http://${HOST}/audiobookshelf`;
|
||||
|
||||
test("an SSO signer in no Audiobookshelf group lands as a listener", async ({ page }) => {
|
||||
test.setTimeout(180_000);
|
||||
|
||||
await page.goto(`${ABS}/login`, { waitUntil: "domcontentloaded" });
|
||||
const sso = page.getByText("Sign in with Authentik");
|
||||
await expect(sso).toBeVisible();
|
||||
await sso.click();
|
||||
|
||||
// The standard project's storageState carries a live authentik session, so the round trip is
|
||||
// silent: authorize -> code -> callback -> token. Still being on /login means the app refused
|
||||
// the identity (the denial this change removes) or the exchange failed.
|
||||
await expect
|
||||
.poll(() => !new URL(page.url()).pathname.endsWith("/login"), { timeout: 90_000 })
|
||||
.toBe(true);
|
||||
|
||||
const grabToken = () =>
|
||||
page.evaluate(() => {
|
||||
for (let i = 0; i < localStorage.length; i++) {
|
||||
const v = localStorage.getItem(localStorage.key(i) as string) || "";
|
||||
if (/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(v)) return v;
|
||||
}
|
||||
const c = document.cookie.match(/(?:^|;\s*)abs_token=([^;]+)/);
|
||||
return c ? decodeURIComponent(c[1]) : "";
|
||||
});
|
||||
await expect.poll(grabToken, { timeout: 60_000 }).not.toBe("");
|
||||
|
||||
const me = await page.request.get(`${ABS}/api/me`, {
|
||||
headers: { Authorization: `Bearer ${await grabToken()}` },
|
||||
});
|
||||
expect(me.status()).toBe(200);
|
||||
const user = await me.json();
|
||||
expect(user.username).toBe("qa-e2e");
|
||||
// The default, not a grant: the listener role for a signer in neither group. Being `user`
|
||||
// rather than `admin` is also the proof that the default never escalates.
|
||||
expect(user.type).toBe("user");
|
||||
});
|
||||
@@ -981,9 +981,12 @@ DRV
|
||||
# there is no address the checker could dial, and a card that can never be
|
||||
# green is worse than no card. The façade carries the pair's signal — and
|
||||
# since the 2026-10-18 deploy that signal is a real probe (`wip: false`), not
|
||||
# a placeholder. It stays `showLink: false` and admin-only: an HTTP API has no
|
||||
# page to open, and the checker counts ANY HTTP response as up, so the card is
|
||||
# honest without a link and without a `/health` path in the probe.
|
||||
# a placeholder. It stays `showLink: false`: an HTTP API has no page to open,
|
||||
# and the checker counts ANY HTTP response as up, so the card is honest
|
||||
# without a link and without a `/health` path in the probe.
|
||||
# 2026-11-04: the card moved from admin to the user page's Model Information
|
||||
# block — the household guide already documents the voices (/docs#models), so
|
||||
# the family reading about them should also see whether the door is up.
|
||||
python3 - "$REPO_ROOT" <<'PY'
|
||||
import json, os, sys
|
||||
root = sys.argv[1]
|
||||
@@ -996,8 +999,10 @@ assert fts["wip"] is False, \
|
||||
"the endpoint is deployed; a wip card would hide its real state: " + repr(fts.get("wip"))
|
||||
assert fts["showLink"] is False and fts["url"] == "", \
|
||||
"an API endpoint has no UI to link to — it is a status card, not a tile"
|
||||
assert fts["category"] == "admin", \
|
||||
"consumers dial this; it is not an app the user homepage should advertise"
|
||||
assert fts["category"] == "user" and fts["section"] == "AI", \
|
||||
"the voice endpoint belongs to the user page's Model Information block: " + repr(fts["category"])
|
||||
assert isinstance(fts.get("model"), dict) and fts["model"].get("role"), \
|
||||
"the card renders as a Model Information card, which needs a model object"
|
||||
assert os.path.exists(f"{root}/shared/homepage{fts['art']}"), f"missing art {fts['art']}"
|
||||
PY
|
||||
}
|
||||
|
||||
@@ -37,7 +37,7 @@ PY
|
||||
import json, sys
|
||||
data = json.load(open(sys.argv[1]))
|
||||
allowed = {
|
||||
"user": {"AI", "Web Apps", "Media Library Apps"},
|
||||
"user": {"AI", "Web Apps", "Media Library Apps", "Work in progress"},
|
||||
"admin": {"Media Library Apps", "Observability", "AI", "Infrastructure"},
|
||||
}
|
||||
for svc in data:
|
||||
@@ -76,23 +76,33 @@ PY
|
||||
|
||||
@test "the Model Information component is complete: text, art, voices, warnings" {
|
||||
# The fused model panel is the family's only description of the fleet.
|
||||
# Every model card must carry: equivalence note, supervision level, the
|
||||
# unit art, at least one playable voice line, and (for uncensored models)
|
||||
# the legal warning. The compaction recommendation must name the zealot.
|
||||
# Every card must carry: role, spec, use, an inputs list, and art that
|
||||
# exists. A card with a StarCraft unit identity (a chat model) must also
|
||||
# carry the equivalence note, the supervision level, and a playable unit
|
||||
# voice line whose file ships; a unit-less tool card (frank-tts speaks, it
|
||||
# does not chat) quotes no unit, so it carries the ask-Frank permission
|
||||
# note instead. Uncensored models must carry the legal warning. The
|
||||
# compaction recommendation must name the zealot.
|
||||
python3 - "$REPO_ROOT/shared/homepage/services.json" "$REPO_ROOT/shared/homepage" <<'PY'
|
||||
import json, os, sys
|
||||
services = json.load(open(sys.argv[1]))
|
||||
root = sys.argv[2]
|
||||
fleet = [s for s in services if "model" in s]
|
||||
assert {s["name"] for s in fleet} == {"frank-dragoon", "frank-archon", "frank-zealot"}, [s["name"] for s in fleet]
|
||||
assert {s["name"] for s in fleet} == {"frank-dragoon", "frank-archon", "frank-zealot", "frank-tts"}, [s["name"] for s in fleet]
|
||||
for s in fleet:
|
||||
m = s["model"]
|
||||
assert m["equiv"].strip(), f"{s['name']} has no equivalence note"
|
||||
assert m["supervision"].strip(), f"{s['name']} has no supervision note"
|
||||
assert m["sfx"], f"{s['name']} has no voice line"
|
||||
assert os.path.isfile(os.path.join(root, "img", "unit-" + m["unit"] + ".jpg")), f"{s['name']} art missing"
|
||||
for f in m["sfx"] + m.get("voice", {}).get("sfx", []):
|
||||
assert m["role"].strip() and m["spec"].strip() and m["use"].strip(), f"{s['name']} needs role, spec and use"
|
||||
assert m.get("inputs"), f"{s['name']} has no inputs list"
|
||||
art = s.get("art") or "img/unit-%s.jpg" % m["unit"]
|
||||
assert os.path.isfile(os.path.join(root, art.lstrip("/"))), f"{s['name']} art missing: {art}"
|
||||
for f in list(m.get("sfx", [])) + m.get("voice", {}).get("sfx", []):
|
||||
assert os.path.isfile(os.path.join(root, "sfx", f)), f"{s['name']} voice {f} missing"
|
||||
if m.get("unit"):
|
||||
assert m["equiv"].strip(), f"{s['name']} has no equivalence note"
|
||||
assert m["supervision"].strip(), f"{s['name']} has no supervision note"
|
||||
assert m.get("sfx"), f"{s['name']} has no voice line"
|
||||
else:
|
||||
assert m.get("permission", "").strip(), f"{s['name']} has no permission note"
|
||||
zealot = next(s for s in fleet if s["name"] == "frank-zealot")
|
||||
assert "compaction" in zealot["model"].get("compaction", ""), "zealot must be recommended as compaction model"
|
||||
assert zealot["model"].get("compaction", "") != "", "compaction advice must be its own visible field"
|
||||
@@ -102,13 +112,11 @@ for s in fleet:
|
||||
uncensored = "ncensored" in m["spec"] or "bliterated" in m["spec"]
|
||||
if uncensored:
|
||||
assert "legal" in m.get("warn", "").lower(), f"{s['name']} is uncensored but has no legal warning"
|
||||
assert [s["name"] for s in fleet] == ["frank-zealot", "frank-dragoon", "frank-archon"], \
|
||||
"card render order must be zealot, dragoon, archon: " + str([s["name"] for s in fleet])
|
||||
assert [s["name"] for s in fleet] == ["frank-zealot", "frank-dragoon", "frank-archon", "frank-tts"], \
|
||||
"card render order must be zealot, dragoon, archon, then the voice tool: " + str([s["name"] for s in fleet])
|
||||
w = {s["name"]: s["model"].get("warn", "") for s in fleet}
|
||||
assert "legal risk" in w["frank-dragoon"].lower() and "everybody" in w["frank-dragoon"].lower(), \
|
||||
"the abliterated endpoint must carry the common legal warning"
|
||||
assert "censored" in w["frank-zealot"].lower() and "uncensored" not in w["frank-zealot"].lower(), \
|
||||
"zealot ships the censored base since 2026-10-17: its card warning must not claim uncensored output"
|
||||
archon = next(s for s in fleet if s["name"] == "frank-archon")
|
||||
assert "permission" in archon["model"], "archon card needs the ask-Frank permission callout"
|
||||
PY
|
||||
|
||||
@@ -41,8 +41,8 @@ setup() {
|
||||
grep -q "repeat(auto-fill" "$BASE/style.css"
|
||||
}
|
||||
|
||||
@test "the user page keeps the AI section at the bottom" {
|
||||
grep -q "user: \['Web Apps', 'Media Library Apps', 'AI'\]" "$BASE/app.js"
|
||||
@test "the user page puts the Work in progress section last, below AI" {
|
||||
grep -q "user: \['Web Apps', 'Media Library Apps', 'AI', 'Work in progress'\]" "$BASE/app.js"
|
||||
}
|
||||
|
||||
@test "WIP badges distinguish open from closed work" {
|
||||
|
||||
@@ -13,9 +13,12 @@
|
||||
# 3. a downloaded plugin artifact is accepted only on its pinned sha256, its
|
||||
# official guid and its pinned version (a swapped or downgraded artifact
|
||||
# is refused before anything lands in plugins/);
|
||||
# 4. the TMDb key is written into the EXISTING config with every other
|
||||
# setting preserved — Jellyfin owns that file, and clobbering it would
|
||||
# silently reset country, language and adult settings;
|
||||
# 4. a key is written into the EXISTING config with every other setting
|
||||
# preserved, under whatever root that file's owner chose — Jellyfin's core
|
||||
# configs say `<Configuration>`, TheTVDB writes `<PluginConfiguration>` and
|
||||
# omits a null setting outright, so the element to fill can be absent and
|
||||
# the anchor has to come from the file; clobbering the file would silently
|
||||
# reset country, language and adult settings;
|
||||
# 5. a TVDB config the operator already saved through the UI is kept;
|
||||
# 6. install is idempotent: a second run changes no byte;
|
||||
# 7. whatever is replaced is copied to the trash first (fleet law: never
|
||||
@@ -24,6 +27,9 @@
|
||||
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
||||
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
||||
# installer to jellyfin.
|
||||
# 10. verify refuses a library that is a SILENT FILE INDEX — internet providers
|
||||
# switched off, or a type whose ImageFetchers is empty while its
|
||||
# MetadataFetchers list is populated (the 2026-11-01 missing-posters fault).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -59,6 +65,32 @@ setup() {
|
||||
TRASH="$BATS_TEST_TMPDIR/trash"
|
||||
SEC="$BATS_TEST_TMPDIR/secrets"
|
||||
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
||||
# The per-library option file Jellyfin 12 owns (it was library.xml). Conforming
|
||||
# by default so the pre-existing verify tests keep testing what they tested.
|
||||
mkdir -p "$JF/root/default/Movies"
|
||||
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<LibraryOptions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||
<EnableInternetProviders>true</EnableInternetProviders>
|
||||
<MetadataSavers>
|
||||
<string>Nfo</string>
|
||||
</MetadataSavers>
|
||||
<TypeOptions>
|
||||
<TypeOptions>
|
||||
<Type>Movie</Type>
|
||||
<MetadataFetchers>
|
||||
<string>TheMovieDb</string>
|
||||
</MetadataFetchers>
|
||||
<ImageFetchers>
|
||||
<string>TheMovieDb</string>
|
||||
</ImageFetchers>
|
||||
<ImageFetcherOrder>
|
||||
<string>TheMovieDb</string>
|
||||
</ImageFetcherOrder>
|
||||
</TypeOptions>
|
||||
</TypeOptions>
|
||||
</LibraryOptions>
|
||||
XML
|
||||
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||
@@ -225,6 +257,38 @@ XML
|
||||
grep -qF '<EnableSeasonPathMatching>true</EnableSeasonPathMatching>' "$cfg"
|
||||
}
|
||||
|
||||
@test "the key lands in a plugin-written config with no ApiKey element" {
|
||||
# What TheTVDB actually writes on a fresh install: its own root element and
|
||||
# no <ApiKey /> at all, because its serializer drops null settings. The
|
||||
# installer used to anchor on </Configuration> and refuse this file.
|
||||
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<PluginConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||
<SubscriberPIN />
|
||||
<CacheDurationInHours>1</CacheDurationInHours>
|
||||
<OriginalTitle>true</OriginalTitle>
|
||||
</PluginConfiguration>
|
||||
XML
|
||||
run prov install
|
||||
[ "$status" -eq 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF 'filled the ApiKey'
|
||||
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||
grep -qF "<ApiKey>$TVDB_KEY</ApiKey>" "$cfg"
|
||||
grep -qF '<CacheDurationInHours>1</CacheDurationInHours>' "$cfg"
|
||||
grep -qF '<OriginalTitle>true</OriginalTitle>' "$cfg"
|
||||
}
|
||||
|
||||
@test "a config file with no root element is refused by name" {
|
||||
# The guard still has to fire when there is genuinely nowhere to write —
|
||||
# a refusal naming the file beats inventing a document shape.
|
||||
printf '<?xml version="1.0" encoding="utf-8"?>\n' \
|
||||
>"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||
run prov install
|
||||
[ "$status" -ne 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qF 'Jellyfin.Plugin.Tvdb.xml'
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF 'no place to write'
|
||||
}
|
||||
|
||||
@test "a pin in the TVDB secret lands in the plugin config" {
|
||||
# A subscriber-shaped key needs the account's PIN beside it; a licensed key
|
||||
# does not, so the element appears only when the secret carries one.
|
||||
@@ -270,6 +334,69 @@ XML
|
||||
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
||||
}
|
||||
|
||||
@test "verify refuses a library whose internet providers are switched off" {
|
||||
# The whole library becomes a file index: refreshes answer 204 and store
|
||||
# nothing, and the item count still looks like a healthy scan.
|
||||
prov install >/dev/null
|
||||
sed -i 's#<EnableInternetProviders>true</EnableInternetProviders>#<EnableInternetProviders>false</EnableInternetProviders>#' \
|
||||
"$JF/root/default/Movies/options.xml"
|
||||
run prov verify
|
||||
[ "$status" -ne 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF 'SILENT INDEX Movies'
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF 'EnableInternetProviders'
|
||||
}
|
||||
|
||||
@test "verify refuses a type with no image fetchers while metadata fetchers are named" {
|
||||
prov install >/dev/null
|
||||
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<LibraryOptions>
|
||||
<EnableInternetProviders>true</EnableInternetProviders>
|
||||
<TypeOptions>
|
||||
<TypeOptions>
|
||||
<Type>Movie</Type>
|
||||
<MetadataFetchers>
|
||||
<string>TheMovieDb</string>
|
||||
</MetadataFetchers>
|
||||
<ImageFetchers />
|
||||
<ImageFetcherOrder>
|
||||
<string>TheMovieDb</string>
|
||||
</ImageFetcherOrder>
|
||||
</TypeOptions>
|
||||
</TypeOptions>
|
||||
</LibraryOptions>
|
||||
XML
|
||||
run prov verify
|
||||
[ "$status" -ne 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF 'ImageFetchers empty'
|
||||
}
|
||||
|
||||
@test "an older options.xml that never carried the switch is not refused" {
|
||||
# BoxSets on this host were written before the property existed; absence is
|
||||
# the vendor default, not a switched-off library, so it must not fail.
|
||||
prov install >/dev/null
|
||||
mkdir -p "$JF/root/default/Collections"
|
||||
cat >"$JF/root/default/Collections/options.xml" <<XML
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<LibraryOptions>
|
||||
<TypeOptions>
|
||||
<TypeOptions>
|
||||
<Type>BoxSet</Type>
|
||||
<MetadataFetchers>
|
||||
<string>TheMovieDb</string>
|
||||
</MetadataFetchers>
|
||||
<ImageFetchers>
|
||||
<string>TheMovieDb</string>
|
||||
</ImageFetchers>
|
||||
</TypeOptions>
|
||||
</TypeOptions>
|
||||
</LibraryOptions>
|
||||
XML
|
||||
run prov verify
|
||||
[ "$status" -eq 0 ]
|
||||
printf '%s\n' "${lines[@]}" | grep -qiF '= Collections'
|
||||
}
|
||||
|
||||
@test "no plugin configuration XML is tracked in this repo" {
|
||||
# These files hold provider keys and are owned by the running container;
|
||||
# tracking them would put credentials in git and guarantee drift.
|
||||
|
||||
@@ -10,7 +10,8 @@
|
||||
# 4. both media units bind the VIEW and refuse to start without it;
|
||||
# 5. only those two units touch the view — the LLM and library tenants keep the
|
||||
# raw volume, so the blast radius is two containers;
|
||||
# 6. the deployer brings the view up BEFORE it restarts either server;
|
||||
# 6. the deployer creates the upper base and brings the view up BEFORE it
|
||||
# restarts either server;
|
||||
# 7. the planner routes a change to these files to both media services.
|
||||
|
||||
load helpers
|
||||
@@ -110,6 +111,30 @@ UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
||||
done
|
||||
}
|
||||
|
||||
@test "ensure-base creates a base directory that install refuses to invent" {
|
||||
base="$BATS_TEST_TMPDIR/fresh-base"
|
||||
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||
[ "$status" -eq 0 ]
|
||||
[ -d "$base" ]
|
||||
# The deployer runs this on every media deploy, so a second call must be a
|
||||
# no-op and must say so rather than recreate anything.
|
||||
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "${lines[0]}" == *"exists"* ]]
|
||||
}
|
||||
|
||||
@test "the media deployer creates the upper base before it installs the view" {
|
||||
body="$(sed -n '/^ensure_media_view()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||
[ -n "$body" ]
|
||||
base="$(printf '%s\n' "$body" | grep -n 'media-view.sh ensure-base' | head -1 | cut -d: -f1)"
|
||||
inst="$(printf '%s\n' "$body" | grep -n 'media-view.sh install' | head -1 | cut -d: -f1)"
|
||||
# Order is the whole test: install refuses a missing base, so an installer
|
||||
# that runs first dies on a rebuilt host (measured 2026-10-31).
|
||||
[ -n "$base" ]
|
||||
[ -n "$inst" ]
|
||||
[ "$base" -lt "$inst" ]
|
||||
}
|
||||
|
||||
@test "the planner routes the view files to both media services" {
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
||||
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: scripts/media-refresh-jellyfin.py + scripts/media-refresh-plex.sh —
|
||||
# the two library-refresh drivers every media pass ends with.
|
||||
#
|
||||
# The rescan is the part of a media move that actually makes it visible, and
|
||||
# both servers punish the naive version of it. Jellyfin 12.4: the quoted
|
||||
# MediaBrowser header is the only auth form; POST /Library/Refresh is the only
|
||||
# door that queues work; a SECOND accepted POST cancels the scan already
|
||||
# running, so the index looks frozen while you hammer it; 429 means wait. Plex:
|
||||
# its HTTPD answers nobody outside its own netns and speaks TLS to everyone who
|
||||
# is inside the LAN, so the request has to be shipped INTO the container; the
|
||||
# token is PlexOnlineToken in the container's Preferences.xml; and progress is
|
||||
# readable without any of that, from section_locations.scanned_at. This seam
|
||||
# pins, against stubs that speak those APIs:
|
||||
# 1. the Jellyfin driver authenticates as `Authorization: MediaBrowser
|
||||
# Token="…"`, and an auth refusal ends the run naming the code;
|
||||
# 2. a trigger while Scan Media Library is running is REFUSED with no POST at
|
||||
# all (the cancel-restart trap); --force overrides;
|
||||
# 3. a 429 is waited out with a bounded number of attempts, never hammered,
|
||||
# and a still-busy server ends in a non-zero exit that says 429;
|
||||
# 4. `status` prints {"scan": …, "recentMedia": …} from the task table plus a
|
||||
# fixture DB, and `wait` returns only after N consecutive quiet reads;
|
||||
# 5. the Plex driver carries the token from the container's Preferences.xml
|
||||
# into a header — never argv, never a URL;
|
||||
# 6. a Plex trigger that gets no answer is a warming server: retried, bounded,
|
||||
# and reported as 000 rather than a service restart;
|
||||
# 7. Plex `status` answers from section_locations.scanned_at and is exit-0
|
||||
# only when that epoch has passed --after;
|
||||
# 8. a missing credential or database is refused by NAME before any request.
|
||||
|
||||
load helpers
|
||||
|
||||
setup() {
|
||||
JF="$REPO_ROOT/scripts/media-refresh-jellyfin.py"
|
||||
PX="$REPO_ROOT/scripts/media-refresh-plex.sh"
|
||||
TMP="$(mktemp -d)"
|
||||
PORT=18947
|
||||
printf 'stubkey\n' > "$TMP/jf-key"
|
||||
printf '<Preferences PlexOnlineToken="stubplextoken"/>\n' > "$TMP/prefs.xml"
|
||||
echo Idle > "$TMP/scan-state"
|
||||
echo ok > "$TMP/refresh-mode"
|
||||
echo 0 > "$TMP/polls"
|
||||
mkdir -p "$TMP/bin" "$TMP/container"
|
||||
# A podman that is really a local directory: the shim runs the runner the
|
||||
# driver copied in, so the in-container half is exercised for real.
|
||||
cat > "$TMP/bin/podman" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
echo "podman $*" >> "$PODMAN_LOG"
|
||||
case "$1" in
|
||||
cp) cp "$2" "$CONTAINER_FS/$(basename "$3" | sed 's#.*:##')" ;;
|
||||
exec) shift; [ "$1" = "$CONTAINER" ] || exit 1; shift
|
||||
[ "$1" = rm ] && exit 0
|
||||
[ "$1" = sh ] || exit 1; shift
|
||||
script="$1"; shift
|
||||
sh "$CONTAINER_FS/$(basename "$script")" "$@" ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
SH
|
||||
chmod +x "$TMP/bin/podman"
|
||||
cat > "$TMP/stub.py" <<'PY'
|
||||
import json, os, sys
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
port, logf = int(sys.argv[1]), sys.argv[2]
|
||||
|
||||
|
||||
def read(var, default=""):
|
||||
# Scenarios are flipped by rewriting a file; the env var names that file.
|
||||
try:
|
||||
return open(os.environ.get(var, "")).read().strip()
|
||||
except OSError:
|
||||
return default
|
||||
|
||||
|
||||
def log(line):
|
||||
with open(logf, "a") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def send(self, code, body, as_json=False):
|
||||
data = json.dumps(body).encode() if as_json else str(body).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json" if as_json else "text/xml")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def jf_authed(self):
|
||||
# Jellyfin 12.4 accepts the quoted MediaBrowser form ONLY: a bare
|
||||
# X-Emby-Token reads as unauthenticated here, exactly as it does live.
|
||||
return self.headers.get("Authorization") == 'MediaBrowser Token="stubkey"'
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
log("GET %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||
if path == "/System/Info":
|
||||
if self.jf_authed():
|
||||
return self.send(200, {"ServerName": "stub", "Version": "12.4.0"}, True)
|
||||
return self.send(401, "Unauthorized")
|
||||
if path == "/ScheduledTasks":
|
||||
with open(os.environ["POLL"]) as f:
|
||||
n = int(f.read().strip() or 0)
|
||||
with open(os.environ["POLL"], "w") as f:
|
||||
f.write(str(n + 1))
|
||||
state = read("SCAN", "Idle")
|
||||
flip = read("FLIP", "")
|
||||
if flip and n + 1 >= int(flip):
|
||||
state = "Idle"
|
||||
return self.send(200, [{"Name": "Audio Normalization", "State": "Running"},
|
||||
{"Name": "Scan Media Library", "State": state,
|
||||
"EndTimeUtc": read("ENDED", "") or None}], True)
|
||||
return self.send(404, "nope")
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0]
|
||||
log("POST %s auth=%s" % (self.path, "yes" if self.headers.get("Authorization") else "no"))
|
||||
if path == "/Library/Refresh":
|
||||
if not self.jf_authed():
|
||||
return self.send(401, "Unauthorized")
|
||||
if read("MODE", "ok") == "busy":
|
||||
return self.send(429, "too many requests")
|
||||
return self.send(204, "")
|
||||
if path.endswith("/library/refresh") or "/library/sections/" in path:
|
||||
if self.headers.get("X-Plex-Token") != "stubplextoken":
|
||||
return self.send(401, "Unauthorized")
|
||||
return self.send(204, "")
|
||||
return self.send(404, "nope")
|
||||
|
||||
|
||||
ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()
|
||||
PY
|
||||
export SCAN="$TMP/scan-state" MODE="$TMP/refresh-mode" POLL="$TMP/polls"
|
||||
export FLIP="$TMP/flip-after"
|
||||
export ENDED="$TMP/scan-ended"
|
||||
export PODMAN_LOG="$TMP/podman.log" CONTAINER_FS="$TMP/container" CONTAINER=plex
|
||||
: > "$PODMAN_LOG"
|
||||
python3 "$TMP/stub.py" "$PORT" "$TMP/requests.log" > "$TMP/stub.out" 2>&1 &
|
||||
STUB_PID=$!
|
||||
for _ in $(seq 1 50); do
|
||||
curl -s -o /dev/null "http://127.0.0.1:$PORT/System/Info" && break
|
||||
sleep 0.1
|
||||
done
|
||||
}
|
||||
|
||||
teardown() {
|
||||
[ -n "${STUB_PID:-}" ] && kill "$STUB_PID" 2>/dev/null || true
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
|
||||
plex_db() { # plex_db <path> <scanned_at epoch>
|
||||
python3 - "$1" "$2" <<'PY'
|
||||
import sqlite3, sys
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE section_locations (id INTEGER PRIMARY KEY, library_section_id INT,"
|
||||
" root_path TEXT, available INT, scanned_at INT)")
|
||||
db.execute("INSERT INTO section_locations VALUES (1, 1, '/mnt/entropy/Movies', 1, ?)",
|
||||
(int(sys.argv[2]),))
|
||||
db.commit()
|
||||
PY
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- jellyfin
|
||||
|
||||
@test "the jellyfin driver authenticates as MediaBrowser Token, never argv" {
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key"
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q '^POST /Library/Refresh' "$TMP/requests.log"
|
||||
! grep -q "stubkey" "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "a server that rejects the auth form is a failure, not a success" {
|
||||
# The stub 401s anything but the quoted form; the driver must report it
|
||||
# rather than exit 0 — a green trigger over a 401 is a lie.
|
||||
printf 'wrongkey\n' > "$TMP/bad-key"
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/bad-key"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"401"* ]]
|
||||
}
|
||||
|
||||
@test "a trigger during a running scan is refused with no POST at all" {
|
||||
echo Running > "$TMP/scan-state"
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"running"* ]]
|
||||
! grep -q '^POST /Library/Refresh' "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "--force triggers anyway while a scan runs" {
|
||||
echo Running > "$TMP/scan-state"
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" --force
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q '^POST /Library/Refresh' "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "a busy server is waited out with a bounded number of attempts" {
|
||||
echo busy > "$TMP/refresh-mode"
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--retries 2 --retry-sleep 0.1
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"429"* ]]
|
||||
posts=$(grep -c '^POST /Library/Refresh' "$TMP/requests.log")
|
||||
[ "$posts" -le 3 ]
|
||||
}
|
||||
|
||||
@test "status reports the scan task and recent media writes" {
|
||||
python3 - "$TMP/jellyfin.db" <<'PY'
|
||||
import sqlite3, sys, time
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE BaseItems (Id TEXT, DateModified TEXT)")
|
||||
now = int(time.time())
|
||||
for i in range(3):
|
||||
db.execute("INSERT INTO BaseItems VALUES (?, datetime(?, 'unixepoch'))", (str(i), now - 30))
|
||||
db.execute("INSERT INTO BaseItems VALUES ('old', datetime(?, 'unixepoch'))", (now - 86400,))
|
||||
db.commit()
|
||||
PY
|
||||
run python3 "$JF" status --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --window 5
|
||||
[ "$status" -eq 0 ]
|
||||
python3 -c 'import json,sys; d=json.loads(sys.argv[1]); assert d["scan"] is False, d; assert d["recentMedia"] == 3, d' "$output"
|
||||
echo Running > "$TMP/scan-state"
|
||||
run python3 "$JF" status --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --window 5
|
||||
python3 -c 'import json,sys; assert json.loads(sys.argv[1])["scan"] is True' "$output"
|
||||
}
|
||||
|
||||
@test "wait returns only after the scan reads quiet repeatedly" {
|
||||
python3 - "$TMP/jellyfin.db" <<'PY'
|
||||
import sqlite3, sys
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE BaseItems (Id TEXT, DateModified TEXT)")
|
||||
db.commit()
|
||||
PY
|
||||
echo Running > "$TMP/scan-state"
|
||||
echo 4 > "$TMP/flip-after"
|
||||
run python3 "$JF" wait --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --interval 0 --require-quiet 2 --timeout 30
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "${lines[-1]}" == *'"quiet": 2'* ]]
|
||||
# every read before poll 4 was busy, so no earlier line may claim quiet
|
||||
[ "$(printf '%s\n' "$output" | grep -c '"quiet": 0')" -ge 3 ]
|
||||
}
|
||||
|
||||
@test "wait gives up out loud when the library never settles" {
|
||||
python3 - "$TMP/jellyfin.db" <<'PY'
|
||||
import sqlite3, sys
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE BaseItems (Id TEXT, DateModified TEXT)")
|
||||
db.commit()
|
||||
PY
|
||||
echo Running > "$TMP/scan-state"
|
||||
run python3 "$JF" wait --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --interval 0 --timeout 0.2
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"still busy"* ]]
|
||||
}
|
||||
|
||||
@test "wait refuses a scan that never started instead of calling the library settled" {
|
||||
python3 - "$TMP/jellyfin.db" <<'PY'
|
||||
import sqlite3, sys
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE BaseItems (Id TEXT, DateModified TEXT)")
|
||||
db.commit()
|
||||
PY
|
||||
echo Idle > "$TMP/scan-state"
|
||||
run python3 "$JF" wait --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --interval 0.05 --start-window 0.4 --timeout 60
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"never started"* ]]
|
||||
[[ "$output" == *"cancels it"* ]]
|
||||
# a watcher that never saw the scan may not hand back a quiet count
|
||||
! grep -Eq '"quiet": [1-9]' <<<"$output"
|
||||
}
|
||||
|
||||
@test "status --after refuses to certify a scan that predates the moves" {
|
||||
python3 - "$TMP/jellyfin.db" <<'PY'
|
||||
import sqlite3, sys
|
||||
db = sqlite3.connect(sys.argv[1])
|
||||
db.execute("CREATE TABLE BaseItems (Id TEXT, DateModified TEXT)")
|
||||
db.commit()
|
||||
PY
|
||||
owed=$(python3 -c 'import time; print(int(time.time()) - 7200)')
|
||||
future=$(python3 -c 'import time; print(int(time.time()) + 7200)')
|
||||
date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%SZ > "$TMP/scan-ended"
|
||||
run python3 "$JF" status --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --after "$future"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"still owed"* ]]
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > "$TMP/scan-ended"
|
||||
run python3 "$JF" status --url "http://127.0.0.1:$PORT" --key-file "$TMP/jf-key" \
|
||||
--db "$TMP/jellyfin.db" --after "$owed"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *'"scannedAfter": true'* ]]
|
||||
}
|
||||
|
||||
@test "a missing jellyfin credential file is refused by name" {
|
||||
run python3 "$JF" trigger --url "http://127.0.0.1:$PORT" --key-file "$TMP/nope"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"$TMP/nope"* ]]
|
||||
! grep -q '^POST' "$TMP/requests.log"
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------------- plex
|
||||
|
||||
@test "the plex driver carries the token from Preferences.xml into a header" {
|
||||
run env PATH="$TMP/bin:$PATH" bash "$PX" trigger --library 1 \
|
||||
--url "http://127.0.0.1:$PORT" --prefs "$TMP/prefs.xml"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"refresh accepted (204)"* ]]
|
||||
grep -q "^POST /library/sections/1/refresh" "$TMP/requests.log"
|
||||
# the token reaches PMS as a header only: never in a URL, never in an argv
|
||||
! grep -q "stubplextoken" "$TMP/requests.log"
|
||||
! grep -q "stubplextoken" "$TMP/podman.log"
|
||||
}
|
||||
|
||||
@test "a plex trigger with no answer is a warming server, retried and named" {
|
||||
run env PATH="$TMP/bin:$PATH" bash "$PX" trigger --library 1 \
|
||||
--url "http://127.0.0.1:1" --prefs "$TMP/prefs.xml" --tries 2 --warm-sleep 0.1
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"000"* ]]
|
||||
[[ "$output" == *"warming"* ]] || [[ "$output" == *"never answered"* ]]
|
||||
[ "$(grep -c '^podman cp' "$TMP/podman.log")" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "a plex trigger with no token in Preferences.xml fails before any request" {
|
||||
printf '<Preferences/>\n' > "$TMP/prefs-empty.xml"
|
||||
run env PATH="$TMP/bin:$PATH" bash "$PX" trigger --library 1 \
|
||||
--url "http://127.0.0.1:$PORT" --prefs "$TMP/prefs-empty.xml"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"PlexOnlineToken"* ]]
|
||||
! grep -q '^POST' "$TMP/requests.log"
|
||||
}
|
||||
|
||||
@test "plex status reads the scan epoch from the database" {
|
||||
plex_db "$TMP/library.db" 1000
|
||||
run bash "$PX" status --library 1 --db "$TMP/library.db"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"section=1 scanned_at=1000"* ]]
|
||||
[[ "$output" == *"/mnt/entropy/Movies"* ]]
|
||||
run bash "$PX" status --library 1 --db "$TMP/library.db" --after 2000
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"still owed"* ]]
|
||||
run bash "$PX" status --library 1 --db "$TMP/library.db" --after 500
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "a plex status with no readable database is refused by name" {
|
||||
run bash "$PX" status --library 1 --db "$TMP/no-library.db"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"$TMP/no-library.db"* ]]
|
||||
}
|
||||
|
||||
@test "plex status names a section that has no location" {
|
||||
plex_db "$TMP/library.db" 1000
|
||||
run bash "$PX" status --library 77 --db "$TMP/library.db"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"section 77"* ]]
|
||||
}
|
||||
@@ -460,6 +460,39 @@ print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "
|
||||
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||
}
|
||||
|
||||
# A hand-written plan row whose verb is not in the executor's vocabulary used
|
||||
# to be dropped in silence: read_plan's if/elif chain had no else, so a plan of
|
||||
# nothing but typos applied as a successful no-op (hit for real on 2026-11-02,
|
||||
# when a pass meant to rename two series folders reported "applied: 0 moved"
|
||||
# and exit 0). An unread row is a plan bug, and a plan bug refuses.
|
||||
@test "a plan row with a verb this tool does not know stops the whole run" {
|
||||
plan
|
||||
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||
at "$LIB/Stray/Stray.S01E01.mkv" 4000
|
||||
printf 'rename\t%s\t%s\tfile\n' \
|
||||
"$LIB/Stray/Stray.S01E01.mkv" "$LIB/Stray/Stray - S01E01.mkv" >> "$OUT"
|
||||
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"rename"* ]]
|
||||
# all-or-nothing: the rows this tool DOES understand did not run either
|
||||
[ -e "$LIB/Stray/Stray.S01E01.mkv" ]
|
||||
[ ! -e "$JOURNAL" ]
|
||||
}
|
||||
|
||||
@test "a plan whose every row is an unknown verb refuses instead of reporting success" {
|
||||
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||
at "$LIB/Stray/Stray.S01E01.mkv" 4000
|
||||
{ printf 'action\tsrc\tdst\tkind\n'
|
||||
printf 'rename\t%s\t%s\tfile\n' \
|
||||
"$LIB/Stray/Stray.S01E01.mkv" "$LIB/Stray/Stray - S01E01.mkv"
|
||||
} > "$BATS_TEST_TMPDIR/typo.tsv"
|
||||
run python3 "$APPLY" apply --plan "$BATS_TEST_TMPDIR/typo.tsv" --root "$LIB" --journal "$JOURNAL" --apply
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"unknown action"* ]]
|
||||
[[ "$output" != *"applied"* ]]
|
||||
[ -e "$LIB/Stray/Stray.S01E01.mkv" ]
|
||||
}
|
||||
|
||||
@test "a run interrupted between the journal write and the move finishes the job" {
|
||||
plan
|
||||
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: scripts/media-tvdupes.py — the episode-level duplicate census for the TV
|
||||
# library.
|
||||
#
|
||||
# The failure this seam exists to prevent is the one that almost cost real
|
||||
# content: two folders of one show look like copies, and a folder-level "keep
|
||||
# the bigger folder" deletes the half of the series only the smaller folder
|
||||
# holds (the 2026-11-02 census found 10 such complementary pairs). So every
|
||||
# assertion here is about WHICH files land in the plan and WHICH the tool
|
||||
# refuses to touch — never about its internals. Everything runs on a synthetic
|
||||
# library in the test's own tmpdir: no media server, no network, no array.
|
||||
|
||||
load helpers
|
||||
|
||||
TOOL="$REPO_ROOT/scripts/media-tvdupes.py"
|
||||
|
||||
# at PATH SIZE — a file of deterministic bytes.
|
||||
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||
|
||||
# sources FILE — the sources the plan moves, one per line.
|
||||
sources() { awk -F'\t' '$1=="move"{print $2}' "$1"; }
|
||||
|
||||
# destinations FILE — where the plan puts them.
|
||||
destinations() { awk -F'\t' '$1=="move"{print $3}' "$1"; }
|
||||
|
||||
# plan_moves FILE — how many files the plan stages.
|
||||
plan_moves() {
|
||||
[ -f "$1" ] || { echo 0; return; }
|
||||
awk -F'\t' '$1=="move"' "$1" | wc -l
|
||||
}
|
||||
|
||||
setup() {
|
||||
LIB="$BATS_TEST_TMPDIR/TVShows"
|
||||
HOLD="$BATS_TEST_TMPDIR/.AGENTTRASH/20991231-tv-duplicates"
|
||||
PLAN="$BATS_TEST_TMPDIR/plan.tsv"
|
||||
mkdir -p "$LIB" "$HOLD"
|
||||
}
|
||||
|
||||
@test "the smaller copy of an episode is staged and the larger stays put" {
|
||||
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/Doc Who - S01E01 - Pilot.mkv" 4000
|
||||
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/backup/Doc.Who.S01E01.720p.mkv" 1200
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" == *"backup/Doc.Who.S01E01.720p.mkv"* ]]
|
||||
[[ "$output" != *"Pilot.mkv"* ]]
|
||||
run destinations "$PLAN"
|
||||
[[ "$output" == "$HOLD/"* ]]
|
||||
}
|
||||
|
||||
@test "an equal-size pair stages exactly one copy" {
|
||||
at "$LIB/Equal (2001) [tvdbid-11]/Season 01/Equal - S01E01.mkv" 2500
|
||||
at "$LIB/Equal (2001) [tvdbid-11]/Rip/Equal - S01E01.mkv" 2500
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "a file holding one episode nothing else has is never staged" {
|
||||
# The shape that defeats a folder-level verdict: Season 02 exists only here,
|
||||
# so the folder that looks like a leftover is the only copy of an episode.
|
||||
at "$LIB/King (2010) [tvdbid-21]/Season 01/King - S01E01 - One.mkv" 3000
|
||||
at "$LIB/King (2010) [tvdbid-21]/Season 01/backup/King - S01E01 - One.mkv" 900
|
||||
at "$LIB/King (2010) [tvdbid-21]/Season 02/King - S02E01 - Two.mkv" 800
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 0 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" != *"S02E01"* ]]
|
||||
[[ "$output" == *"backup/King - S01E01 - One.mkv"* ]]
|
||||
}
|
||||
|
||||
@test "a two-in-one pack competes for every episode in its range" {
|
||||
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv" 1500
|
||||
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 1500
|
||||
at "$LIB/Range (2012) [tvdbid-31]/Packs/Range - S01E01-02.mkv" 2400
|
||||
|
||||
# The pack is the biggest carrier of BOTH episodes, so the two rips go.
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" == *"Season 01/Range - S01E01.mkv"* ]]
|
||||
[[ "$output" != *"Packs/Range - S01E01-02.mkv"* ]]
|
||||
|
||||
# Make the season file the bigger carrier of E02 and the pack becomes the
|
||||
# only carrier of E01: it stays, however small it is.
|
||||
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv"
|
||||
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv"
|
||||
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 9000
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "two tvdb ids on the same words are never merged" {
|
||||
at "$LIB/Avatar The Last Airbender (2024) [tvdbid-385925]/Season 01/Avatar - S01E01.mkv" 9000
|
||||
at "$LIB/Avatar The Last Airbender (1999) [tvdbid-74852]/Season 01/Avatar - S01E01.mkv" 500
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||
[[ "$output" == *CONFLICT* ]]
|
||||
[[ "$output" == *"tvdb [74852] vs [385925]"* ]]
|
||||
}
|
||||
|
||||
@test "a derived id that contradicts the folder's own tag never merges them" {
|
||||
# Measured 2026-11-02: Jellyfin's provider table carried a 2026 special's tvdb
|
||||
# id on the folder tagged [tvdbid-73838]. Unioning the two makes one show of a
|
||||
# 2000 series and a 2026 special, and the special's bigger files dominate the
|
||||
# original's pilot. The tag is the operator's word; the nfo and `--ids` layers
|
||||
# fill an untagged folder and never overrule a tagged one.
|
||||
at "$LIB/Middle (2000) [tvdbid-73838]/Season 01/Middle - S01E01 - Pilot.mkv" 800
|
||||
at "$LIB/Middle (2026) [tvdbid-457779]/Season 01/Middle - S01E01 - Episode 1.mkv" 1100
|
||||
IDS="$BATS_TEST_TMPDIR/ids.tsv"
|
||||
printf '%s\t%s\n' "Middle (2000) [tvdbid-73838]" 457779 > "$IDS"
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --ids "$IDS"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||
[[ "$output" == *"DISAGREE"* ]]
|
||||
[[ "$output" == *"tag tvdb [73838]"* ]]
|
||||
[[ "$output" == *"derived tvdb [457779]"* ]]
|
||||
}
|
||||
|
||||
@test "an untagged drop is identified by the tvshow.nfo in the metadata view" {
|
||||
at "$LIB/Clone Wars/Season 01/Clone Wars - S01E01.mkv" 1200
|
||||
at "$LIB/Clone Wars (2008) [tvdbid-84859]/Season 01/Clone Wars - S01E01.mkv" 4000
|
||||
VIEW="$BATS_TEST_TMPDIR/view"
|
||||
mkdir -p "$VIEW/Clone Wars"
|
||||
printf '%s\n' '<tvshow><title>Clone Wars</title>' \
|
||||
'<uniqueid type="tvdb" default="true">84859</uniqueid></tvshow>' \
|
||||
> "$VIEW/Clone Wars/tvshow.nfo"
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --meta "$VIEW"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" == *"Clone Wars/Season 01/Clone Wars - S01E01.mkv"* ]]
|
||||
|
||||
# Without the nfo layer nothing identifies the drop, so nothing moves.
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" \
|
||||
--out "$BATS_TEST_TMPDIR/no-meta.tsv"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *UNRESOLVED* ]]
|
||||
[ "$(plan_moves "$BATS_TEST_TMPDIR/no-meta.tsv")" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "a file named by an id is keyed only when the witness names it" {
|
||||
BIG="$LIB/Edd (1999) [tvdbid-75897]/Season 01/Edd - S01E01 - Down the Line.mkv"
|
||||
SMALL="$LIB/Edd (1999) [tvdbid-75897]/Season 01/104348.mkv"
|
||||
at "$BIG" 2100
|
||||
at "$SMALL" 700
|
||||
WIT="$BATS_TEST_TMPDIR/witness.tsv"
|
||||
printf '%s\t1\t1\n' "$SMALL" > "$WIT"
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||
[[ "$output" == *"unkeyed, never staged: 1 files"* ]]
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --witness "$WIT"
|
||||
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" == *104348.mkv* ]]
|
||||
}
|
||||
|
||||
@test "the sidecars of a staged episode are staged with it" {
|
||||
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.mkv" 3000
|
||||
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.mkv" 1000
|
||||
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.srt" 400
|
||||
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.srt" 450
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||
run sources "$PLAN"
|
||||
[[ "$output" == *"Rip/Side - S01E01 - First.srt"* ]]
|
||||
[[ "$output" != *"Season 01/Side - S01E01 - First.srt"* ]]
|
||||
}
|
||||
|
||||
@test "a hold inside the library is refused" {
|
||||
at "$LIB/Hostage (2007) [tvdbid-51]/Season 01/Hostage - S01E01.mkv" 100
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$LIB/.AGENTTRASH/hold" \
|
||||
--out "$PLAN"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"--hold"* ]]
|
||||
}
|
||||
|
||||
@test "a plan whose hold target is occupied is refused, plan and all" {
|
||||
at "$LIB/Gone (2009) [tvdbid-61]/Season 01/Gone - S01E01.mkv" 1000
|
||||
at "$LIB/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||
|
||||
# The hold already holds one of the destinations: the executor would refuse
|
||||
# it, so a plan that cannot run must not be handed over at all.
|
||||
mkdir -p "$HOLD/Gone (2009) [tvdbid-61]/Rip"
|
||||
at "$HOLD/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||
rm "$PLAN"
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"hold target occupied"* ]]
|
||||
[ ! -e "$PLAN" ]
|
||||
}
|
||||
|
||||
@test "a library with nothing duplicated writes an empty plan" {
|
||||
at "$LIB/Unique (2015) [tvdbid-71]/Season 01/Unique - S01E01.mkv" 1000
|
||||
at "$LIB/Other Show (2016) [tvdbid-72]/Season 01/Other Show - S01E01.mkv" 1000
|
||||
|
||||
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||
[[ "$output" == *"plan: 0 moves"* ]]
|
||||
}
|
||||
@@ -799,14 +799,13 @@ assert not any('container_network' in e or 'container_fs_' in e for e in names),
|
||||
skip "franksdank/alloy image not present on this box"
|
||||
fi
|
||||
local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' "$REPO_ROOT/shared/alloy/config.alloy" > "$cfg"
|
||||
printf 'bogus_attr = "x"\n' >> "$cfg"
|
||||
! (
|
||||
cd "$REPO_ROOT"
|
||||
eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
|
||||
run_privileged() { "$@"; }
|
||||
cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
|
||||
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
|
||||
export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox" ALLOY_CONFIG_SRC="$cfg"
|
||||
copy_alloy_config
|
||||
)
|
||||
[ ! -f "$sandbox/config/config.alloy" ]
|
||||
@@ -820,7 +819,7 @@ assert not any('container_network' in e or 'container_fs_' in e for e in names),
|
||||
skip "franksdank/alloy image not present on this box"
|
||||
fi
|
||||
local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
|
||||
sed 's/GRAFANA_HOST_LABEL/dankclaw/g' "$REPO_ROOT/shared/alloy/config.alloy" > "$cfg"
|
||||
podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
|
||||
localhost/franksdank/alloy:latest validate /c.alloy
|
||||
}
|
||||