Compare commits
30
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0951d909c9 | ||
|
|
69effb542d | ||
|
|
8889a4ceff | ||
|
|
c21c48e61e | ||
|
|
7a8c6bd2e3 | ||
|
|
3c035ccef3 | ||
|
|
a6ec644ecb | ||
|
|
b7f4b7e148 | ||
|
|
ed1a20a960 | ||
|
|
511d8dca93 | ||
|
|
29aefc3b6f | ||
|
|
2e69215920 | ||
|
|
b93e8481f9 | ||
|
|
51ed5c0b2a | ||
|
|
9f9043c037 | ||
|
|
a25f139281 | ||
|
|
52600b28ab | ||
|
|
7cf388367e | ||
|
|
13830865f6 | ||
|
|
010b9affc7 | ||
|
|
ed5eed6d54 | ||
|
|
14f340370c | ||
|
|
50e0c5a786 | ||
|
|
649bbafa17 | ||
|
|
b7c19592e5 | ||
|
|
6c115c1394 | ||
|
|
6d79709cc0 | ||
|
|
0336f42e2a | ||
|
|
0f4a527337 | ||
|
|
1d92627935 |
No files matched your search
@@ -735,7 +735,12 @@ linked runbooks.
|
|||||||
**Ops**
|
**Ops**
|
||||||
|
|
||||||
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
||||||
Use `pkill -x <exact-name>`.
|
Use `pkill -x <exact-name>`, or the bracket trick — `pkill -f "serve[-]script.sh"` —
|
||||||
|
when the pattern has to be named: the bracket makes the pattern stop matching the
|
||||||
|
command line that carries it. Quote it: a zsh remote glob-expands an unquoted `[s]`
|
||||||
|
and the kill silently never runs. Re-hit 2026-11-02 —
|
||||||
|
`ssh host 'pkill -f jfshowswatch.py; …'` killed the remote shell mid-command, and
|
||||||
|
ssh exits **255 with zero output**, which reads exactly like a network fault.
|
||||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
||||||
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
||||||
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
||||||
@@ -818,6 +823,7 @@ per-attempt page and no per-swap ledger anywhere else.
|
|||||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||||
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||||
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||||
|
| Duplicate films and the purge hold (Dankserver) | `serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md` |
|
||||||
|
|
||||||
There is no page for a retired endpoint and no change diary. A payload swap
|
There is no page for a retired endpoint and no change diary. A payload swap
|
||||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||||
@@ -860,6 +866,6 @@ history.
|
|||||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||||
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||||
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
|
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Deployed on Dankserver 2026-10-31, re-verified 2026-11-01: both quadlets bind the view, not the raw array. A restart for any reason discards that server's in-flight library scan, so re-trigger the refresh — see [[Media Metadata View]].** |
|
||||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# Quadlet unit: Audiobookshelf audiobook server (rootless, boot-persistent).
|
||||||
|
#
|
||||||
|
# Install path on dankserver: ~/.config/containers/systemd/audiobookshelf.container
|
||||||
|
# Unit name: audiobookshelf.service (podman 5.x drops the container- prefix)
|
||||||
|
#
|
||||||
|
# Everything this server writes is its own state — the SQLite library, accounts,
|
||||||
|
# playback progress, cached covers — and it lives under
|
||||||
|
# /home/dank/Containers/audiobookshelf on the NVMe, so no metadata byte ever
|
||||||
|
# lands on the spindle. The audio itself is bound READ-ONLY off the array: the
|
||||||
|
# only sanctioned write path into /mnt/entropy is the overlay view
|
||||||
|
# (scripts/media-view.sh), Audiobookshelf has no need of it, and a read-only bind
|
||||||
|
# makes a stray sidecar write structurally impossible rather than discouraged.
|
||||||
|
#
|
||||||
|
# The digest is the linux/amd64 manifest of the tag, which is what podman
|
||||||
|
# verifies and what RepoDigests records on this single-arch host. Pinning the
|
||||||
|
# manifest-list digest instead would let a re-pull resolve to a different image.
|
||||||
|
# The upstream version tags stop at 2.19.0 (built 2025-02-02) while `latest` is
|
||||||
|
# newer by a year and a half, so `latest@digest` is the only pin that names the
|
||||||
|
# build actually running — same reasoning as the jellyfin and romm units.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Audiobookshelf audiobook server container
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Container]
|
||||||
|
ContainerName=audiobookshelf
|
||||||
|
Image=ghcr.io/advplyr/audiobookshelf:latest@sha256:37c2057e150694970442579815c646fe6bdc01c95b9d1547c4c9068be8844ef3
|
||||||
|
Network=pasta
|
||||||
|
# The names are the image's own: the server reads PORT, CONFIG_PATH, METADATA_PATH
|
||||||
|
# and ROUTER_BASE_PATH (measured against the digest below — `podman inspect` shows
|
||||||
|
# PORT=80 CONFIG_PATH=/config METADATA_PATH=/metadata in the image ENV, and the
|
||||||
|
# server's source greps for exactly these). The ABS_* family printed by the server
|
||||||
|
# README is the npm package's naming and is INERT here: with ABS_PORT=8000 the
|
||||||
|
# server listened on :80 under base path /audiobookshelf, so the unit sat
|
||||||
|
# `active (running)` while every probe to :8000 was refused (measured 2026-11-02,
|
||||||
|
# first deploy). ROUTER_BASE_PATH=/ is what makes the URL the plain host:port.
|
||||||
|
Environment=PORT=8000
|
||||||
|
Environment=ROUTER_BASE_PATH=/
|
||||||
|
Environment=CONFIG_PATH=/config
|
||||||
|
Environment=METADATA_PATH=/metadata
|
||||||
|
Environment=TZ=America/Vancouver
|
||||||
|
PublishPort=8000:8000
|
||||||
|
Volume=/home/dank/Containers/audiobookshelf/config:/config
|
||||||
|
Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata
|
||||||
|
Volume=/mnt/entropy/Audiobooks:/audiobooks:ro
|
||||||
|
# The image is Alpine: it ships wget and no curl, so the health check must name
|
||||||
|
# the binary that exists or the container is unhealthy while serving perfectly.
|
||||||
|
HealthCmd=wget -q -O /dev/null http://127.0.0.1:8000/healthcheck
|
||||||
|
HealthInterval=30s
|
||||||
|
HealthTimeout=10s
|
||||||
|
HealthRetries=3
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
# Refuse to start until the array is a real mount — with the bind source missing,
|
||||||
|
# podman invents an empty directory and Audiobookshelf records an empty library as
|
||||||
|
# truth. See the header of scripts/wait-for-mount.sh for the blind-boot incident.
|
||||||
|
ExecStartPre=/home/dank/serverAppConfigs/scripts/wait-for-mount.sh /mnt/entropy
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
# The first boot scans the whole audiobook tree. Weighted LOW (default 100) so a
|
||||||
|
# scan loses to playback when the array is contended. Proportional share, not a
|
||||||
|
# cap: inert while the disk is idle.
|
||||||
|
IOWeight=10
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -60,6 +60,7 @@ SERVICE_ORDER = (
|
|||||||
"calibre-server",
|
"calibre-server",
|
||||||
"plex",
|
"plex",
|
||||||
"tautulli",
|
"tautulli",
|
||||||
|
"audiobookshelf",
|
||||||
"prowlarr",
|
"prowlarr",
|
||||||
"qbittorrent",
|
"qbittorrent",
|
||||||
"radarr",
|
"radarr",
|
||||||
@@ -123,6 +124,7 @@ DANKSERVER_SERVICES = {
|
|||||||
"calibre-server",
|
"calibre-server",
|
||||||
"plex",
|
"plex",
|
||||||
"tautulli",
|
"tautulli",
|
||||||
|
"audiobookshelf",
|
||||||
"prowlarr",
|
"prowlarr",
|
||||||
"qbittorrent",
|
"qbittorrent",
|
||||||
"radarr",
|
"radarr",
|
||||||
@@ -420,6 +422,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
"copyparty": {"copyparty"},
|
"copyparty": {"copyparty"},
|
||||||
"plex": {"plex"},
|
"plex": {"plex"},
|
||||||
"tautulli": {"tautulli"},
|
"tautulli": {"tautulli"},
|
||||||
|
"audiobookshelf": {"audiobookshelf"},
|
||||||
}
|
}
|
||||||
_add(services, *quadlet_services.get(parts[1], set()))
|
_add(services, *quadlet_services.get(parts[1], set()))
|
||||||
if len(parts) >= 2 and parts[1] == "host":
|
if len(parts) >= 2 and parts[1] == "host":
|
||||||
|
|||||||
@@ -95,6 +95,7 @@ service_is_allowed() {
|
|||||||
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
|
dankserver:audiobookshelf|\
|
||||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||||
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||||
@@ -1212,6 +1213,10 @@ ensure_media_view() {
|
|||||||
# scripts/wait-for-mount.sh, which waits and then refuses the start. A
|
# scripts/wait-for-mount.sh, which waits and then refuses the start. A
|
||||||
# relayout must be followed by `media-view.sh reap`
|
# relayout must be followed by `media-view.sh reap`
|
||||||
# (see the Media Relayout page), which this is not.
|
# (see the Media Relayout page), which this is not.
|
||||||
|
# A fresh host has no upper base, and `install` refuses to invent one — so
|
||||||
|
# without this the media deploy dies on a rebuilt machine with
|
||||||
|
# "put it on the SSD first" (measured 2026-10-31).
|
||||||
|
run_privileged bash scripts/media-view.sh ensure-base
|
||||||
run_privileged bash scripts/media-view.sh install
|
run_privileged bash scripts/media-view.sh install
|
||||||
bash scripts/media-view.sh verify
|
bash scripts/media-view.sh verify
|
||||||
}
|
}
|
||||||
@@ -1296,6 +1301,25 @@ deploy_calibre_web() {
|
|||||||
systemctl --user restart calibre-web.service
|
systemctl --user restart calibre-web.service
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Audiobookshelf keeps its whole state (SQLite library, accounts, progress,
|
||||||
|
# cached covers) on the NVMe under /home/dank/Containers/audiobookshelf; the
|
||||||
|
# audio is a read-only bind off the array, so the only thing the deployer has to
|
||||||
|
# create is the two state roots. mkdir names the owner — letting podman invent
|
||||||
|
# the directories leaves the SQLite file unwritable by the container user on a
|
||||||
|
# rootless first boot. No `systemctl --user enable`: a generated Quadlet unit
|
||||||
|
# cannot be enabled, and WantedBy=default.target is what carries it across a
|
||||||
|
# reboot.
|
||||||
|
deploy_audiobookshelf() {
|
||||||
|
cp "dankserver/audiobookshelf/quadlet/audiobookshelf.container" "$quadlet_unit_dir/"
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
mkdir -p /home/dank/Containers/audiobookshelf/config \
|
||||||
|
/home/dank/Containers/audiobookshelf/metadata
|
||||||
|
systemctl --user restart audiobookshelf.service
|
||||||
|
# /healthcheck answers once the HTTP server is bound; the library scan that
|
||||||
|
# follows runs in the background and must not gate the deploy.
|
||||||
|
wait_for_health audiobookshelf http://127.0.0.1:8000/healthcheck 90
|
||||||
|
}
|
||||||
|
|
||||||
# The content server's single admin account and the desktop's basic-auth
|
# The content server's single admin account and the desktop's basic-auth
|
||||||
# credential (same pair) are bootstrapped exactly once: the env file is the
|
# credential (same pair) are bootstrapped exactly once: the env file is the
|
||||||
# plaintext store (mode 600), the userdb's pw column is plaintext too, so
|
# plaintext store (mode 600), the userdb's pw column is plaintext too, so
|
||||||
@@ -2172,6 +2196,7 @@ deploy_service() {
|
|||||||
copyparty) deploy_copyparty ;;
|
copyparty) deploy_copyparty ;;
|
||||||
plex) deploy_plex ;;
|
plex) deploy_plex ;;
|
||||||
tautulli) deploy_tautulli ;;
|
tautulli) deploy_tautulli ;;
|
||||||
|
audiobookshelf) deploy_audiobookshelf ;;
|
||||||
alloy) deploy_alloy ;;
|
alloy) deploy_alloy ;;
|
||||||
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
|
||||||
gpu-exporter) deploy_gpu_exporter ;;
|
gpu-exporter) deploy_gpu_exporter ;;
|
||||||
|
|||||||
@@ -148,6 +148,7 @@ valid_service() {
|
|||||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
|
dankserver:audiobookshelf|\
|
||||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
|
||||||
dankserver:prowlarr|dankserver:frank-archon|\
|
dankserver:prowlarr|dankserver:frank-archon|\
|
||||||
|
|||||||
@@ -35,7 +35,8 @@
|
|||||||
# Subcommands
|
# Subcommands
|
||||||
# plan print the resolved shape and every pending change, touch nothing
|
# plan print the resolved shape and every pending change, touch nothing
|
||||||
# install fetch+verify the plugin, then write the keys (backing up first)
|
# install fetch+verify the plugin, then write the keys (backing up first)
|
||||||
# verify prove the plugin and both keys are actually in place
|
# verify prove the plugin and both keys are in place, and that no library
|
||||||
|
# is a silent file index (providers off / image fetchers empty)
|
||||||
#
|
#
|
||||||
# Nothing here restarts Jellyfin: the deployer owns that
|
# Nothing here restarts Jellyfin: the deployer owns that
|
||||||
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
||||||
@@ -340,6 +341,64 @@ cmd_install() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# A library whose options.xml carries EnableInternetProviders=false, or an empty
|
||||||
|
# ImageFetchers list beside a populated MetadataFetchers list, is a SILENT FILE
|
||||||
|
# INDEX: it indexes every file, answers 204 to every refresh, stores no provider
|
||||||
|
# id, plot, rating or artwork, and logs nothing — so the item count reads like a
|
||||||
|
# healthy scan. Jellyfin 12 keeps these options in root/default/<Library>/options.xml
|
||||||
|
# (it was library.xml) and stores NONE of them in the database, so this audit is
|
||||||
|
# the only place the state can be read without a browser session.
|
||||||
|
verify_library_options() {
|
||||||
|
local root="$JF_ROOT/root/default" f lib problems
|
||||||
|
if [ ! -d "$root" ]; then
|
||||||
|
printf '= no library options under %s (nothing audited)\n' "$root"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local rc=0
|
||||||
|
for f in "$root"/*/options.xml; do
|
||||||
|
[ -f "$f" ] || continue
|
||||||
|
lib="$(basename "$(dirname "$f")")"
|
||||||
|
problems="$(python3 -s - "$f" <<'PY'
|
||||||
|
import sys, xml.dom.minidom as X
|
||||||
|
root = X.parse(sys.argv[1]).documentElement
|
||||||
|
def child(el, name):
|
||||||
|
for n in el.childNodes:
|
||||||
|
if n.nodeType == n.ELEMENT_NODE and n.tagName == name:
|
||||||
|
return n
|
||||||
|
def flag(el, name):
|
||||||
|
n = child(el, name)
|
||||||
|
return None if n is None else (n.firstChild.data.strip() if n.firstChild else "")
|
||||||
|
def names(el, name):
|
||||||
|
n = child(el, name)
|
||||||
|
return None if n is None else [s.firstChild.data.strip() for s in n.childNodes
|
||||||
|
if s.nodeType == s.ELEMENT_NODE and s.firstChild]
|
||||||
|
bad = []
|
||||||
|
ip = flag(root, "EnableInternetProviders")
|
||||||
|
if ip is not None and ip.lower() != "true":
|
||||||
|
bad.append("EnableInternetProviders=%s" % (ip or "empty"))
|
||||||
|
types = child(root, "TypeOptions")
|
||||||
|
if types is not None:
|
||||||
|
for t in types.childNodes:
|
||||||
|
if t.nodeType != t.ELEMENT_NODE or t.tagName != "TypeOptions":
|
||||||
|
continue
|
||||||
|
meta = names(t, "MetadataFetchers") or []
|
||||||
|
if meta and not (names(t, "ImageFetchers") or []):
|
||||||
|
bad.append("%s: ImageFetchers empty while MetadataFetchers names %d"
|
||||||
|
% (flag(t, "Type") or "?", len(meta)))
|
||||||
|
print("; ".join(bad))
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
if [ -n "$problems" ]; then
|
||||||
|
printf 'SILENT INDEX %s: %s\n' "$lib" "$problems" >&2
|
||||||
|
rc=1
|
||||||
|
else
|
||||||
|
printf '= %s: library options fetch from providers\n' "$lib"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
cmd_verify() {
|
cmd_verify() {
|
||||||
check_root
|
check_root
|
||||||
local bad=0
|
local bad=0
|
||||||
@@ -378,6 +437,7 @@ cmd_verify() {
|
|||||||
printf '%s %s chars\n' "$elem" "${#n}"
|
printf '%s %s chars\n' "$elem" "${#n}"
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
verify_library_options || bad=1
|
||||||
[ "$bad" -eq 0 ] || return 1
|
[ "$bad" -eq 0 ] || return 1
|
||||||
printf 'providers configured\n'
|
printf 'providers configured\n'
|
||||||
}
|
}
|
||||||
|
|||||||
Executable
+473
@@ -0,0 +1,473 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""media-tvdupes.py — find duplicate TV episodes and plan their staging, one
|
||||||
|
EPISODE at a time.
|
||||||
|
|
||||||
|
A TV library grows the same show into several folders: a tagged season box, an
|
||||||
|
untagged packager drop, a `Complete` folder that is not complete. The folders
|
||||||
|
look like copies of each other and are not, so the unit of comparison is the
|
||||||
|
episode and never the folder. A file is droppable only when EVERY episode it
|
||||||
|
carries is also held, somewhere else in the same show, by a file at least as
|
||||||
|
large — Frank's rule: the same episode in two places keeps the larger one. A
|
||||||
|
file that holds one episode nothing else has stays, whatever else it carries.
|
||||||
|
|
||||||
|
It writes a plan and moves nothing; `scripts/media-apply.py` is the only thing
|
||||||
|
that renames. The plan stages into a dated hold, so the pass is undoable from
|
||||||
|
the journal and frees no space until the hold is purged.
|
||||||
|
|
||||||
|
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||||
|
--hold /mnt/entropy/.AGENTTRASH/20261102-tv-duplicates \
|
||||||
|
--out /mnt/entropy/tv-dupes-plan.tsv
|
||||||
|
python3 scripts/media-apply.py apply --plan /mnt/entropy/tv-dupes-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/tv-dupes.jsonl \
|
||||||
|
--metadata-view /var/lib/media-meta/view --apply
|
||||||
|
python3 scripts/media-apply.py verify --plan /mnt/entropy/tv-dupes-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/tv-dupes.jsonl
|
||||||
|
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/tv-dupes.jsonl --apply # empty folders only
|
||||||
|
scripts/media-view.sh reap # ghost sidecars above
|
||||||
|
|
||||||
|
Then both servers have to be told: the plan moves files on the raw array, and no
|
||||||
|
watcher crosses the metadata view — [[Media Relayout]] owns the refresh order.
|
||||||
|
|
||||||
|
Guards, and why each one is here:
|
||||||
|
|
||||||
|
* **Identity before geometry.** Two folders holding the same words are not the
|
||||||
|
same show: the 2026-11-02 census found `Avatar: The Last Airbender` (1999,
|
||||||
|
tvdb 74852) beside the 2024 remake (385925), `Star Trek: Voyager` beside
|
||||||
|
`Star Trek: The Animated Series`, and a German thriller Jellyfin had titled
|
||||||
|
`Millennium (2022) [tvdbid-472933]` beside `Millennium (1996)`. A folder's
|
||||||
|
identity is read from three places, in this order: its `[tvdbid-N]` tag, the
|
||||||
|
`tvshow.nfo` Jellyfin wrote — which lives in the metadata view, not on the
|
||||||
|
array, and is what `--meta` reads — and a `folder<TAB>id` map passed with
|
||||||
|
`--ids`. The tag outranks the derived layers: an id that contradicts a folder's
|
||||||
|
own tag is reported and ignored, never unioned — measured 2026-11-02, Jellyfin's
|
||||||
|
provider table carried a 2026 special's id on a folder tagged `[tvdbid-73838]`,
|
||||||
|
and unioning them makes one show of the 2000 series and the special, whose bigger
|
||||||
|
files then dominate the original's pilot. Two folders are one show when they share
|
||||||
|
an id, or when neither has one and name and year both agree. Two disjoint id sets
|
||||||
|
is an absolute stop: the pair is reported as a conflict and nothing under either is
|
||||||
|
staged. Same name with nothing to decide on is reported unresolved — the fix is the
|
||||||
|
tag, not a guess.
|
||||||
|
* **A folder-level verdict destroys content.** 10 of the 25 shows that looked
|
||||||
|
duplicated held COMPLEMENTARY halves: `Game of Thrones Complete` carries no
|
||||||
|
S01 and no S08, `Modern Family S05` is the only copy of season 5, `Loiter
|
||||||
|
Squad Uncensored` is the only copy of season 1. Judged per folder, 20 shows
|
||||||
|
and some 340 GB of unique episodes would have been staged; judged per episode
|
||||||
|
the true figure was a sixth of that.
|
||||||
|
* **A bare number is never an episode.** `256491.mp4` may be an episode id, a
|
||||||
|
poster, or a season, so a name that states no `SxxEyy` / `1x04` marker is
|
||||||
|
unkeyed: never staged, and counted in the report. A media server's own episode
|
||||||
|
table does know those files — pass it with `--witness` and its rows key
|
||||||
|
whatever the server has matched.
|
||||||
|
* **The audit re-reads the disk.** Sizes and the survivor's existence are
|
||||||
|
checked again when the plan is written, because the census is a snapshot: a
|
||||||
|
file that grew, vanished, or landed in the hold in between must fail the plan,
|
||||||
|
not surprise the executor.
|
||||||
|
* **A sidecar that shares an episode's stem follows it.** Subtitles and `.nfo`
|
||||||
|
left at the old coordinates are ghost bait for `media-view.sh reap`.
|
||||||
|
* A multi-episode file (`S01E01-02`) is dominated only when every episode in its
|
||||||
|
range is held elsewhere — and it competes for the survivor seat on each of
|
||||||
|
them, because the byte law compares files. A pack that states its range in
|
||||||
|
the name is a name both media servers parse the same way, so the pack winning
|
||||||
|
against two single-episode rips is the law applied, not a misread.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import collections
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||||
|
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||||
|
eng = importlib.util.module_from_spec(_spec)
|
||||||
|
_spec.loader.exec_module(eng)
|
||||||
|
|
||||||
|
# The executor's definition of a media file, plus the two the title engine omits:
|
||||||
|
# an audit that disagrees with media-apply.py is a plan that dies at its
|
||||||
|
# preflight instead of at the census.
|
||||||
|
VIDEO = tuple(sorted(set(eng.VIDEO_SUFFIX) | {".ts", ".wmv"}))
|
||||||
|
TAG_RE = re.compile(r"\[\s*tvdbid[\s._-]*(\d+)\s*\]", re.I)
|
||||||
|
YEAR_RE = re.compile(r"[(\[]\s*((?:19|20)\d\d)\s*[)\]]")
|
||||||
|
SEASON_EP = re.compile(r"(?i)\bs(?:eason)?[\s._-]*(\d{1,2})[\s._-]*e(?:p(?:isode)?)?"
|
||||||
|
r"[\s._-]*(\d{1,3})(?:[\s._-]*(?:-|to|\+)[\s._-]*e?"
|
||||||
|
r"[\s._-]*(\d{1,3}))?")
|
||||||
|
NXN = re.compile(r"(?i)(?<!\d)(\d{1,2})x(\d{1,3})(?!\d)")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--library", default="/mnt/entropy/TVShows",
|
||||||
|
help="the TV library to census (never written)")
|
||||||
|
ap.add_argument("--hold", required=True,
|
||||||
|
help="dated trash the plan stages into, e.g. "
|
||||||
|
"/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates")
|
||||||
|
ap.add_argument("--out", default=None,
|
||||||
|
help="plan TSV for scripts/media-apply.py (default: "
|
||||||
|
"tv-dupes-plan.tsv beside --library)")
|
||||||
|
ap.add_argument("--witness", default=None,
|
||||||
|
help="TSV of path<TAB>season<TAB>episode out of a media "
|
||||||
|
"server's episode table; keys files whose names carry"
|
||||||
|
" no marker")
|
||||||
|
ap.add_argument("--meta", default=None,
|
||||||
|
help="the merged metadata view (on the host "
|
||||||
|
"/var/lib/media-meta/view): tvshow.nfo lives in the"
|
||||||
|
" upper layer, so this is where an untagged folder's"
|
||||||
|
" tvdb id is readable")
|
||||||
|
ap.add_argument("--ids", default=None,
|
||||||
|
help="TSV of folder<TAB>tvdbid, one line per folder, for"
|
||||||
|
" identity a lookup settled and the folder name does"
|
||||||
|
" not state")
|
||||||
|
ap.add_argument("--json", default=None,
|
||||||
|
help="also dump the analysis as JSON")
|
||||||
|
return ap.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
ARGS = parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def episode_keys(name):
|
||||||
|
"""Every (season, episode) an episode name states.
|
||||||
|
|
||||||
|
`S01E03`, `Season 1 E3`, `1x04`, and a range `S01E01-02`, which expands so a
|
||||||
|
two-in-one file is only droppable when both halves are held elsewhere. A
|
||||||
|
bare number states nothing and yields nothing."""
|
||||||
|
keys = set()
|
||||||
|
for m in SEASON_EP.finditer(name):
|
||||||
|
s, first = int(m.group(1)), int(m.group(2))
|
||||||
|
keys.add((s, first))
|
||||||
|
if m.group(3):
|
||||||
|
keys.update((s, e) for e in range(first + 1, int(m.group(3)) + 1))
|
||||||
|
keys.update((int(m.group(1)), int(m.group(2))) for m in NXN.finditer(name))
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def folder_identity(folder):
|
||||||
|
"""(fold_sep name, year, tvdb ids) for a top-level show folder.
|
||||||
|
|
||||||
|
Tag and year come in both orders — `Name (2008) [tvdbid-84859]` and `Name
|
||||||
|
[tvdbid-84859] (2008)` are both in the library — so neither is assumed to
|
||||||
|
sit at the end, and both are cut out of the name folders compare on."""
|
||||||
|
ids = frozenset(int(x) for x in TAG_RE.findall(folder))
|
||||||
|
year = None
|
||||||
|
for _ in range(3):
|
||||||
|
y = YEAR_RE.search(folder)
|
||||||
|
if not y:
|
||||||
|
break
|
||||||
|
year = year or y.group(1)
|
||||||
|
folder = TAG_RE.sub(" ", f"{folder[:y.start()]} {folder[y.end():]}")
|
||||||
|
return eng.fold_sep(TAG_RE.sub(" ", folder)), year, ids
|
||||||
|
|
||||||
|
|
||||||
|
def nfo_id(path):
|
||||||
|
"""The tvdb id a `tvshow.nfo` states, or None.
|
||||||
|
|
||||||
|
Three shapes are in the library: Jellyfin's `<uniqueid type="tvdb">`, Kodi's
|
||||||
|
`<tvdbid>`, and a packager's `<id>` + `<id_type>` pair; some carry the id
|
||||||
|
only in the title line as `[tvdbid-N]`."""
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
text = fh.read(16384)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
for rx in (r"<tvdbid\s*>\s*(\d+)\s*<",
|
||||||
|
r"<uniqueid[^>]*type\s*=\s*[\"']?tvdb[^>]*>\s*(\d+)\s*<",
|
||||||
|
r"<id\s*>\s*(\d+)\s*</id>\s*<id_type>\s*(?:tvdb|thedotdb)",
|
||||||
|
r"\[\s*tvdbid[\s._-]*(\d+)\s*\]"):
|
||||||
|
m = re.search(rx, text, re.I)
|
||||||
|
if m:
|
||||||
|
return int(m.group(1))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def collect_ids(library, meta, ids_file):
|
||||||
|
"""top-level folder -> {tvdb ids} from the nfo layer and an operator map.
|
||||||
|
|
||||||
|
`meta` is the merged metadata view: Jellyfin's sidecars live in the upper
|
||||||
|
layer, so a folder's `tvshow.nfo` is not on the raw array. Reading the id
|
||||||
|
from there is what makes an untagged packager folder identifiable at all —
|
||||||
|
without it the pair stays unresolved and nothing under it is staged."""
|
||||||
|
out = collections.defaultdict(set)
|
||||||
|
if meta:
|
||||||
|
try:
|
||||||
|
tops = sorted(os.listdir(library))
|
||||||
|
except OSError:
|
||||||
|
tops = []
|
||||||
|
for folder in tops:
|
||||||
|
i = nfo_id(os.path.join(meta, folder, "tvshow.nfo"))
|
||||||
|
if i:
|
||||||
|
out[folder].add(i)
|
||||||
|
if ids_file:
|
||||||
|
with open(ids_file, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for line in fh:
|
||||||
|
cols = line.rstrip("\n").split("\t")
|
||||||
|
if len(cols) < 2 or not cols[1].strip().isdigit():
|
||||||
|
continue
|
||||||
|
out[cols[0]].add(int(cols[1].strip()))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def witness_rows(path):
|
||||||
|
"""absolute path -> {(season, episode)} from a media server's episode table."""
|
||||||
|
out = collections.defaultdict(set)
|
||||||
|
if not path:
|
||||||
|
return out
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||||
|
for line in fh:
|
||||||
|
cols = line.rstrip("\n").split("\t")
|
||||||
|
if len(cols) < 3 or not cols[0].startswith("/"):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out[os.path.abspath(cols[0])].add((int(cols[1]), int(cols[2])))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def sidecars(path):
|
||||||
|
"""The files beside PATH carrying its exact stem — subtitles, nfo, artwork."""
|
||||||
|
folder, stem = os.path.split(path)
|
||||||
|
stem = os.path.splitext(stem)[0]
|
||||||
|
out = []
|
||||||
|
try:
|
||||||
|
entries = sorted(os.listdir(folder))
|
||||||
|
except OSError:
|
||||||
|
return out
|
||||||
|
for e in entries:
|
||||||
|
p = os.path.join(folder, e)
|
||||||
|
if p == path or os.path.islink(p) or not os.path.isfile(p):
|
||||||
|
continue
|
||||||
|
if os.path.splitext(e)[0] == stem:
|
||||||
|
out.append(p)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def scan(library, hold, witness):
|
||||||
|
"""Every video file under LIBRARY: relative path, size, episode keys.
|
||||||
|
|
||||||
|
A file keys from the witness when it has a row (a server matched it), and
|
||||||
|
from its own name otherwise; with neither it is unkeyed."""
|
||||||
|
files = []
|
||||||
|
for dirpath, dirnames, filenames in os.walk(library):
|
||||||
|
dirnames[:] = sorted(d for d in dirnames
|
||||||
|
if not d.startswith(".") and d != "@eaDir"
|
||||||
|
and os.path.abspath(os.path.join(dirpath, d)) != hold)
|
||||||
|
for f in sorted(filenames):
|
||||||
|
p = os.path.join(dirpath, f)
|
||||||
|
if os.path.islink(p) or not f.lower().endswith(VIDEO):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
size = os.path.getsize(p)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
files.append({"path": os.path.abspath(p),
|
||||||
|
"rel": os.path.relpath(os.path.abspath(p), library),
|
||||||
|
"size": size,
|
||||||
|
"keys": set(witness.get(os.path.abspath(p))
|
||||||
|
or episode_keys(os.path.splitext(f)[0]))})
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
def show_groups(folders, extra_ids):
|
||||||
|
"""(show groups, conflicts, unresolved, disagreements) over top-level folders.
|
||||||
|
|
||||||
|
A folder's id is its `[tvdbid-N]` tag; the nfo layer and `--ids` fill a folder
|
||||||
|
only when it carries no tag. The rule, in order of trust: an id in common makes
|
||||||
|
one show; two disjoint id sets are two shows, reported as a conflict and never
|
||||||
|
merged; no id on either side merges only on name AND year (the 1999 miniseries
|
||||||
|
and the 2015 series are not one show); one side ided and the other bare is
|
||||||
|
unresolved, because the fix is the tag.
|
||||||
|
|
||||||
|
The derived layers never overrule a tag. Measured 2026-11-02: Jellyfin's
|
||||||
|
provider table carried a 2026 special's tvdb id on the folder tagged
|
||||||
|
`[tvdbid-73838]`, and unioning the two made one show of the 2000 series and the
|
||||||
|
special — whose bigger files then dominate the original's pilot. A contradiction
|
||||||
|
is reported instead, because the tag is the operator's word and the provider
|
||||||
|
table is a guess that can be wrong.
|
||||||
|
"""
|
||||||
|
names = {f: folder_identity(f) for f in folders}
|
||||||
|
tags = {f: set(names[f][2]) for f in folders}
|
||||||
|
derived = {f: set(extra_ids.get(f, ())) for f in folders}
|
||||||
|
disagree = {f: (sorted(tags[f]), sorted(derived[f] - tags[f]))
|
||||||
|
for f in folders if tags[f] and derived[f] - tags[f]}
|
||||||
|
ids = {f: (tags[f] or derived[f]) for f in folders}
|
||||||
|
parent = {f: f for f in folders}
|
||||||
|
|
||||||
|
def find(f):
|
||||||
|
while parent[f] != f:
|
||||||
|
parent[f] = parent[parent[f]]
|
||||||
|
f = parent[f]
|
||||||
|
return f
|
||||||
|
|
||||||
|
conflicts, unresolved = [], []
|
||||||
|
order = sorted(folders)
|
||||||
|
for i, a in enumerate(order):
|
||||||
|
for b in order[i + 1:]:
|
||||||
|
na, ya, _ia = names[a]
|
||||||
|
nb, yb, _ib = names[b]
|
||||||
|
if na != nb:
|
||||||
|
continue
|
||||||
|
ia, ib = ids[a], ids[b]
|
||||||
|
if ia & ib:
|
||||||
|
parent[find(b)] = find(a)
|
||||||
|
elif ia and ib:
|
||||||
|
conflicts.append((a, b, f"tvdb {sorted(ia)} vs {sorted(ib)}"))
|
||||||
|
elif not ia and not ib and ya and ya == yb:
|
||||||
|
parent[find(b)] = find(a)
|
||||||
|
elif not ia and not ib:
|
||||||
|
conflicts.append((a, b,
|
||||||
|
f"same name, years {ya or '?'} vs {yb or '?'}"))
|
||||||
|
else:
|
||||||
|
unresolved.append((a, b, "one folder carries a tvdb id,"
|
||||||
|
" the other none"))
|
||||||
|
groups = collections.defaultdict(list)
|
||||||
|
for f in order:
|
||||||
|
groups[find(f)].append(f)
|
||||||
|
return sorted(groups.values()), conflicts, unresolved, disagree
|
||||||
|
|
||||||
|
|
||||||
|
def rank(rec):
|
||||||
|
"""Survivor order: biggest, then shortest path, then lexicographic.
|
||||||
|
|
||||||
|
A strict total order per key means a key's winner is never itself dominated,
|
||||||
|
so no chain of staging can cycle and exactly one copy of a byte-identical
|
||||||
|
pair is staged."""
|
||||||
|
return (-rec["size"], len(rec["path"]), rec["path"])
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
library = os.path.abspath(ARGS.library)
|
||||||
|
hold = os.path.abspath(ARGS.hold)
|
||||||
|
out = ARGS.out or os.path.join(os.path.dirname(library), "tv-dupes-plan.tsv")
|
||||||
|
if not os.path.isdir(library):
|
||||||
|
sys.exit(f"refused: {library} is not a directory")
|
||||||
|
if hold == library or hold.startswith(library + os.sep):
|
||||||
|
sys.exit(f"refused: --hold {hold} sits inside the library; a hold the"
|
||||||
|
" census walks is a plan that eats its own output")
|
||||||
|
|
||||||
|
files = scan(library, hold, witness_rows(ARGS.witness))
|
||||||
|
folders = sorted({r["rel"].split(os.sep)[0] for r in files})
|
||||||
|
ids = collect_ids(library, ARGS.meta, ARGS.ids)
|
||||||
|
groups, conflicts, unresolved, disagree = show_groups(folders, ids)
|
||||||
|
|
||||||
|
by_show = collections.defaultdict(list)
|
||||||
|
for members in groups:
|
||||||
|
for r in files:
|
||||||
|
if r["rel"].split(os.sep)[0] in members:
|
||||||
|
by_show[members[0]].append(r)
|
||||||
|
|
||||||
|
rows, audits, report, shows_json = [], [], [], []
|
||||||
|
for show in sorted(by_show):
|
||||||
|
recs = by_show[show]
|
||||||
|
holders = collections.defaultdict(list)
|
||||||
|
for r in recs:
|
||||||
|
for k in r["keys"]:
|
||||||
|
holders[k].append(r)
|
||||||
|
winner = {k: min(v, key=rank) for k, v in holders.items()}
|
||||||
|
dominated = [r for r in recs if r["keys"]
|
||||||
|
and all(winner[k] is not r for k in r["keys"])]
|
||||||
|
if not dominated:
|
||||||
|
continue
|
||||||
|
dup_keys = sorted({k for r in dominated for k in r["keys"]})
|
||||||
|
sole = sum(1 for v in holders.values() if len(v) == 1)
|
||||||
|
staged_folders = {os.path.dirname(r["rel"]) for r in dominated}
|
||||||
|
for r in dominated:
|
||||||
|
for k in r["keys"]:
|
||||||
|
t = winner[k]
|
||||||
|
try:
|
||||||
|
now_src = os.path.getsize(r["path"])
|
||||||
|
now_twin = os.path.getsize(t["path"])
|
||||||
|
except OSError:
|
||||||
|
audits.append(f"gone since the census: {r['rel']}"
|
||||||
|
f" or its survivor {t['rel']}")
|
||||||
|
break
|
||||||
|
if now_twin < now_src:
|
||||||
|
audits.append(f"survivor smaller on disk: {r['rel']}"
|
||||||
|
f" {now_src} > {t['rel']} {now_twin}")
|
||||||
|
for r in dominated:
|
||||||
|
dst = os.path.join(hold, r["rel"])
|
||||||
|
rows.append((r["path"], dst))
|
||||||
|
for side in sidecars(r["path"]):
|
||||||
|
rows.append((side, os.path.join(hold, os.path.relpath(side, library))))
|
||||||
|
shows_json.append({"show": show, "files": len(dominated),
|
||||||
|
"bytes": sum(r["size"] for r in dominated),
|
||||||
|
"episodes": len(dup_keys),
|
||||||
|
"sole-episode files left": sole,
|
||||||
|
"folders": sorted(staged_folders)})
|
||||||
|
n_ep, n_folder = len(dup_keys), len(staged_folders)
|
||||||
|
report.append(
|
||||||
|
f"{show}: stage {len(dominated)} files"
|
||||||
|
f" ({sum(r['size'] for r in dominated) / 2**30:.1f} GiB) — {n_ep}"
|
||||||
|
f" episode{n_ep != 1 and 's' or ''} held twice or more; {sole}"
|
||||||
|
f" episode{sole != 1 and 's' or ''} exist in one copy only, so"
|
||||||
|
f" {n_folder} folder{n_folder != 1 and 's' or ''} keep their files")
|
||||||
|
|
||||||
|
rows = sorted(set(rows))
|
||||||
|
dests = set()
|
||||||
|
for src, dst in rows:
|
||||||
|
if dst in dests:
|
||||||
|
audits.append(f"destination twice: {dst}")
|
||||||
|
dests.add(dst)
|
||||||
|
if os.path.exists(dst):
|
||||||
|
audits.append(f"hold target occupied: {dst}")
|
||||||
|
if not os.path.isfile(src):
|
||||||
|
audits.append(f"source missing: {src}")
|
||||||
|
|
||||||
|
unkeyed = collections.Counter(os.path.dirname(r["rel"]) for r in files
|
||||||
|
if not r["keys"])
|
||||||
|
if ARGS.json:
|
||||||
|
with open(ARGS.json, "w") as fh:
|
||||||
|
json.dump({"shows": shows_json,
|
||||||
|
"conflicts": [list(c) for c in conflicts],
|
||||||
|
"unresolved": [list(u) for u in unresolved],
|
||||||
|
"disagreement": {f: [t, d] for f, (t, d) in disagree.items()},
|
||||||
|
"unkeyed": dict(unkeyed)}, fh, indent=1)
|
||||||
|
|
||||||
|
for line in report:
|
||||||
|
print(line)
|
||||||
|
for a, b, why in conflicts:
|
||||||
|
print(f"CONFLICT (never merged): {a} | {b} — {why}")
|
||||||
|
for a, b, why in unresolved:
|
||||||
|
print(f"UNRESOLVED (tag it, then re-run): {a} | {b} — {why}")
|
||||||
|
for f, (tagged, extra) in sorted(disagree.items()):
|
||||||
|
print(f"DISAGREE (the tag wins): {f} — tag tvdb {tagged},"
|
||||||
|
f" derived tvdb {extra}")
|
||||||
|
if unkeyed:
|
||||||
|
print(f"unkeyed, never staged: {sum(unkeyed.values())} files in"
|
||||||
|
f" {len(unkeyed)} folders name no SxxEyy and have no --witness row;"
|
||||||
|
" most: " + ", ".join(f"{d} ({n})" for d, n in unkeyed.most_common(5)))
|
||||||
|
|
||||||
|
# The gate runs before the plan is written: a plan file that exists is a plan
|
||||||
|
# somebody will point the executor at, so a failed audit leaves no artifact.
|
||||||
|
if audits:
|
||||||
|
print(f"AUDIT FAILED ({len(audits)})", file=sys.stderr)
|
||||||
|
for a in audits[:10]:
|
||||||
|
print(f" {a}", file=sys.stderr)
|
||||||
|
print(f"refused: no plan written to {out}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
mkdirs = sorted({os.path.dirname(dst) for _s, dst in rows},
|
||||||
|
key=lambda p: (p.count(os.sep), p))
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
fh.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in mkdirs:
|
||||||
|
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for src, dst in rows:
|
||||||
|
fh.write(f"move\t{src}\t{dst}\ttvdup\n")
|
||||||
|
|
||||||
|
held = sum(os.path.getsize(s) for s, _d in rows if os.path.isfile(s))
|
||||||
|
print(f"plan: {len(rows)} moves ({held / 2**30:.1f} GiB) into"
|
||||||
|
f" {len(mkdirs)} hold folders -> {out}")
|
||||||
|
print("audit: every staged file has a survivor in the same show holding each of"
|
||||||
|
" its episodes at >= its size on disk, one destination each")
|
||||||
|
if rows:
|
||||||
|
root = os.path.dirname(library)
|
||||||
|
print(f"next: python3 scripts/media-apply.py apply --plan {out} --root"
|
||||||
|
f" {root} --journal {os.path.join(root, 'tv-dupes.jsonl')}"
|
||||||
|
" --metadata-view /var/lib/media-meta/view (dry run; add --apply)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
+18
-1
@@ -60,7 +60,7 @@ die() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
|
printf 'usage: %s plan|ensure-base|verify|install|reap|status [options]\n' "$0" >&2
|
||||||
exit 2
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -98,6 +98,22 @@ require_upper_base() {
|
|||||||
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# The SSD placement of the metadata layer is a human choice, so `install` refuses
|
||||||
|
# to invent a base directory: a mistyped --upper-base must not quietly create a
|
||||||
|
# tree on whatever filesystem the caller landed on. The deployer owns the
|
||||||
|
# documented location, so it calls this first — measured 2026-10-31, when a
|
||||||
|
# fresh-host `install` died with "put it on the SSD first" and nothing in git
|
||||||
|
# would have created it.
|
||||||
|
do_ensure_base() {
|
||||||
|
if [ -d "$upper_base" ]; then
|
||||||
|
printf 'base: %s exists\n' "$upper_base"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
install -d -m 0755 "$upper_base" 2>/dev/null \
|
||||||
|
|| die "refused: cannot create $upper_base — it lives outside the checkout, run this as root"
|
||||||
|
printf 'base: created %s\n' "$upper_base"
|
||||||
|
}
|
||||||
|
|
||||||
do_plan() {
|
do_plan() {
|
||||||
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
||||||
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
||||||
@@ -234,6 +250,7 @@ do_status() {
|
|||||||
case "$cmd" in
|
case "$cmd" in
|
||||||
plan) do_plan ;;
|
plan) do_plan ;;
|
||||||
verify) do_verify ;;
|
verify) do_verify ;;
|
||||||
|
ensure-base) do_ensure_base ;;
|
||||||
install) do_install ;;
|
install) do_install ;;
|
||||||
reap) do_reap ;;
|
reap) do_reap ;;
|
||||||
status) do_status ;;
|
status) do_status ;;
|
||||||
|
|||||||
@@ -334,7 +334,7 @@ for path in "${CHANGED[@]}"; do
|
|||||||
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
# Cutover-era dankserver quadlet units (romm trio, jellyfin, calibre-web,
|
||||||
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
# copyparty, plex, tautulli): the seam checks boot-persistence +
|
||||||
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
# v1-generator contract; deploy-flow seams guard the deployer routing.
|
||||||
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*)
|
dankserver/romm/*|dankserver/jellyfin/*|dankserver/calibre-web/*|dankserver/copyparty/*|dankserver/plex/*|dankserver/tautulli/*|dankserver/audiobookshelf/*)
|
||||||
add_test tests/dankserver_quadlet.bats
|
add_test tests/dankserver_quadlet.bats
|
||||||
add_test tests/deployment_flow.bats
|
add_test tests/deployment_flow.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
|
|||||||
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
description: "Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, first-run account, laws."
|
||||||
|
tags:
|
||||||
|
- dankserver
|
||||||
|
- audiobooks
|
||||||
|
- quadlet
|
||||||
|
timestamp: 2026-11-02
|
||||||
|
---
|
||||||
|
|
||||||
|
# Audiobookshelf
|
||||||
|
|
||||||
|
Audiobookshelf (ABS) serves the [[Entropy Volume]] audiobook library to browsers
|
||||||
|
and phones: it scans `/mnt/entropy/Audiobooks`, keeps its own SQLite library with
|
||||||
|
per-listener playback positions, and streams the audio. It is **not** a media
|
||||||
|
server — it shares nothing with [[Plex]] or [[Jellyfin]] and owns no files on the
|
||||||
|
array.
|
||||||
|
|
||||||
|
**Host:** Dankserver — **Unit:** `audiobookshelf.service` (rootless Quadlet, user
|
||||||
|
manager) — **Internal-only:** LAN address, no vhost, no DNS name.
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|-------|
|
||||||
|
| Unit file | `dankserver/audiobookshelf/quadlet/audiobookshelf.container` (the authority for every number here) |
|
||||||
|
| Image | `ghcr.io/advplyr/audiobookshelf` pinned by **amd64 manifest digest** — see Laws |
|
||||||
|
| Port | `Environment=PORT=8000` + `PublishPort=8000:8000` — web UI and API on one port |
|
||||||
|
| Base path | `Environment=ROUTER_BASE_PATH=/` — the URL is the plain `host:port`, no `/audiobookshelf` prefix |
|
||||||
|
| Network | `pasta` (standalone; publishes nothing to other containers) |
|
||||||
|
| Audio bind | `/mnt/entropy/Audiobooks:/audiobooks:ro` — read-only, by design |
|
||||||
|
| State | `/home/dank/Containers/audiobookshelf/config` → `/config`, `…/metadata` → `/metadata` (both NVMe) |
|
||||||
|
| Mount gate | `ExecStartPre=…/wait-for-mount.sh /mnt/entropy` |
|
||||||
|
| Health | `wget` against `/healthcheck` inside the container, 30 s / 3 retries |
|
||||||
|
| I/O share | `IOWeight=10` — a scan loses the array to every other tenant |
|
||||||
|
| Self-healing | `Restart=always`, `RestartSec=5`, `WantedBy=default.target` |
|
||||||
|
| Deploy | `./scripts/deploy.sh dankserver --service audiobookshelf` |
|
||||||
|
| Handler | `deploy_audiobookshelf` in `scripts/deploy-remote.sh` |
|
||||||
|
|
||||||
|
## First run
|
||||||
|
|
||||||
|
1. Open `http://192.168.0.235:8000` **from the LAN**. ABS accepts the initial
|
||||||
|
account creation only from a local-network address; a remote address gets a
|
||||||
|
refusal, not a form.
|
||||||
|
2. Create the owner account, then add one **Audi Book** library with folder
|
||||||
|
`/audiobooks`. The path is the in-container name of the read-only bind.
|
||||||
|
3. The first scan reads the whole tree once. It runs in the background — the
|
||||||
|
deploy gate is already green by then, and `IOWeight=10` keeps it off the
|
||||||
|
critical path.
|
||||||
|
|
||||||
|
Nothing in this repo holds an ABS credential: the account is created in the
|
||||||
|
browser and lives in the `config` bind on the SSD.
|
||||||
|
|
||||||
|
## Laws
|
||||||
|
|
||||||
|
- **Speak the image's env names, not the README's.** The server reads `PORT`,
|
||||||
|
`CONFIG_PATH`, `METADATA_PATH` and `ROUTER_BASE_PATH`; the `ABS_*` family printed
|
||||||
|
by the server's own README is the npm package's naming and is **inert** in this
|
||||||
|
image. Measured 2026-11-02 on the first deploy: with `ABS_PORT=8000` the server
|
||||||
|
listened on `:80` under base path `/audiobookshelf`, so `systemctl` said
|
||||||
|
`active (running)` while every probe of `:8000` was refused and the deploy gate
|
||||||
|
failed at 180 s. Two seconds with the pinned digest settles it:
|
||||||
|
`podman inspect --format '{{.Config.Env}}' <image>`.
|
||||||
|
- **The audio bind stays `:ro`.** ABS writes nothing that belongs beside the
|
||||||
|
audio: its library, covers, progress and sidecar metadata all land in
|
||||||
|
`/metadata` on the SSD. A writable bind would put stray writes on the spindle
|
||||||
|
and hand the array to a service with no reason to hold it. If per-book
|
||||||
|
`.opf`/`.nfo` sidecars on disk are ever wanted, the rollback is one line —
|
||||||
|
point the audio bind at the [[Media Metadata View]] the media servers already
|
||||||
|
use — but that view's upper layer exists for those two servers' sidecars, so a
|
||||||
|
third writer needs a reason first ([[Media Metadata View]]).
|
||||||
|
- **Do not bind the raw array without the mount gate.** In a rootless unit
|
||||||
|
`ConditionPathIsMountPoint=` is fail-*dead* and `RequiresMountsFor=` is inert,
|
||||||
|
so a missing array does not stop the start: podman invents an empty directory,
|
||||||
|
ABS indexes it, and an empty library becomes recorded truth. `wait-for-mount.sh`
|
||||||
|
is the only gate that works here — same law the media servers obey
|
||||||
|
([[Quadlet Lifecycle]]).
|
||||||
|
- **The `HealthCmd` must use `wget`.** The image is Alpine and ships `wget`, not
|
||||||
|
`curl`; a `curl` health command reads a perfectly healthy server as unhealthy
|
||||||
|
and `Restart=always` turns that into a loop.
|
||||||
|
- **The image pin is the amd64 manifest digest, not the index digest.**
|
||||||
|
`podman inspect` reports the platform manifest (`RepoDigests[0]`) for what was
|
||||||
|
pulled, which differs from the manifest-list digest `latest` points at. Pin the
|
||||||
|
one this host runs. `latest` is newer than the newest version tag: upstream
|
||||||
|
stopped publishing version tags after `2.19.0` while `latest` kept moving, so
|
||||||
|
"highest version tag" is stale here — read the index, don't guess from tags.
|
||||||
|
- **`IOWeight=10` is inert without the host drop-ins.** `dankserver/host/systemd/`
|
||||||
|
carries `user.slice.d/io-accounting.conf` and the `user@1000` delegate, and the
|
||||||
|
deployer installs both; without them systemd silently drops the weight
|
||||||
|
([[Quadlet Lifecycle]]).
|
||||||
|
- **No `systemctl --user enable`.** Generated Quadlet units cannot be enabled on
|
||||||
|
this host; `WantedBy=default.target` in the unit file is what carries the
|
||||||
|
service across a reboot ([[Quadlet Lifecycle]]).
|
||||||
|
- **LAN-only until Frank says otherwise.** `0.0.0.0:8000` with no reverse proxy
|
||||||
|
in front. Any public vhost requires the Anubis + Authentik chain first, and the
|
||||||
|
outpost proxy provider has to exist before the vhost flips (external exposure
|
||||||
|
policy in `AGENTS.md`).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
`./tests/run.sh` covers this unit through `tests/dankserver_quadlet.bats`
|
||||||
|
(boot persistence, the SSD state roots, the `:ro` array bind, the mount gate, the
|
||||||
|
`wget` health check) and `tests/media_overlay.bats` (only [[Plex]] and
|
||||||
|
[[Jellyfin]] may reference the metadata view).
|
||||||
@@ -66,3 +66,24 @@ exactly describes the year this bug survived.
|
|||||||
volume absent they start `active (healthy)` against empty directories, which is
|
volume absent they start `active (healthy)` against empty directories, which is
|
||||||
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
|
||||||
[[Storage Policy]] for what may be written here.
|
[[Storage Policy]] for what may be written here.
|
||||||
|
|
||||||
|
## Write doors
|
||||||
|
|
||||||
|
Something renamed movie folders on this array with no journal in any of the
|
||||||
|
`/mnt/entropy/*.jsonl` plans, no bash history, no cron, no timer, and no job
|
||||||
|
artifact. The census of things that can mutate it:
|
||||||
|
|
||||||
|
- **A local process** as `dank` or root. `scripts/media-apply.py` is the only
|
||||||
|
one that journals; anything else (a hand `mv`, another session's script) is
|
||||||
|
silent.
|
||||||
|
- **SFTP by a human.** GNOME Files on a workstation mounts the array through
|
||||||
|
`gvfsd-sftp`, which runs `sftp-server` on this host as `dank`. Renames and
|
||||||
|
moves are direct `rename()` calls: no shell, no history, no journal. A folder
|
||||||
|
rename bumps **only that directory's ctime** — its mtime and every child's
|
||||||
|
ctime stay at their old values, so the files look untouched while their paths
|
||||||
|
are gone. Diagnose it with `pgrep -af gvfsd-sftp` on the workstation and
|
||||||
|
`ss -tnp` for the peer.
|
||||||
|
- **Copyparty is not a door**: the bind is `ro` at the mount level, so no HTTP
|
||||||
|
or WebDAV write reaches the array regardless of account flags ([[Copyparty]]).
|
||||||
|
- **No SMB or NFS server exists here** — the `*arr` stack on FrankBook-Pro
|
||||||
|
cannot reach the array at all, so it cannot rename, import, or rescan it.
|
||||||
@@ -18,7 +18,7 @@ clients.
|
|||||||
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||||
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
||||||
|
|
||||||
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
|
## Facts (the repo quadlet — deployed as the live unit on 2026-10-31)
|
||||||
|
|
||||||
| Item | Value |
|
| Item | Value |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
||||||
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
||||||
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
||||||
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
|
| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**, live since the 2026-10-31 cutover: sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched |
|
||||||
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
||||||
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
||||||
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
||||||
@@ -37,26 +37,124 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
||||||
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
||||||
/var/lib/media-meta/view -> /data (git: the overlay view)
|
/var/lib/media-meta/view -> /data (git: the overlay view)
|
||||||
/mnt/entropy -> /data (live today: the array, read-only)
|
/mnt/entropy -> /data (the overlay's lower layer; also the rollback bind)
|
||||||
```
|
```
|
||||||
|
|
||||||
Library folders point at `/data` paths: stable across container recreation and
|
Library folders point at `/data` paths: stable across container recreation and
|
||||||
identical to the raw array layout below. Which side of that path the container
|
identical to the raw array layout below. Which side of that path the container
|
||||||
is on is the open question on this host — the installed unit binds the read-only
|
sees is the [[Media Metadata View]] overlay, live since the 2026-10-31 cutover:
|
||||||
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
|
`/var/lib/media-meta/view:/data`, read-write.
|
||||||
|
|
||||||
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
||||||
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
|
reason the view exists: `movie.nfo`, `tvshow.nfo` and `episode.nfo` written once
|
||||||
written once on the NVMe upper layer are files [[Plex]] can read too. While
|
on the NVMe upper layer are files [[Plex]] can read too — and since the
|
||||||
`/data` is bound read-only every item update logs the error below and no sidecar
|
2026-10-31 cutover that is what happens: Jellyfin is the fleet's only metadata
|
||||||
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
|
writer, and every sidecar it saves is an SSD byte the array never sees.
|
||||||
that raises the view.
|
**Artwork is not one of those sidecars.** No image saver is bound for any
|
||||||
|
library, so posters and fanart go to the internal store
|
||||||
|
(`config/metadata/library/<xx>/<item-id>/poster.jpg`), which [[Plex]] never
|
||||||
|
reads — movie art has never existed beside the media on either layer.
|
||||||
|
|
||||||
|
**Library coverage is the first thing to check when a title is missing.** The
|
||||||
|
four libraries are `movies → /data/Movies`, `tvshows → /data/TVShows`,
|
||||||
|
`music → /data/Music`, `boxsets → /config/data/collections`
|
||||||
|
(`GET /Library/VirtualFolders`). `/data/media-park` is in **none** of them, so
|
||||||
|
everything filed under it — the `_unsorted` release bundles and the `extras/`
|
||||||
|
folders the relayout wrote there — is invisible to Jellyfin by configuration,
|
||||||
|
not by fault: a recursive refresh answers 204, burns CPU for minutes and
|
||||||
|
indexes nothing, because it never looks there. Check coverage before re-scanning.
|
||||||
|
|
||||||
**Failure signature:** `System.IO.IOException: Read-only file system :
|
**Failure signature:** `System.IO.IOException: Read-only file system :
|
||||||
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
||||||
instead of the view. On Dankserver that is today's expected state — the view has
|
instead of the view — always a fault on this host since the 2026-10-31 cutover.
|
||||||
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
|
`scripts/media-view.sh status` says which: the quadlet drifted back to the raw
|
||||||
same line is a fault: the quadlet drifted or the mount died.
|
array, or the mount died.
|
||||||
|
|
||||||
|
**`EnableInternetProviders` false turns a library into a silent file index.**
|
||||||
|
Every refresh answers `204`, stores nothing and logs nothing — no provider ids,
|
||||||
|
no plot, no rating, no art — while the item count reads like a healthy scan. The
|
||||||
|
switch lives in `config/root/default/<Library>/options.xml`, Jellyfin 12's name
|
||||||
|
for the per-library option file: grepping for the old `library.xml` concludes "no
|
||||||
|
per-library options exist", and the database holds none, so the trail dies there.
|
||||||
|
Two more traps in the same file: `TypeOptions/ImageFetchers` is the *enabled* set,
|
||||||
|
so an empty `<ImageFetchers />` switches art off while `ImageFetcherOrder` stays
|
||||||
|
fully populated — order is not enablement — and the populated `<MetadataFetchers>`
|
||||||
|
built next to it makes the library look healthy.
|
||||||
|
|
||||||
|
**Live posture, measured 2026-11-02** from `GET /Library/VirtualFolders`. The DTO is
|
||||||
|
PascalCase — `Name`, `CollectionType`, `Locations`, `ItemId`, `RefreshStatus` — and a
|
||||||
|
census written against the 10.x camelCase keys reads `None` for everything, which
|
||||||
|
looks exactly like an unconfigured library.
|
||||||
|
|
||||||
|
| Library | `CollectionType` | `EnableInternetProviders` | `EnableRealtimeMonitor` |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `Movies` | movies | true (fixed 2026-11-02) | **false** |
|
||||||
|
| `Shows` | tvshows | true (fixed 2026-11-02) | true |
|
||||||
|
| `Music` | music | **false** | true |
|
||||||
|
| `Collections` | boxsets | **false** | false |
|
||||||
|
|
||||||
|
`Music` and `Collections` are silent file indexes in the sense above: a refresh answers
|
||||||
|
`204`, stores nothing, logs nothing. Any metadata or artwork work on either one
|
||||||
|
starts by flipping the switch, never by firing a scan. `Shows` sat in that set
|
||||||
|
until 2026-11-02; the door is `POST /Library/VirtualFolders/LibraryOptions` with
|
||||||
|
the library's own `ItemId` (below), and `scripts/jellyfin-providers.sh verify` is
|
||||||
|
the one command that says which libraries are still in it.
|
||||||
|
|
||||||
|
**The per-library option door on 12.x is `POST /Library/VirtualFolders/LibraryOptions`**
|
||||||
|
— `GET` there answers `405`, which is how the route announces itself. The 10.x doors
|
||||||
|
are gone: `/Library/VirtualFolders/Update` and `/Library/VirtualFolders/Refresh`
|
||||||
|
answer `404` on this build, so every recipe naming them fails outright.
|
||||||
|
|
||||||
|
**A library record can be a phantom.** A per-library option capture taken 2026-11-01
|
||||||
|
carried seven entries — `Shows2`, `Shows3`, `Music2` among them, each with
|
||||||
|
`Locations: []` and a live-looking `ItemId` — while `GET /Library/VirtualFolders` the
|
||||||
|
same day listed four. A numeric suffix on a library name is what a deleted and
|
||||||
|
recreated library leaves behind. The live set is the entries whose `Locations` are
|
||||||
|
non-empty; count those and never the records.
|
||||||
|
|
||||||
|
**`RemoteImages` is the call that separates a provider fault from a library
|
||||||
|
fault.** `GET /Items/{id}/RemoteImages?Type=Primary` returns the provider's real
|
||||||
|
poster list even with every automatic pass switched off — plugin, key and CDN all
|
||||||
|
proven by that one response — so a refresh that stores nothing beside it is a
|
||||||
|
library-switch problem, never a provider problem. It is also the delivery door:
|
||||||
|
`POST /Items/{id}/RemoteImages/Download` saves the chosen image to the internal
|
||||||
|
store on a build where the automatic image pass will not run at all — it is how
|
||||||
|
the Movies library got its art: 2,515 of 2,564 films carry a primary now, and the
|
||||||
|
rest have none in TMDb (shorts like `Astartes`, files whose names match nothing).
|
||||||
|
|
||||||
|
**The Download door's exact wire shape:** the list response carries the address in
|
||||||
|
`Url` — there is no `ImageUrl` field — and the POST takes camelCase query params,
|
||||||
|
`?type=Primary&imageUrl=<Url>`, answering `204`; the stored tag then appears in
|
||||||
|
`ImageTags`. PascalCase params, or an `imageUrl` left empty, answer `500`, which
|
||||||
|
reads as a server fault and is a field-name fault.
|
||||||
|
|
||||||
|
**`RemoteImages` answers empty for an item with no `ProviderIds`** — no provider id,
|
||||||
|
nothing to look up. Art coverage is capped by identification, not by the image door:
|
||||||
|
the items a campaign returns nothing for are the unidentified ones, and no image
|
||||||
|
retry will change that.
|
||||||
|
|
||||||
|
**A `Download` campaign over a library that is already dressed does nothing in
|
||||||
|
seconds, and that is the measurement proving identification is the only lever
|
||||||
|
left.** Run over all 753 series and 2,805 seasons of `Shows` on 2026-11-02 it
|
||||||
|
classified 3,558 items in 20 s: 2,908 already carried a primary, 624 have no
|
||||||
|
provider id at all, 26 have an id and no poster in TMDb — zero downloads. The
|
||||||
|
artless remainder is not an image-door problem: it is the unnamed long tail (clip
|
||||||
|
collections, `(Warhammer TV)`-shaped directories), and the lever is folder names
|
||||||
|
or an identify pass, never another retry loop.
|
||||||
|
|
||||||
|
**Series and season art is fetched; episode stills are not, deliberately.** Every
|
||||||
|
episode query is a fresh provider round-trip — nothing caches per episode — measured
|
||||||
|
at ~11 s per item beside ~2 s per series on the same host, with one hit in five in a
|
||||||
|
sample. Across 42k episodes that is days of provider traffic to dress a fifth of them,
|
||||||
|
and clients display the series and season art anyway. If episode stills are ever
|
||||||
|
ordered, the loop is the same door and its log makes the pass resumable.
|
||||||
|
|
||||||
|
**API shape on 12.x:** `/Items` and `/Items/{id}` need `UserId` when
|
||||||
|
authenticated by API key — without it the answer is `400 Error processing
|
||||||
|
request`, which reads exactly like an authentication failure. `MetadataRefreshMode`
|
||||||
|
accepts `Default` (and `Realtime`) only; `Full` and `ValidationFailure` are
|
||||||
|
rejected outright, and what forces the providers to re-run is
|
||||||
|
`ReplaceAllMetadata=true`.
|
||||||
|
|
||||||
## Ports & access
|
## Ports & access
|
||||||
|
|
||||||
@@ -164,9 +262,17 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
|||||||
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
||||||
`community_supported: false`), so it needs **no PIN** — the optional
|
`community_supported: false`), so it needs **no PIN** — the optional
|
||||||
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
||||||
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
|
- **TVDB is installed but is not a metadata agent**: TMDb still identifies
|
||||||
identifies everything. Enabling TVDB as a provider would re-fetch show
|
everything. Enabling TVDB as a provider would re-fetch show metadata across
|
||||||
metadata across the library, so it is a deliberate, separate change.
|
the library, so it is a deliberate, separate change.
|
||||||
|
- **A plugin owns its own config file, under its own root, and omits null
|
||||||
|
settings** — so the element an installer wants to fill may simply not exist,
|
||||||
|
and its root is `<PluginConfiguration>`, not `<Configuration>`. TVDB's file had
|
||||||
|
no `<ApiKey>` element at all, which is why `install` refused with
|
||||||
|
`no place to write <ApiKey>` from October until the anchor fix: the secret sat
|
||||||
|
valid in `provider-keys` the whole time, and `verify`'s `MISSING key` was this
|
||||||
|
fault, not a missing key. The writer now anchors on the file's own root and
|
||||||
|
still refuses by name when a file has none.
|
||||||
- **The deployer places the keys before it restarts jellyfin**
|
- **The deployer places the keys before it restarts jellyfin**
|
||||||
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
||||||
failing when `provider-keys` is absent, so a host without it still deploys.
|
failing when `provider-keys` is absent, so a host without it still deploys.
|
||||||
@@ -212,17 +318,44 @@ receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answe
|
|||||||
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
||||||
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
||||||
later `204` is accepted and then discarded — the status code stops meaning
|
later `204` is accepted and then discarded — the status code stops meaning
|
||||||
anything. The fix is `systemctl --user restart jellyfin.service` (never
|
anything. The fix is `systemctl --user restart jellyfin.service`, one POST after
|
||||||
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
|
|
||||||
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
||||||
response code.
|
response code.
|
||||||
|
|
||||||
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
|
**The door that scans is `POST /Library/Refresh` (204), and `ScheduledTasks` is
|
||||||
is the incremental pass, and inotify does not cross a rootless podman bind, so
|
the progress read.** A task that has never completed answers
|
||||||
folders created on the host stay invisible to it. The move is
|
`LastTaskResult=None, EndTimeUtc=None`, and that pair is what separates "never
|
||||||
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
|
ran" from "finished a moment ago" — the difference between a door that is dead
|
||||||
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
|
and one that merely answered `204`.
|
||||||
record's `RefreshStatus` is the progress read.
|
|
||||||
|
**The per-library `POST /Items/{id}/Refresh` queues nothing, and the provider
|
||||||
|
switch is not the reason.** Measured twice on 2026-11-02 against `Shows`: before
|
||||||
|
the flip and again after it, the call answered `204`, the journal wrote zero lines
|
||||||
|
over the following ten minutes, and `Scan Media Library` kept its never-run
|
||||||
|
signature; a service restart changes nothing. `POST /Library/Refresh` does run —
|
||||||
|
`204`, then `Running` inside 15 s at 33 % CPU — but it walks every library, so on
|
||||||
|
this build a library-scoped metadata pass has no working door and the global one
|
||||||
|
is the only lever.
|
||||||
|
|
||||||
|
**That endpoint takes its parameters from one source.** The documented query form
|
||||||
|
(empty body) and the JSON body form (no query) each parse; supplying both at once
|
||||||
|
answers `400` with an empty body whatever the values, and `ValidationFailure` is
|
||||||
|
rejected in the body while the query form accepts it.
|
||||||
|
|
||||||
|
**`RefreshStatus` on the `GET /Library/VirtualFolders` record is not a progress
|
||||||
|
read.** It stayed `Idle` through a scan that was verifiably running at 75 %. The
|
||||||
|
record's useful field is `ItemId`, and the live library set is still the entries
|
||||||
|
whose `Locations` are non-empty.
|
||||||
|
|
||||||
|
`inotify` does not cross a rootless podman bind, so folders created on the host
|
||||||
|
stay invisible until a pass like this runs; every pass that writes into a library
|
||||||
|
ends by firing one.
|
||||||
|
|
||||||
|
**`Movies` has no watcher at all.** `EnableRealtimeMonitor` is per library (table in
|
||||||
|
*Data layout*) and it is false for `Movies` and `Collections`. Nothing notices a
|
||||||
|
folder created under `/data/Movies`: it stays invisible until a recursive refresh
|
||||||
|
names it. So every pass that writes into a library ends by firing one, and "Jellyfin
|
||||||
|
does not see the file I just moved in" is the designed answer, not a fault to chase.
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ timestamp: 2026-10-31T00:00:00Z
|
|||||||
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
||||||
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
||||||
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
||||||
is the NARRATOR, both matched out of the files' own tags. Today one book is split
|
is the NARRATOR, both matched out of the files' own tags. [[Audiobookshelf]]
|
||||||
|
streams the same tree, read-only, and owns nothing in it. Today one book is split
|
||||||
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
||||||
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
||||||
re-files the tree so the property both servers' book scanners assume is true —
|
re-files the tree so the property both servers' book scanners assume is true —
|
||||||
@@ -36,6 +37,9 @@ python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
|
|||||||
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
|
||||||
# as in step 5 of the movie Procedure — from inside the container, never over
|
# as in step 5 of the movie Procedure — from inside the container, never over
|
||||||
# the host port.
|
# the host port.
|
||||||
|
# Audiobookshelf stores absolute paths, so the same refresh has to reach its
|
||||||
|
# library too ([[Audiobookshelf]]) — otherwise it keeps rows for folders that
|
||||||
|
# no longer exist.
|
||||||
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
||||||
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -0,0 +1,254 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Media Duplicate Prune
|
||||||
|
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
|
||||||
|
tags: [dankserver, media-server, storage, procedure]
|
||||||
|
timestamp: 2026-11-02T01:00:00Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Media Duplicate Prune
|
||||||
|
|
||||||
|
The movie library holds the same film twice in two different ways — the same file
|
||||||
|
downloaded again, or the same film grabbed a second time at another quality — and
|
||||||
|
[[Entropy Volume]] is permanently full, so both are worth finding. They need
|
||||||
|
different rules, and neither is decided by name similarity alone.
|
||||||
|
|
||||||
|
A series duplicates differently again: one show spread over several folders, so
|
||||||
|
neither the file nor the folder is the unit — see *A series is measured per
|
||||||
|
episode*.
|
||||||
|
|
||||||
|
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
|
||||||
|
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||||
|
|
||||||
|
## What counts
|
||||||
|
|
||||||
|
| Class | Test | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
|
||||||
|
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
|
||||||
|
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
|
||||||
|
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
|
||||||
|
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |
|
||||||
|
|
||||||
|
## The screen
|
||||||
|
|
||||||
|
The array cannot afford a full-content read, so identity is established in steps
|
||||||
|
and each step only narrows what the next one touches:
|
||||||
|
|
||||||
|
1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
|
||||||
|
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
|
||||||
|
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
|
||||||
|
evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
|
||||||
|
the better part of an hour; the sparse window cost 50 s, a file that merely
|
||||||
|
shares its head and tail cannot survive it, and the destination is a
|
||||||
|
recoverable hold rather than a deletion.
|
||||||
|
4. Durations come from `scripts/media-census.py probe` — the relayout's own
|
||||||
|
census. Do not hand-roll a second probe: it re-seeks the array for an answer
|
||||||
|
the repo tool already produces, and a probe is the one census step that costs
|
||||||
|
hours instead of seconds.
|
||||||
|
|
||||||
|
## Which copy survives
|
||||||
|
|
||||||
|
In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
|
||||||
|
folder (an extra is not a copy of the film, and extra names collide across films);
|
||||||
|
then the shallowest path; then the file whose name equals its `Title (year)`
|
||||||
|
folder; then the largest. For the same-work class the largest is the answer, and
|
||||||
|
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
|
||||||
|
outlive `Film.mkv` for no reason but a space sorting before a dot.
|
||||||
|
|
||||||
|
## A series is measured per episode
|
||||||
|
|
||||||
|
`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
|
||||||
|
executor's own format; it never moves anything itself.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||||
|
--hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
|
||||||
|
--meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
|
||||||
|
--ids /mnt/entropy/tv-tvdb-ids.tsv \
|
||||||
|
--out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json
|
||||||
|
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
|
||||||
|
name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
|
||||||
|
and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
|
||||||
|
match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
|
||||||
|
`BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
|
||||||
|
stripped to the bare top-level folder name — 616 folders, none carrying two
|
||||||
|
ids, written to `/mnt/entropy/tv-tvdb-ids.tsv`. Two folders with disjoint ids
|
||||||
|
are two shows however alike they read — *Avatar: The Last Airbender* (1999)
|
||||||
|
beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
|
||||||
|
had also titled *Millennium* — and a pair whose identity cannot be proven is
|
||||||
|
reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
|
||||||
|
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
|
||||||
|
fill an untagged folder and are **dropped** when they contradict its own
|
||||||
|
`[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
|
||||||
|
provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
|
||||||
|
[tvdbid-73838]`; unioned, those folders are one show and the special's bigger
|
||||||
|
files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
|
||||||
|
the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
|
||||||
|
staged file rests on a borrowed id.
|
||||||
|
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
|
||||||
|
2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
|
||||||
|
Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
|
||||||
|
season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
|
||||||
|
the pass would have staged about 340 GB of unique episodes; judged per episode it
|
||||||
|
staged 54 GiB of true copies.
|
||||||
|
- **A file is droppable only when every episode it carries is held elsewhere in the
|
||||||
|
same show at >= its size** (Frank's rule, applied per episode). A byte-identical
|
||||||
|
pair ties and the tie breaks by path so exactly one copy stages; a file holding
|
||||||
|
one unique episode stays whatever else it carries.
|
||||||
|
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
|
||||||
|
poster or a whole season, so key-less names are never staged and are counted in
|
||||||
|
the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
|
||||||
|
Where a packager named files by id, Jellyfin's own table is the witness: a
|
||||||
|
`path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
|
||||||
|
and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
|
||||||
|
rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
|
||||||
|
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
|
||||||
|
`pragma table_info` before trusting the query, because an empty witness silently
|
||||||
|
falls back to name parsing.
|
||||||
|
Seven clips in the library are still named only by number, and where Jellyfin
|
||||||
|
was asked its own `Name` for them is the number too, so no database holds a
|
||||||
|
better title — that residue is naming, not a task.
|
||||||
|
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
|
||||||
|
when both halves are held elsewhere, and it wins the survivor seat for both when
|
||||||
|
it is the largest carrier: the byte law compares files, and a name both servers
|
||||||
|
parse as a range stays indexed as two episodes.
|
||||||
|
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
|
||||||
|
artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
|
||||||
|
survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
|
||||||
|
copies were bare rips, and counting a hold's non-media files is the census that
|
||||||
|
tells you which case you are in.
|
||||||
|
- **The audit gates the plan file.** Every survivor is re-statted and every
|
||||||
|
destination re-checked before `--out` is written; on failure the report prints
|
||||||
|
`AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
|
||||||
|
- **Two folders that look duplicated and share zero episode keys are complementary,
|
||||||
|
not duplicated.** That is how the two pairs the census still refuses to merge were
|
||||||
|
settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
|
||||||
|
beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
|
||||||
|
Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
|
||||||
|
episode keys: 0` in both cases, so a tag would have freed nothing and cost a
|
||||||
|
rename with library consequences. Re-running the whole census with the id map over
|
||||||
|
all 753 top-level folders (606 already tagged) staged **0 files**: the library is
|
||||||
|
duplicate-free to the limit of provable identity, and what remains is naming.
|
||||||
|
|
||||||
|
## Guards
|
||||||
|
|
||||||
|
Each of these is a file that would have been destroyed without it:
|
||||||
|
|
||||||
|
- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
|
||||||
|
the finished file while qB still counts the download incomplete; moving it
|
||||||
|
breaks the re-check and re-downloads the film. Report it, leave it. Both
|
||||||
|
leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
|
||||||
|
not a prune decision.
|
||||||
|
- **Require a year, and keep digits in the title.** Stripping digits merged
|
||||||
|
*Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
|
||||||
|
happened to share a name merged.
|
||||||
|
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
|
||||||
|
is excluded: half a multi-disc release is not a copy of the other half.
|
||||||
|
- Media containers only. Grouping every extension buries the report in subtitle
|
||||||
|
and cover pairs — and a duration guard that appears to fire on 720 groups is
|
||||||
|
counting subtitles, not protecting films.
|
||||||
|
|
||||||
|
## The hold
|
||||||
|
|
||||||
|
Nothing is deleted. Every doomed copy moves into a dated hold with the same
|
||||||
|
executor the relayout uses, so any file comes back:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/dup-prune-similar-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl # dry
|
||||||
|
scripts/media-apply.py apply --plan … --root /mnt/entropy --journal … --apply
|
||||||
|
scripts/media-apply.py verify --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
|
||||||
|
scripts/media-apply.py sweep --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||||
|
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
|
||||||
|
every hold parent), and the planner re-checks the two things that go stale between
|
||||||
|
analysis and apply: a file that has moved since the census, and a group whose
|
||||||
|
keeper is missing — either one drops the whole group rather than half-applying it.
|
||||||
|
|
||||||
|
- **A hold on the same volume frees nothing.** The purge is a separate, approved
|
||||||
|
act and the only thing that moves the free-space number; `df` does not budge
|
||||||
|
when 1 TiB moves sideways.
|
||||||
|
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
|
||||||
|
ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
|
||||||
|
era, not this pass — read the mtimes before blaming the prune.
|
||||||
|
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
|
||||||
|
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
|
||||||
|
- The 2026-11-02 TV pass holds 220 files (54 GiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
|
||||||
|
journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
|
||||||
|
same day's TVShows detritus pass holds 109 files (552 KiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
|
||||||
|
`tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges
|
||||||
|
pending Frank's word, beside the 2026-11-01 hold above. All four journals and
|
||||||
|
plans live at the `entropy` root, not `/tmp`.
|
||||||
|
- **Re-verify each survivor by size, not by the recorded path, before purging.**
|
||||||
|
Detection artifacts and plans key on paths, and a folder renamed by an
|
||||||
|
interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look
|
||||||
|
deleted while it sits untouched under the new name. The 2026-11-01 audit hit
|
||||||
|
three such groups: every keeper was present, same bytes, different folder. A
|
||||||
|
purge decided from the plan's paths alone would have reported survivors gone.
|
||||||
|
|
||||||
|
## After the pass
|
||||||
|
|
||||||
|
Both servers still hold rows for files that are now in the hold, and neither
|
||||||
|
notices: the media bind is the [[Media Metadata View]], and array changes do not
|
||||||
|
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
|
||||||
|
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
|
||||||
|
are owned by [[Jellyfin]] and [[Plex]].
|
||||||
|
|
||||||
|
The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user
|
||||||
|
stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a
|
||||||
|
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
|
||||||
|
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
|
||||||
|
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
|
||||||
|
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
|
||||||
|
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
|
||||||
|
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.
|
||||||
|
|
||||||
|
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
|
||||||
|
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
|
||||||
|
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
|
||||||
|
from folders whose whole subtree held no media, so it creates no stale rows and
|
||||||
|
needs no scan. The guards that hold a folder back are the finding: external
|
||||||
|
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
|
||||||
|
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
|
||||||
|
extension the census does not recognise is a reason to stop, not to sweep.
|
||||||
|
|
||||||
|
A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
|
||||||
|
file is a manual match — a human decision about what the folder is — and detritus
|
||||||
|
rules do not overrule one.
|
||||||
|
|
||||||
|
`scripts/media-verify.py` reads differently for a prune than for a relayout:
|
||||||
|
`landed: 0` is the **correct** end state (no server may index the hold), so the
|
||||||
|
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
|
||||||
|
from the database are the pass succeeding.
|
||||||
|
|
||||||
|
One residue is normal, so do not chase it: a section pass rewrites the feature rows
|
||||||
|
first, and rows for clips in a film's `other/` or `extras/` subfolder can still name
|
||||||
|
a held path after the section's `refreshing` flag is back to `0` — and a second
|
||||||
|
section pass does not clear them either, so do not run one hoping for `0`. Beside
|
||||||
|
`REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch
|
||||||
|
the cache, the database, or the hold.
|
||||||
|
|
||||||
|
## Failure signatures
|
||||||
|
|
||||||
|
| Symptom | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
|
||||||
|
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
|
||||||
|
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
|
||||||
|
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |
|
||||||
|
|
||||||
|
## Links
|
||||||
|
|
||||||
|
- [[Media Relayout]] — the census tools, the executor, the journal format.
|
||||||
|
- [[Media Metadata View]] — why neither server sees the change without a scan.
|
||||||
|
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
|
||||||
|
- [[Entropy Volume]] — the array that is always full.
|
||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Media Metadata View
|
title: Media Metadata View
|
||||||
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||||
tags: [dankserver, media-server, jellyfin, plex, storage]
|
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||||
timestamp: 2026-10-30T00:00:00Z
|
timestamp: 2026-11-02T01:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Media Metadata View
|
# Media Metadata View
|
||||||
@@ -20,18 +20,40 @@ The mount point is not a volume and holds nothing while unmounted. It sits insid
|
|||||||
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
||||||
next to the array.
|
next to the array.
|
||||||
|
|
||||||
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
|
**Deployed on Dankserver — 2026-10-31, both servers.** `plex` binds
|
||||||
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
|
`/var/lib/media-meta/view:/mnt/entropy:ro` and `jellyfin` binds
|
||||||
`/var/lib/media-meta` does not exist, and the installed quadlets under
|
`/var/lib/media-meta/view:/data:rw`; `findmnt` reports the overlay with
|
||||||
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
|
`lowerdir=/mnt/entropy upperdir=/var/lib/media-meta/upper`, and each container
|
||||||
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
|
was proven to resolve a real media file through it — `podman exec <svc> ls -la
|
||||||
is the shape git describes, not the shape the servers run. The cutover is an
|
<its data path>` is the whole proof, because both servers keep the paths their
|
||||||
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
|
databases recorded. Two service-scoped deploys, each gated by
|
||||||
`ensure_media_view` (`install` then `verify`) before it restarts the container,
|
`media-view.sh verify` and the Playwright suite.
|
||||||
so it must not be fired through the middle of a library scan. Until that deploy
|
|
||||||
lands there is no upper layer, so no relayout can leave ghost metadata and
|
Re-verified 2026-11-01: `mount` shows the overlay on
|
||||||
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
|
`/var/lib/media-meta/view` (`lowerdir=/mnt/entropy`,
|
||||||
view really is a mountpoint.
|
`upperdir=/var/lib/media-meta/upper`), and the installed quadlets under
|
||||||
|
`~/.config/containers/systemd/` carry the view binds:
|
||||||
|
`Volume=/var/lib/media-meta/view:/data` for [[Jellyfin]],
|
||||||
|
`Volume=/var/lib/media-meta/view:/mnt/entropy:ro` for [[Plex]]. Prove it in that
|
||||||
|
order: `scripts/media-view.sh verify` (the deploy gate), then
|
||||||
|
`grep -n Volume ~/.config/containers/systemd/{jellyfin,plex}.container`.
|
||||||
|
|
||||||
|
Moving a bind means restarting the server, and **a restart discards that
|
||||||
|
server's in-flight library scan** — Jellyfin's 7 h 25 m walk ended
|
||||||
|
`"Status": "Aborted"` in `data/ScheduledTasks/<task-id>.js` and nothing resumes
|
||||||
|
it. Renames made on the array while the view was unmounted are invisible to both
|
||||||
|
watchers too, so the additive index has to be asked for by hand; the door is an
|
||||||
|
API key, and the row in the failure table below says what to check when it
|
||||||
|
refuses.
|
||||||
|
|
||||||
|
The cutover itself is an ordinary deploy: `deploy.sh dankserver --service
|
||||||
|
jellyfin|plex` runs `ensure_media_view` (`install` then `verify`) before it
|
||||||
|
restarts the container, so it must not be fired through the middle of a library
|
||||||
|
scan. Two consequences are now live rather than theoretical: inotify events for
|
||||||
|
array writes do not cross the merge, so any relayout or prune needs a manual
|
||||||
|
library refresh; and a move can leave ghost sidecars above, so `reap` runs after
|
||||||
|
a relayout — a pass that moves only media files with no sidecars next to them
|
||||||
|
leaves the upper layer untouched, and `reap` correctly finds nothing.
|
||||||
|
|
||||||
## Shape
|
## Shape
|
||||||
|
|
||||||
@@ -112,6 +134,19 @@ the defaults are the host values above.
|
|||||||
upper layer*, not for changes on the array underneath. After any relayout or
|
upper layer*, not for changes on the array underneath. After any relayout or
|
||||||
externally-written metadata, trigger a library refresh — see
|
externally-written metadata, trigger a library refresh — see
|
||||||
[[Media Relayout]] for the refresh procedure.
|
[[Media Relayout]] for the refresh procedure.
|
||||||
|
- **`stat` through the view can outlive the file.** Measured 2026-11-02 while a TV
|
||||||
|
dedupe pass moved folders on the raw path: the merged tree still answered `stat`
|
||||||
|
at the old coordinates — link count `0`, and `ls` of the same parent no longer
|
||||||
|
listed the name. Overlayfs keeps a dentry for a lower file that was moved
|
||||||
|
underneath it, so a script that tests existence with `[ -e ]` against the view can
|
||||||
|
believe a file its own `readdir` denies. Prove existence by listing the parent.
|
||||||
|
The media servers walk by `readdir`, so ghosts never reach the library — this bites
|
||||||
|
a tool, never a user, which is why no server symptom accompanies it.
|
||||||
|
- **Tools that move or measure media walk the raw array, never the view** — every
|
||||||
|
relayout, prune and detritus pass, and every journal it writes. The view is for
|
||||||
|
Jellyfin's writes and Plex's reads only. One deliberate exception: `tvshow.nfo`
|
||||||
|
sidecars live in the upper layer, so a tool reading show identity points at the
|
||||||
|
view (`media-tvdupes.py --meta /var/lib/media-meta/view/TVShows`).
|
||||||
|
|
||||||
## Failure signatures
|
## Failure signatures
|
||||||
|
|
||||||
@@ -122,7 +157,11 @@ the defaults are the host values above.
|
|||||||
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
||||||
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
||||||
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
||||||
|
| `install` prints `refused: upper base /var/lib/media-meta does not exist (put it on the SSD first)` | A fresh host. `install` deliberately never creates the base — a mistyped `--upper-base` must not build a metadata tree on the wrong filesystem — so `scripts/media-view.sh ensure-base` makes the documented one, which the deployer calls first. |
|
||||||
|
| A media server's scan record reads `"Status": "Aborted"` | Its container restarted (a cutover, an image bump, an OOM) and the walk was discarded; systemd's `Restart=always` restarts the *server*, never the scan. Trigger a refresh — see [[Media Relayout]]. |
|
||||||
|
| The Jellyfin API answers 401 to every token header | Almost always the header **form**, not the key: 12.4 authenticates only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for every key, valid or not — and one collapsed backslash in the curl config produces the same 401. `[[Jellyfin]]` owns the block that works. Before concluding anything, prove the key against the server's own table: `select count(*) from ApiKeys where AccessToken = ?` on `jellyfin.db` opened `mode=ro&immutable=1`. |
|
||||||
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
||||||
|
| Stale dentry through the view: `stat` succeeds with link count `0` while the parent's `ls` omits the name | The lower file was moved on the raw path after the view cached its dentry. Nothing is lost; existence through the view is proved by the parent listing. |
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Media Relayout
|
title: Media Relayout
|
||||||
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||||
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||||
timestamp: 2026-10-30T00:00:00Z
|
timestamp: 2026-11-01T01:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Media Relayout
|
# Media Relayout
|
||||||
@@ -138,6 +138,8 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
|||||||
no `Accept` and client-identifier header (see the scan block above for the form
|
no `Accept` and client-identifier header (see the scan block above for the form
|
||||||
that works).
|
that works).
|
||||||
|
|
||||||
|
A duplicate prune is the same census and the same proof, with its own keep rules — films by title and duration, a series per episode: [[Media Duplicate Prune]].
|
||||||
|
|
||||||
## TV in the movie library
|
## TV in the movie library
|
||||||
|
|
||||||
A box set that arrives in the movie share stays there until someone moves it:
|
A box set that arrives in the movie share stays there until someone moves it:
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||||
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — live since the 2026-10-31 cutover, so the PMS database's recorded paths keep resolving with zero re-scan |
|
||||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||||
@@ -40,7 +40,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
The migration moved zero bytes: the container bind-mounts the native data
|
The migration moved zero bytes: the container bind-mounts the native data
|
||||||
root where it already lived, and the media archive re-mounts at
|
root where it already lived, and the media archive re-mounts at
|
||||||
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||||
(the array today; the overlay view in git has never been deployed), so every
|
(the array until the 2026-10-31 view cutover, the overlay since), so every
|
||||||
absolute library path in the
|
absolute library path in the
|
||||||
PMS database keeps resolving. The one metadata change: rootless podman
|
PMS database keeps resolving. The one metadata change: rootless podman
|
||||||
cannot map the real host uid 997, so the fleet model applies — container
|
cannot map the real host uid 997, so the fleet model applies — container
|
||||||
@@ -86,7 +86,22 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
|||||||
4. **Cold container boot is slow.** PMS pre-binds 32400 and empty-replies
|
4. **Cold container boot is slow.** PMS pre-binds 32400 and empty-replies
|
||||||
while the HTTP layer finishes loading; the first boot of a years-old
|
while the HTTP layer finishes loading; the first boot of a years-old
|
||||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||||
600 s before failing; later boots are much faster (warm page cache).
|
600 s before failing; later boots are much faster (warm page cache). A
|
||||||
|
`systemctl --user start` issued after an `unmask` is a job in the manager,
|
||||||
|
not in the shell that issued it: it lands minutes later and survives that
|
||||||
|
client dying, so a wrapper script that looks hung is usually only waiting —
|
||||||
|
read `is-active` and `podman ps` before declaring a fault, and give every
|
||||||
|
step of such a script its own `timeout`.
|
||||||
|
5. **`32400` answers TLS only to remote clients.** PMS requires secure
|
||||||
|
connections, so `curl http://192.168.0.235:32400/identity` from any other
|
||||||
|
machine closes without a single byte (curl exit 52) — that reads as a dead
|
||||||
|
server beside a healthy one, and it is how a post-cutover health check gets
|
||||||
|
misread. Probe with `curl -k https://192.168.0.235:32400/identity`. Two
|
||||||
|
consequences: `plex.franksdank.cloud` works **only** because its `transport
|
||||||
|
http` block carries `tls_insecure_skip_verify` (that option makes Caddy speak
|
||||||
|
TLS to PMS's self-signed cert — delete it and the vhost 502s), and the scan
|
||||||
|
block below rides `http://127.0.0.1` **inside** the container, where insecure
|
||||||
|
connections are allowed.
|
||||||
|
|
||||||
## Triggering a scan
|
## Triggering a scan
|
||||||
|
|
||||||
@@ -121,10 +136,13 @@ shred -u /tmp/px.tok
|
|||||||
```
|
```
|
||||||
|
|
||||||
Every section answers `204` — or `200` with a Directory body when
|
Every section answers `204` — or `200` with a Directory body when
|
||||||
`Accept: application/json` is sent. Inside the quoted `printf` format a
|
`Accept: application/json` is sent. A refresh fired while PMS is still warming
|
||||||
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
|
answers `000` instead (the empty reply of gotcha 4, arriving mid-request): retry
|
||||||
one glued line and one stray backslash is a config file curl rejects, and the
|
until it answers and never restart the service over it. Inside the quoted
|
||||||
failure prints as `option -K: is unknown`, never as a bad config line.
|
`printf` format a backslash-newline is LITERAL, not a continuation, so `mk`
|
||||||
|
writes two `printf`s: one glued line and one stray backslash is a config file
|
||||||
|
curl rejects, and the failure prints as `option -K: is unknown`, never as a bad
|
||||||
|
config line.
|
||||||
|
|
||||||
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
||||||
replays the run's journal against the database — the only read that
|
replays the run's journal against the database — the only read that
|
||||||
|
|||||||
@@ -107,6 +107,12 @@ job.sh watch # refreshing status table
|
|||||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
||||||
every failure disappeared once the script was a file.
|
every failure disappeared once the script was a file.
|
||||||
|
- **A script fed over stdin owns that stdin.** `ssh host 'bash -s' <<'EOS'` hands
|
||||||
|
the script itself to stdin, so any child that reads stdin — `podman exec -i`,
|
||||||
|
`ssh -t`, a bare `read` — swallows everything after it and the remainder
|
||||||
|
vanishes without a word. It reads exactly like a hang: the commands before the
|
||||||
|
child print, the ones after never existed. Drop `-i`, or give every child
|
||||||
|
`< /dev/null`.
|
||||||
|
|
||||||
## Verify before trusting it (3am law)
|
## Verify before trusting it (3am law)
|
||||||
|
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ is a locator: one row, one clause, no numbers.
|
|||||||
|
|
||||||
| Page | Owns |
|
| Page | Owns |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
| [[Audiobookshelf]] | Audiobookshelf on Dankserver: quadlet shape, the read-only array bind, state on the SSD, first-run account, laws. |
|
||||||
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
|
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
|
||||||
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
|
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
|
||||||
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
||||||
@@ -52,6 +53,7 @@ is a locator: one row, one clause, no numbers.
|
|||||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||||
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
||||||
|
| [[Media Duplicate Prune]] | Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge. |
|
||||||
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||||
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
||||||
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96"><polygon points="48,6 86,27 86,69 48,90 10,69 10,27" fill="#1c1300" stroke="#3fc1c9" stroke-width="3"/><text x="48" y="60" font-family="monospace" font-size="30" font-weight="bold" fill="#3fc1c9" text-anchor="middle">AB</text></svg>
|
||||||
|
After Width: | Height: | Size: 293 B |
@@ -511,6 +511,19 @@
|
|||||||
"showLink": true,
|
"showLink": true,
|
||||||
"art": "/img/apps/mono-tt.svg"
|
"art": "/img/apps/mono-tt.svg"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "Audiobookshelf",
|
||||||
|
"category": "admin",
|
||||||
|
"section": "Media Library Apps",
|
||||||
|
"description": "Audiobook server for the entropy library: its own SQLite library, its own login, per-listener positions.",
|
||||||
|
"url": "http://192.168.0.235:8000",
|
||||||
|
"docs": "",
|
||||||
|
"probe": "192.168.0.235:8000",
|
||||||
|
"method": "http",
|
||||||
|
"wip": false,
|
||||||
|
"showLink": true,
|
||||||
|
"art": "/img/apps/mono-ab.svg"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "Postgres",
|
"name": "Postgres",
|
||||||
"category": "admin",
|
"category": "admin",
|
||||||
|
|||||||
@@ -127,3 +127,63 @@ load helpers
|
|||||||
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
|
grep -q "dankserver/host/systemd/$f" "$REPO_ROOT/scripts/deploy-remote.sh" || return 1
|
||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf unit exists, is boot-persistent and self-healing" {
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
[ -f "$unit" ] || return 1
|
||||||
|
grep -q '^WantedBy=default.target$' "$unit" || return 1
|
||||||
|
grep -q '^Restart=always$' "$unit" || return 1
|
||||||
|
grep -q '^RestartSec=' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf keeps every byte it writes on the SSD" {
|
||||||
|
# The point of the unit: its library database, accounts, progress and cached
|
||||||
|
# covers are all server state, and server state belongs on the NVMe. A bind
|
||||||
|
# of /mnt/entropy that is not :ro would silently put writes on the spindle.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^Volume=/home/dank/Containers/audiobookshelf/config:/config$' "$unit" || return 1
|
||||||
|
grep -q '^Volume=/home/dank/Containers/audiobookshelf/metadata:/metadata$' "$unit" || return 1
|
||||||
|
grep -q '^Volume=/mnt/entropy/Audiobooks:/audiobooks:ro$' "$unit" || return 1
|
||||||
|
# The overlay view is reserved for the two media servers that write sidecars
|
||||||
|
# (tests/media_overlay.bats pins that set); a reader must not join it.
|
||||||
|
! grep -q 'media-meta/view' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf refuses to start against an unmounted array" {
|
||||||
|
# Without the gate podman invents an empty bind source and the first scan
|
||||||
|
# records an empty library as truth.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^ExecStartPre=.*/wait-for-mount.sh /mnt/entropy$' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf health check names a binary its image actually ships" {
|
||||||
|
# The image is Alpine: wget yes, curl no. A curl HealthCmd reads the
|
||||||
|
# container as unhealthy while it is serving perfectly.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^HealthCmd=wget ' "$unit" || return 1
|
||||||
|
! grep -q '^HealthCmd=curl' "$unit" || return 1
|
||||||
|
grep -q '^PublishPort=8000:8000$' "$unit" || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the audiobookshelf deployer creates both state roots before it starts" {
|
||||||
|
body="$(sed -n '/^deploy_audiobookshelf()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
mkdir_line="$(printf '%s\n' "$body" | grep -n 'mkdir -p' | head -1 | cut -d: -f1)"
|
||||||
|
start_line="$(printf '%s\n' "$body" | grep -n 'systemctl --user restart' | head -1 | cut -d: -f1)"
|
||||||
|
[ -n "$mkdir_line" ] && [ -n "$start_line" ] || return 1
|
||||||
|
[ "$mkdir_line" -lt "$start_line" ] || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'audiobookshelf/config' || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'audiobookshelf/metadata' || return 1
|
||||||
|
printf '%s\n' "$body" | grep -q 'wait_for_health audiobookshelf' || return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "audiobookshelf sets its port and base path with the names the image reads" {
|
||||||
|
# Measured 2026-11-02 on the first deploy: ABS_PORT / ABS_CONFIG_DIR are the npm
|
||||||
|
# package's names and this image ignores them, so the server came up on :80
|
||||||
|
# under base path /audiobookshelf and the deploy gate timed out beside a unit
|
||||||
|
# that was active (running). PORT and ROUTER_BASE_PATH are what the image's own
|
||||||
|
# ENV block and the server source actually use.
|
||||||
|
unit="$REPO_ROOT/dankserver/audiobookshelf/quadlet/audiobookshelf.container"
|
||||||
|
grep -q '^Environment=PORT=8000$' "$unit" || return 1
|
||||||
|
grep -q '^Environment=ROUTER_BASE_PATH=/$' "$unit" || return 1
|
||||||
|
! grep -qE '^Environment=ABS_' "$unit" || return 1
|
||||||
|
}
|
||||||
@@ -28,15 +28,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
|||||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the retired ik chat template plans nothing, and its archive is still here" {
|
@test "the retired ik chat template plans nothing" {
|
||||||
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
||||||
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
||||||
# llama.cpp frank-zealot runtime serves the template the GGUF itself
|
# llama.cpp frank-zealot runtime serves the template the GGUF itself
|
||||||
# carries, which is the one with reasoning_effort, preserve_thinking and
|
# carries, which is the one with reasoning_effort, preserve_thinking and
|
||||||
# the tool-call grammar. Editing the archive must deploy nothing -- and the
|
# the tool-call grammar. The planner's rule is a PATH rule, so an edit at
|
||||||
# archive must still be present, so this cannot pass by the file having
|
# that path deploys nothing no matter where the bytes currently sit: the seam
|
||||||
# vanished.
|
# deliberately does not assert the archive exists. Trash is a purgeable hold
|
||||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp" ]
|
# (untracked in #111, and the deletion policy says the purger deletes it), so
|
||||||
|
# pinning it fails every fresh checkout and proves nothing about planning.
|
||||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
[ -z "$output" ]
|
[ -z "$output" ]
|
||||||
|
|||||||
@@ -24,6 +24,9 @@
|
|||||||
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
||||||
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
||||||
# installer to jellyfin.
|
# installer to jellyfin.
|
||||||
|
# 10. verify refuses a library that is a SILENT FILE INDEX — internet providers
|
||||||
|
# switched off, or a type whose ImageFetchers is empty while its
|
||||||
|
# MetadataFetchers list is populated (the 2026-11-01 missing-posters fault).
|
||||||
|
|
||||||
load helpers
|
load helpers
|
||||||
|
|
||||||
@@ -59,6 +62,32 @@ setup() {
|
|||||||
TRASH="$BATS_TEST_TMPDIR/trash"
|
TRASH="$BATS_TEST_TMPDIR/trash"
|
||||||
SEC="$BATS_TEST_TMPDIR/secrets"
|
SEC="$BATS_TEST_TMPDIR/secrets"
|
||||||
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
||||||
|
# The per-library option file Jellyfin 12 owns (it was library.xml). Conforming
|
||||||
|
# by default so the pre-existing verify tests keep testing what they tested.
|
||||||
|
mkdir -p "$JF/root/default/Movies"
|
||||||
|
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<EnableInternetProviders>true</EnableInternetProviders>
|
||||||
|
<MetadataSavers>
|
||||||
|
<string>Nfo</string>
|
||||||
|
</MetadataSavers>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>Movie</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetchers>
|
||||||
|
<ImageFetcherOrder>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetcherOrder>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
||||||
<?xml version="1.0" encoding="utf-8"?>
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
@@ -270,6 +299,69 @@ XML
|
|||||||
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "verify refuses a library whose internet providers are switched off" {
|
||||||
|
# The whole library becomes a file index: refreshes answer 204 and store
|
||||||
|
# nothing, and the item count still looks like a healthy scan.
|
||||||
|
prov install >/dev/null
|
||||||
|
sed -i 's#<EnableInternetProviders>true</EnableInternetProviders>#<EnableInternetProviders>false</EnableInternetProviders>#' \
|
||||||
|
"$JF/root/default/Movies/options.xml"
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'SILENT INDEX Movies'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'EnableInternetProviders'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "verify refuses a type with no image fetchers while metadata fetchers are named" {
|
||||||
|
prov install >/dev/null
|
||||||
|
cat >"$JF/root/default/Movies/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions>
|
||||||
|
<EnableInternetProviders>true</EnableInternetProviders>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>Movie</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers />
|
||||||
|
<ImageFetcherOrder>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetcherOrder>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'ImageFetchers empty'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an older options.xml that never carried the switch is not refused" {
|
||||||
|
# BoxSets on this host were written before the property existed; absence is
|
||||||
|
# the vendor default, not a switched-off library, so it must not fail.
|
||||||
|
prov install >/dev/null
|
||||||
|
mkdir -p "$JF/root/default/Collections"
|
||||||
|
cat >"$JF/root/default/Collections/options.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<LibraryOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<TypeOptions>
|
||||||
|
<Type>BoxSet</Type>
|
||||||
|
<MetadataFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</MetadataFetchers>
|
||||||
|
<ImageFetchers>
|
||||||
|
<string>TheMovieDb</string>
|
||||||
|
</ImageFetchers>
|
||||||
|
</TypeOptions>
|
||||||
|
</TypeOptions>
|
||||||
|
</LibraryOptions>
|
||||||
|
XML
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF '= Collections'
|
||||||
|
}
|
||||||
|
|
||||||
@test "no plugin configuration XML is tracked in this repo" {
|
@test "no plugin configuration XML is tracked in this repo" {
|
||||||
# These files hold provider keys and are owned by the running container;
|
# These files hold provider keys and are owned by the running container;
|
||||||
# tracking them would put credentials in git and guarantee drift.
|
# tracking them would put credentials in git and guarantee drift.
|
||||||
|
|||||||
@@ -10,7 +10,8 @@
|
|||||||
# 4. both media units bind the VIEW and refuse to start without it;
|
# 4. both media units bind the VIEW and refuse to start without it;
|
||||||
# 5. only those two units touch the view — the LLM and library tenants keep the
|
# 5. only those two units touch the view — the LLM and library tenants keep the
|
||||||
# raw volume, so the blast radius is two containers;
|
# raw volume, so the blast radius is two containers;
|
||||||
# 6. the deployer brings the view up BEFORE it restarts either server;
|
# 6. the deployer creates the upper base and brings the view up BEFORE it
|
||||||
|
# restarts either server;
|
||||||
# 7. the planner routes a change to these files to both media services.
|
# 7. the planner routes a change to these files to both media services.
|
||||||
|
|
||||||
load helpers
|
load helpers
|
||||||
@@ -110,6 +111,30 @@ UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
|||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "ensure-base creates a base directory that install refuses to invent" {
|
||||||
|
base="$BATS_TEST_TMPDIR/fresh-base"
|
||||||
|
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -d "$base" ]
|
||||||
|
# The deployer runs this on every media deploy, so a second call must be a
|
||||||
|
# no-op and must say so rather than recreate anything.
|
||||||
|
run bash "$VIEW" ensure-base --upper-base "$base"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "${lines[0]}" == *"exists"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the media deployer creates the upper base before it installs the view" {
|
||||||
|
body="$(sed -n '/^ensure_media_view()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
[ -n "$body" ]
|
||||||
|
base="$(printf '%s\n' "$body" | grep -n 'media-view.sh ensure-base' | head -1 | cut -d: -f1)"
|
||||||
|
inst="$(printf '%s\n' "$body" | grep -n 'media-view.sh install' | head -1 | cut -d: -f1)"
|
||||||
|
# Order is the whole test: install refuses a missing base, so an installer
|
||||||
|
# that runs first dies on a rebuilt host (measured 2026-10-31).
|
||||||
|
[ -n "$base" ]
|
||||||
|
[ -n "$inst" ]
|
||||||
|
[ "$base" -lt "$inst" ]
|
||||||
|
}
|
||||||
|
|
||||||
@test "the planner routes the view files to both media services" {
|
@test "the planner routes the view files to both media services" {
|
||||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
||||||
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
||||||
|
|||||||
Executable
+215
@@ -0,0 +1,215 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/media-tvdupes.py — the episode-level duplicate census for the TV
|
||||||
|
# library.
|
||||||
|
#
|
||||||
|
# The failure this seam exists to prevent is the one that almost cost real
|
||||||
|
# content: two folders of one show look like copies, and a folder-level "keep
|
||||||
|
# the bigger folder" deletes the half of the series only the smaller folder
|
||||||
|
# holds (the 2026-11-02 census found 10 such complementary pairs). So every
|
||||||
|
# assertion here is about WHICH files land in the plan and WHICH the tool
|
||||||
|
# refuses to touch — never about its internals. Everything runs on a synthetic
|
||||||
|
# library in the test's own tmpdir: no media server, no network, no array.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
TOOL="$REPO_ROOT/scripts/media-tvdupes.py"
|
||||||
|
|
||||||
|
# at PATH SIZE — a file of deterministic bytes.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
# sources FILE — the sources the plan moves, one per line.
|
||||||
|
sources() { awk -F'\t' '$1=="move"{print $2}' "$1"; }
|
||||||
|
|
||||||
|
# destinations FILE — where the plan puts them.
|
||||||
|
destinations() { awk -F'\t' '$1=="move"{print $3}' "$1"; }
|
||||||
|
|
||||||
|
# plan_moves FILE — how many files the plan stages.
|
||||||
|
plan_moves() {
|
||||||
|
[ -f "$1" ] || { echo 0; return; }
|
||||||
|
awk -F'\t' '$1=="move"' "$1" | wc -l
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
LIB="$BATS_TEST_TMPDIR/TVShows"
|
||||||
|
HOLD="$BATS_TEST_TMPDIR/.AGENTTRASH/20991231-tv-duplicates"
|
||||||
|
PLAN="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
mkdir -p "$LIB" "$HOLD"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the smaller copy of an episode is staged and the larger stays put" {
|
||||||
|
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/Doc Who - S01E01 - Pilot.mkv" 4000
|
||||||
|
at "$LIB/Doc Who (2005) [tvdbid-78804]/Season 01/backup/Doc.Who.S01E01.720p.mkv" 1200
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"backup/Doc.Who.S01E01.720p.mkv"* ]]
|
||||||
|
[[ "$output" != *"Pilot.mkv"* ]]
|
||||||
|
run destinations "$PLAN"
|
||||||
|
[[ "$output" == "$HOLD/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an equal-size pair stages exactly one copy" {
|
||||||
|
at "$LIB/Equal (2001) [tvdbid-11]/Season 01/Equal - S01E01.mkv" 2500
|
||||||
|
at "$LIB/Equal (2001) [tvdbid-11]/Rip/Equal - S01E01.mkv" 2500
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file holding one episode nothing else has is never staged" {
|
||||||
|
# The shape that defeats a folder-level verdict: Season 02 exists only here,
|
||||||
|
# so the folder that looks like a leftover is the only copy of an episode.
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 01/King - S01E01 - One.mkv" 3000
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 01/backup/King - S01E01 - One.mkv" 900
|
||||||
|
at "$LIB/King (2010) [tvdbid-21]/Season 02/King - S02E01 - Two.mkv" 800
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" != *"S02E01"* ]]
|
||||||
|
[[ "$output" == *"backup/King - S01E01 - One.mkv"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a two-in-one pack competes for every episode in its range" {
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv" 1500
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 1500
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Packs/Range - S01E01-02.mkv" 2400
|
||||||
|
|
||||||
|
# The pack is the biggest carrier of BOTH episodes, so the two rips go.
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Season 01/Range - S01E01.mkv"* ]]
|
||||||
|
[[ "$output" != *"Packs/Range - S01E01-02.mkv"* ]]
|
||||||
|
|
||||||
|
# Make the season file the bigger carrier of E02 and the pack becomes the
|
||||||
|
# only carrier of E01: it stays, however small it is.
|
||||||
|
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E01.mkv"
|
||||||
|
rm "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv"
|
||||||
|
at "$LIB/Range (2012) [tvdbid-31]/Season 01/Range - S01E02.mkv" 9000
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two tvdb ids on the same words are never merged" {
|
||||||
|
at "$LIB/Avatar The Last Airbender (2024) [tvdbid-385925]/Season 01/Avatar - S01E01.mkv" 9000
|
||||||
|
at "$LIB/Avatar The Last Airbender (1999) [tvdbid-74852]/Season 01/Avatar - S01E01.mkv" 500
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *CONFLICT* ]]
|
||||||
|
[[ "$output" == *"tvdb [74852] vs [385925]"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a derived id that contradicts the folder's own tag never merges them" {
|
||||||
|
# Measured 2026-11-02: Jellyfin's provider table carried a 2026 special's tvdb
|
||||||
|
# id on the folder tagged [tvdbid-73838]. Unioning the two makes one show of a
|
||||||
|
# 2000 series and a 2026 special, and the special's bigger files dominate the
|
||||||
|
# original's pilot. The tag is the operator's word; the nfo and `--ids` layers
|
||||||
|
# fill an untagged folder and never overrule a tagged one.
|
||||||
|
at "$LIB/Middle (2000) [tvdbid-73838]/Season 01/Middle - S01E01 - Pilot.mkv" 800
|
||||||
|
at "$LIB/Middle (2026) [tvdbid-457779]/Season 01/Middle - S01E01 - Episode 1.mkv" 1100
|
||||||
|
IDS="$BATS_TEST_TMPDIR/ids.tsv"
|
||||||
|
printf '%s\t%s\n' "Middle (2000) [tvdbid-73838]" 457779 > "$IDS"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --ids "$IDS"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"DISAGREE"* ]]
|
||||||
|
[[ "$output" == *"tag tvdb [73838]"* ]]
|
||||||
|
[[ "$output" == *"derived tvdb [457779]"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an untagged drop is identified by the tvshow.nfo in the metadata view" {
|
||||||
|
at "$LIB/Clone Wars/Season 01/Clone Wars - S01E01.mkv" 1200
|
||||||
|
at "$LIB/Clone Wars (2008) [tvdbid-84859]/Season 01/Clone Wars - S01E01.mkv" 4000
|
||||||
|
VIEW="$BATS_TEST_TMPDIR/view"
|
||||||
|
mkdir -p "$VIEW/Clone Wars"
|
||||||
|
printf '%s\n' '<tvshow><title>Clone Wars</title>' \
|
||||||
|
'<uniqueid type="tvdb" default="true">84859</uniqueid></tvshow>' \
|
||||||
|
> "$VIEW/Clone Wars/tvshow.nfo"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --meta "$VIEW"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Clone Wars/Season 01/Clone Wars - S01E01.mkv"* ]]
|
||||||
|
|
||||||
|
# Without the nfo layer nothing identifies the drop, so nothing moves.
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/no-meta.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *UNRESOLVED* ]]
|
||||||
|
[ "$(plan_moves "$BATS_TEST_TMPDIR/no-meta.tsv")" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file named by an id is keyed only when the witness names it" {
|
||||||
|
BIG="$LIB/Edd (1999) [tvdbid-75897]/Season 01/Edd - S01E01 - Down the Line.mkv"
|
||||||
|
SMALL="$LIB/Edd (1999) [tvdbid-75897]/Season 01/104348.mkv"
|
||||||
|
at "$BIG" 2100
|
||||||
|
at "$SMALL" 700
|
||||||
|
WIT="$BATS_TEST_TMPDIR/witness.tsv"
|
||||||
|
printf '%s\t1\t1\n' "$SMALL" > "$WIT"
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"unkeyed, never staged: 1 files"* ]]
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN" --witness "$WIT"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *104348.mkv* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the sidecars of a staged episode are staged with it" {
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.mkv" 3000
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.mkv" 1000
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Rip/Side - S01E01 - First.srt" 400
|
||||||
|
at "$LIB/Side (2003) [tvdbid-41]/Season 01/Side - S01E01 - First.srt" 450
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 2 ]
|
||||||
|
run sources "$PLAN"
|
||||||
|
[[ "$output" == *"Rip/Side - S01E01 - First.srt"* ]]
|
||||||
|
[[ "$output" != *"Season 01/Side - S01E01 - First.srt"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a hold inside the library is refused" {
|
||||||
|
at "$LIB/Hostage (2007) [tvdbid-51]/Season 01/Hostage - S01E01.mkv" 100
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$LIB/.AGENTTRASH/hold" \
|
||||||
|
--out "$PLAN"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"--hold"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a plan whose hold target is occupied is refused, plan and all" {
|
||||||
|
at "$LIB/Gone (2009) [tvdbid-61]/Season 01/Gone - S01E01.mkv" 1000
|
||||||
|
at "$LIB/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 1 ]
|
||||||
|
|
||||||
|
# The hold already holds one of the destinations: the executor would refuse
|
||||||
|
# it, so a plan that cannot run must not be handed over at all.
|
||||||
|
mkdir -p "$HOLD/Gone (2009) [tvdbid-61]/Rip"
|
||||||
|
at "$HOLD/Gone (2009) [tvdbid-61]/Rip/Gone - S01E01.mkv" 400
|
||||||
|
rm "$PLAN"
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[[ "$output" == *"hold target occupied"* ]]
|
||||||
|
[ ! -e "$PLAN" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a library with nothing duplicated writes an empty plan" {
|
||||||
|
at "$LIB/Unique (2015) [tvdbid-71]/Season 01/Unique - S01E01.mkv" 1000
|
||||||
|
at "$LIB/Other Show (2016) [tvdbid-72]/Season 01/Other Show - S01E01.mkv" 1000
|
||||||
|
|
||||||
|
run python3 "$TOOL" --library "$LIB" --hold "$HOLD" --out "$PLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(plan_moves "$PLAN")" -eq 0 ]
|
||||||
|
[[ "$output" == *"plan: 0 moves"* ]]
|
||||||
|
}
|
||||||
Reference in new issue
Block a user