Compare commits
83
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f430e2264d | ||
|
|
82a2183d40 | ||
|
|
c70f9b3f6d | ||
|
|
32b8901bff | ||
|
|
6d03bbc8e7 | ||
|
|
32b58dd787 | ||
|
|
0a365274b6 | ||
|
|
e531e90278 | ||
|
|
c3ab565d9b | ||
|
|
5dbbfcb8a0 | ||
|
|
02a568fb97 | ||
|
|
1e1527d101 | ||
|
|
9eb85ea1f9 | ||
|
|
96944f0b17 | ||
|
|
33f97cd3b3 | ||
|
|
254fb41fd0 | ||
|
|
3613f69593 | ||
|
|
6bf70c6a64 | ||
|
|
f1b8b60817 | ||
|
|
4ae2bde9ab | ||
|
|
ceb5617a42 | ||
|
|
a7ceda95fd | ||
|
|
a32361ee5e | ||
|
|
ce6b3b1d3b | ||
|
|
382e3eb0b8 | ||
|
|
54a393edab | ||
|
|
a5459b0b1e | ||
|
|
b03931f7ce | ||
|
|
82f6ec55c2 | ||
|
|
536949c18d | ||
|
|
337afac91e | ||
|
|
c0c77590b1 | ||
|
|
441b908937 | ||
|
|
1a1dbac268 | ||
|
|
949d3cb6f4 | ||
|
|
3ce8d62e70 | ||
|
|
38737ab84e | ||
|
|
4437519430 | ||
|
|
3b29e173c2 | ||
|
|
9706f5d2de | ||
|
|
206dc3baf1 | ||
|
|
f5a3badb56 | ||
|
|
3e4b4f24f7 | ||
|
|
86f5903e74 | ||
|
|
bd4ee78811 | ||
|
|
b7b39cdc92 | ||
|
|
09532e5216 | ||
|
|
e1786e7cf2 | ||
|
|
409a8d8b0c | ||
|
|
e83162a2ac | ||
|
|
5cd9a0c4c2 | ||
|
|
a803aa046a | ||
|
|
73f77a13c3 | ||
|
|
dd76fcf5d4 | ||
|
|
1cb28095a7 | ||
|
|
e1760e3316 | ||
|
|
272e06daf6 | ||
|
|
4db985b90d | ||
|
|
44d72b9e78 | ||
|
|
8073d0aa9d | ||
|
|
5c678b804c | ||
|
|
81e0995f7d | ||
|
|
167e6d6de3 | ||
|
|
608b0058e3 | ||
|
|
4043798053 | ||
|
|
521537a34c | ||
|
|
19fdffe8bd | ||
|
|
54a4a4180a | ||
|
|
f7aec7f154 | ||
|
|
299078eaba | ||
|
|
7b2b5b66fe | ||
|
|
e1d4d59773 | ||
|
|
5787c86954 | ||
|
|
acbb539b97 | ||
|
|
49a83ef1d5 | ||
|
|
d012654fc9 | ||
|
|
b45771e87e | ||
|
|
011ceeb344 | ||
|
|
88a9bc02be | ||
|
|
6ac544a7b6 | ||
|
|
33396d2daa | ||
|
|
3e32cb9cd1 | ||
|
|
b4a1e6069e |
No files matched your search
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
|||||||
| Target | Service | Runtime | Payload | Port |
|
| Target | Service | Runtime | Payload | Port |
|
||||||
|--------|---------|---------|---------|------|
|
|--------|---------|---------|---------|------|
|
||||||
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
|
||||||
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
|
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
|
||||||
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
|
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
|
||||||
|
|
||||||
Payload identity, shape, eras, laws and rollback live in ONE page each:
|
Payload identity, shape, eras, laws and rollback live in ONE page each:
|
||||||
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
|
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
|
||||||
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
|
|||||||
2. Bench against the service itself. No scratch ports on an LLM host — one
|
2. Bench against the service itself. No scratch ports on an LLM host — one
|
||||||
process owns the card. Use the runtime's own harness (`llama-bench`; a
|
process owns the card. Use the runtime's own harness (`llama-bench`; a
|
||||||
live llama.cpp server also reports honest `timings` per request, including
|
live llama.cpp server also reports honest `timings` per request, including
|
||||||
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
|
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
|
||||||
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
|
only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
|
||||||
`ignore_eos` run on the raw `/completion` path there answers **500 "The
|
`ignore_eos` run on the raw `/completion` path there answers **500 "The
|
||||||
model produced output that does not match the expected Content-only
|
model produced output that does not match the expected Content-only
|
||||||
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
|
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
|
||||||
@@ -439,16 +439,16 @@ disagrees with it is the bug.
|
|||||||
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
|
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
|
||||||
|
|
||||||
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
|
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
|
||||||
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
|
`scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
|
||||||
loader (the payload carries the fork's custom PQ2_0 tensor type plus
|
is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
|
||||||
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
|
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
|
||||||
loader ignoring that metadata dequantizes wrong); payload
|
loader ignoring that metadata dequantizes wrong); payload
|
||||||
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
|
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
|
||||||
tower can crash the server with concurrent image requests) + the rank-1
|
tower can crash the server with concurrent image requests) + the rank-1
|
||||||
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
|
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
|
||||||
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
|
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
|
||||||
— and build 10777 is also the gate that lets it load: older forks refuse it
|
— and the fork BUILD is also the gate that lets it load: anything older than
|
||||||
with `unsupported prism.hadamard.version: 2`.
|
b10777 refuses it with `unsupported prism.hadamard.version: 2`.
|
||||||
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
|
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
|
||||||
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
|
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
|
||||||
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
|
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
|
||||||
@@ -741,6 +741,13 @@ linked runbooks.
|
|||||||
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
|
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
|
||||||
says identity mode costs nothing, so the XDNA2 NPU returned for free.
|
says identity mode costs nothing, so the XDNA2 NPU returned for free.
|
||||||
[[dankclaw-iommu-translation-off]]
|
[[dankclaw-iommu-translation-off]]
|
||||||
|
- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
|
||||||
|
`systemctl --user start` on an already-active unit does nothing, so installing
|
||||||
|
a new digest under a running service keeps the OLD container serving while the
|
||||||
|
deploy, its health gate and the Playwright suite all pass — the authentik
|
||||||
|
handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
|
||||||
|
outpost, the one handler that restarted). A pin change is applied by
|
||||||
|
`restart`, never `start`.
|
||||||
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
|
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
|
||||||
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
|
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
|
||||||
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
|
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
|
||||||
@@ -800,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
|
|||||||
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
|
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
|
||||||
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||||
|
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
|
||||||
|
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
|
||||||
|
|
||||||
There is no page for a retired endpoint and no change diary. A payload swap
|
There is no page for a retired endpoint and no change diary. A payload swap
|
||||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||||
@@ -841,5 +850,7 @@ history.
|
|||||||
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
|
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
|
||||||
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
|
||||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||||
|
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
|
||||||
|
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
|
||||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||||
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
|
||||||
+6
-28
@@ -94,34 +94,12 @@ services:
|
|||||||
timeout: 10s
|
timeout: 10s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════
|
# Jellyfin left this file when it became a Quadlet: the container is owned by
|
||||||
# Jellyfin — media server (movies, TV, music)
|
# dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
|
||||||
# ═══════════════════════════════════════════════
|
# /home/dank/Containers/jellyfin, and its library arrives through the SSD
|
||||||
jellyfin:
|
# overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
|
||||||
image: jellyfin/jellyfin:latest
|
# was here named /opt/jellyfin/* and a read-only array bind — a second owner for
|
||||||
container_name: jellyfin
|
# the container name `jellyfin`, wrong on every path. One owner per container.
|
||||||
restart: unless-stopped
|
|
||||||
ports:
|
|
||||||
- "8096:8096/tcp"
|
|
||||||
- "8920:8920/tcp"
|
|
||||||
- "7359:7359/udp"
|
|
||||||
- "1900:1900/udp"
|
|
||||||
environment:
|
|
||||||
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
|
|
||||||
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
|
|
||||||
volumes:
|
|
||||||
- /opt/jellyfin/config:/config
|
|
||||||
- /opt/jellyfin/cache:/cache
|
|
||||||
- /mnt/entropy:/data:ro
|
|
||||||
devices:
|
|
||||||
- /dev/dri:/dev/dri
|
|
||||||
group_add:
|
|
||||||
- "992"
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
|
|
||||||
interval: 30s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# ═══════════════════════════════════════════════
|
# ═══════════════════════════════════════════════
|
||||||
# Calibre-Web — e-book web frontend
|
# Calibre-Web — e-book web frontend
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
|
||||||
|
# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
|
||||||
|
# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
|
||||||
|
#
|
||||||
|
# Installed by `scripts/media-view.sh install` — the repo copy is the source of
|
||||||
|
# truth (fleet law: never edit host state by hand).
|
||||||
|
#
|
||||||
|
# An OverlayFS merge that shows the media library as one tree while splitting
|
||||||
|
# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
|
||||||
|
# written), upperdir is a directory on the NVMe that receives every write. The
|
||||||
|
# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
|
||||||
|
# sidecars and artwork are SSD bytes — which is what makes them fast to browse
|
||||||
|
# and what keeps Plex's NFO agent off the spinners — while the media itself stays
|
||||||
|
# on the array untouched.
|
||||||
|
#
|
||||||
|
# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
|
||||||
|
# empty directory and the view would present an empty library as truth.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
|
||||||
|
RequiresMountsFor=/mnt/entropy
|
||||||
|
|
||||||
|
[Mount]
|
||||||
|
What=overlay
|
||||||
|
Where=/var/lib/media-meta/view
|
||||||
|
Type=overlay
|
||||||
|
Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=local-fs.target
|
||||||
@@ -4,8 +4,9 @@
|
|||||||
# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
|
# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
|
||||||
#
|
#
|
||||||
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
|
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
|
||||||
# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
|
# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
|
||||||
# anywhere writes to the entropy volume. Every
|
# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
|
||||||
|
# while the sidecars this server writes land on the NVMe. Every
|
||||||
# creation-time environment variable of the compose-era container is kept as
|
# creation-time environment variable of the compose-era container is kept as
|
||||||
# an Environment= line (zero drift); none of them are secrets, so this unit
|
# an Environment= line (zero drift); none of them are secrets, so this unit
|
||||||
# needs no EnvironmentFile.
|
# needs no EnvironmentFile.
|
||||||
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
|
|||||||
PublishPort=1900:1900/udp
|
PublishPort=1900:1900/udp
|
||||||
Volume=/home/dank/Containers/jellyfin/config:/config
|
Volume=/home/dank/Containers/jellyfin/config:/config
|
||||||
Volume=/home/dank/Containers/jellyfin/cache:/cache
|
Volume=/home/dank/Containers/jellyfin/cache:/cache
|
||||||
Volume=/mnt/entropy:/data:ro
|
# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
|
||||||
|
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
|
||||||
|
# fail-closed guard — starting against a plain, unmounted view directory would let
|
||||||
|
# a scan conclude the library is empty.
|
||||||
|
Volume=/var/lib/media-meta/view:/data
|
||||||
|
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
|
||||||
AddDevice=/dev/dri
|
AddDevice=/dev/dri
|
||||||
GroupAdd=keep-groups
|
GroupAdd=keep-groups
|
||||||
HealthCmd=curl -f http://localhost:8096/health
|
HealthCmd=curl -f http://localhost:8096/health
|
||||||
|
|||||||
@@ -11,7 +11,12 @@ Environment=PUID=0
|
|||||||
Environment=PGID=0
|
Environment=PGID=0
|
||||||
Environment=TZ=America/Los_Angeles
|
Environment=TZ=America/Los_Angeles
|
||||||
Volume=/var/lib/plexmediaserver:/config
|
Volume=/var/lib/plexmediaserver:/config
|
||||||
Volume=/mnt/entropy:/mnt/entropy:ro
|
# The container path stays /mnt/entropy on purpose: every path Plex recorded in
|
||||||
|
# its own database is written against it, so swapping the HOST source of this bind
|
||||||
|
# to the overlay view re-points Plex at the sidecars without touching its DB.
|
||||||
|
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
|
||||||
|
Volume=/var/lib/media-meta/view:/mnt/entropy:ro
|
||||||
|
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
|
||||||
PublishPort=32400:32400
|
PublishPort=32400:32400
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
|
|||||||
@@ -21,7 +21,7 @@
|
|||||||
# deployed knob has since dropped to 262144.
|
# deployed knob has since dropped to 262144.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
|
||||||
CUR=/home/dank/models/halogen-flashnext-v2
|
CUR=/home/dank/models/halogen-flashnext-v2
|
||||||
NEW=/home/dank/models/halogen-v2-abliterated
|
NEW=/home/dank/models/halogen-v2-abliterated
|
||||||
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
|
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
|
||||||
|
|||||||
@@ -47,7 +47,12 @@
|
|||||||
set -u -o pipefail
|
set -u -o pipefail
|
||||||
|
|
||||||
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
|
# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
|
||||||
|
# the same image the tap producer boots: a tap must be converted by the line that
|
||||||
|
# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
|
||||||
|
# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
|
||||||
|
# it. Production stays on its own pin — this is the campaign's toolchain.
|
||||||
|
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
|
||||||
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
|
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
|
||||||
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
|
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
|
||||||
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
|
||||||
|
|||||||
+16
-1
@@ -155,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
|
|||||||
# Quadlet unit now — the last compose services were the hermes trio, whose units
|
# Quadlet unit now — the last compose services were the hermes trio, whose units
|
||||||
# have owned their containers since 2026-09-12 and whose compose blocks were
|
# have owned their containers since 2026-09-12 and whose compose blocks were
|
||||||
# dead weight that could only create a duplicate container.
|
# dead weight that could only create a duplicate container.
|
||||||
|
# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
|
||||||
|
# has owned the container since the cutover, so a compose edit must not restart it.
|
||||||
|
|
||||||
DANKSERVER_COMPOSE_SERVICES = {
|
DANKSERVER_COMPOSE_SERVICES = {
|
||||||
"mariadb",
|
"mariadb",
|
||||||
"redis",
|
"redis",
|
||||||
"romm",
|
"romm",
|
||||||
"copyparty",
|
"copyparty",
|
||||||
"jellyfin",
|
|
||||||
"calibre-web",
|
"calibre-web",
|
||||||
}
|
}
|
||||||
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
|
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
|
||||||
@@ -421,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
"tautulli": {"tautulli"},
|
"tautulli": {"tautulli"},
|
||||||
}
|
}
|
||||||
_add(services, *quadlet_services.get(parts[1], set()))
|
_add(services, *quadlet_services.get(parts[1], set()))
|
||||||
|
if len(parts) >= 2 and parts[1] == "host":
|
||||||
|
# dankserver/host/systemd/media-meta-view.mount is the overlay view the
|
||||||
|
# media servers bind (scripts/media-view.sh installs it on the host),
|
||||||
|
# so editing it restarts jellyfin and plex and nothing else.
|
||||||
|
_add(services, "jellyfin", "plex")
|
||||||
if len(parts) >= 2 and parts[1] == "observability":
|
if len(parts) >= 2 and parts[1] == "observability":
|
||||||
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
|
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
|
||||||
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
|
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
|
||||||
@@ -466,6 +472,15 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
# that service (target_services then keeps it only on the host being
|
# that service (target_services then keeps it only on the host being
|
||||||
# planned). Editing the collector restarts the running exporter.
|
# planned). Editing the collector restarts the running exporter.
|
||||||
_add(services, "gpu-exporter")
|
_add(services, "gpu-exporter")
|
||||||
|
if name == "media-view.sh":
|
||||||
|
# The installer is what creates, mounts and reaps the view both media
|
||||||
|
# servers read, so a change to it is a restart of that pair.
|
||||||
|
_add(services, "jellyfin", "plex")
|
||||||
|
if name == "jellyfin-providers.sh":
|
||||||
|
# The deployer runs this before jellyfin restarts, so a change to
|
||||||
|
# the installer is a change to the provider state Jellyfin loads at
|
||||||
|
# start-up: it selects jellyfin and nothing else.
|
||||||
|
_add(services, "jellyfin")
|
||||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||||
|
|||||||
@@ -1202,7 +1202,42 @@ deploy_romm_stack() {
|
|||||||
systemctl --user restart "$service.service"
|
systemctl --user restart "$service.service"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
ensure_media_view() {
|
||||||
|
# Both media servers read their library through the SSD-backed overlay view
|
||||||
|
# (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
|
||||||
|
# array below stays read-only and every sidecar Jellyfin writes lands on the
|
||||||
|
# NVMe.
|
||||||
|
# Raise it BEFORE either container starts — a start against a bare view
|
||||||
|
# directory would let a scan clear the library database. The units also carry
|
||||||
|
# ConditionPathIsMountPoint, so a missing view skips the start rather than
|
||||||
|
# damaging anything. A relayout must be followed by `media-view.sh reap`
|
||||||
|
# (see the Media Relayout page), which this is not.
|
||||||
|
run_privileged bash scripts/media-view.sh install
|
||||||
|
bash scripts/media-view.sh verify
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_provider_keys() {
|
||||||
|
# Jellyfin's metadata-provider keys live in the container-owned plugin
|
||||||
|
# configuration under the data root, never in git; scripts/jellyfin-providers.sh
|
||||||
|
# is what places them, reading the per-service host env file (the fleet
|
||||||
|
# convention: grafana, caddy, pihole, copyparty, romm, seerr) that holds
|
||||||
|
# TMDB_API_KEY= and TVDB_API_KEY=, mode 600.
|
||||||
|
# Missing keys mean a keyless provider — a degraded library, not a broken
|
||||||
|
# deploy — so this warns rather than failing the restart. But when the keys
|
||||||
|
# ARE placed it must happen before the restart: the plugin and its keys are
|
||||||
|
# read at start-up, so a restart first would boot the old state.
|
||||||
|
local keys=/home/dank/Containers/jellyfin/provider-keys
|
||||||
|
if [ -f "$keys" ]; then
|
||||||
|
bash scripts/jellyfin-providers.sh install
|
||||||
|
else
|
||||||
|
echo "WARN: $keys is absent - provider keys NOT placed"
|
||||||
|
echo "WARN: create it (TMDB_API_KEY=, TVDB_API_KEY=, chmod 600) and re-deploy jellyfin"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
deploy_jellyfin() {
|
deploy_jellyfin() {
|
||||||
|
ensure_media_view
|
||||||
|
ensure_provider_keys
|
||||||
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
|
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
if ! systemctl --user is-active -q jellyfin.service; then
|
if ! systemctl --user is-active -q jellyfin.service; then
|
||||||
@@ -1290,6 +1325,7 @@ deploy_copyparty() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
deploy_plex() {
|
deploy_plex() {
|
||||||
|
ensure_media_view
|
||||||
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
|
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
if ! systemctl --user is-active -q plex.service; then
|
if ! systemctl --user is-active -q plex.service; then
|
||||||
@@ -1362,11 +1398,17 @@ deploy_authentik() {
|
|||||||
# generated". Boot persistence comes from [Install] WantedBy=default.target
|
# generated". Boot persistence comes from [Install] WantedBy=default.target
|
||||||
# inside each .container file, which the generator honors at user-manager
|
# inside each .container file, which the generator honors at user-manager
|
||||||
# start (linger must stay on for the deploy user).
|
# start (linger must stay on for the deploy user).
|
||||||
systemctl --user start authentik-postgres.service
|
# RESTART, not start: `start` on an already-active unit is a no-op, so an
|
||||||
|
# image bump would install the new digest and keep serving the old
|
||||||
|
# container while every gate stayed green (measured 2026-10-30: server and
|
||||||
|
# worker stayed on the previous release beside an outpost that moved,
|
||||||
|
# because only the outpost handler restarts). The generator has already
|
||||||
|
# re-rendered the unit from the .container, so a restart is what applies it.
|
||||||
|
systemctl --user restart authentik-postgres.service
|
||||||
wait_container_healthy authentik-postgres 120 || return 1
|
wait_container_healthy authentik-postgres 120 || return 1
|
||||||
systemctl --user start authentik-server.service
|
systemctl --user restart authentik-server.service
|
||||||
wait_container_healthy authentik-server 300 || return 1
|
wait_container_healthy authentik-server 300 || return 1
|
||||||
systemctl --user start authentik-worker.service
|
systemctl --user restart authentik-worker.service
|
||||||
}
|
}
|
||||||
|
|
||||||
deploy_authentik_outpost() {
|
deploy_authentik_outpost() {
|
||||||
@@ -1482,7 +1524,7 @@ deploy_anubis() {
|
|||||||
# artifact would leave a restart-looping unit and a dark endpoint. The
|
# artifact would leave a restart-looping unit and a dark endpoint. The
|
||||||
# port gate is the same as every cutover since 2026-09-16: nothing
|
# port gate is the same as every cutover since 2026-09-16: nothing
|
||||||
# starts until 7770 is released.
|
# starts until 7770 is released.
|
||||||
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
|
||||||
# The payload dir is its own twin because the NPU lane's containment gate below
|
# The payload dir is its own twin because the NPU lane's containment gate below
|
||||||
# reads it; every artifact path under it stays a full literal as usual.
|
# reads it; every artifact path under it stays a full literal as usual.
|
||||||
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
|
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
|
||||||
|
|||||||
@@ -8,8 +8,10 @@
|
|||||||
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
|
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
|
||||||
# whose files do not match.
|
# whose files do not match.
|
||||||
#
|
#
|
||||||
# The manifest below IS that catalog block (dumped from the pinned 0.16.2
|
# The manifest below IS that catalog block (dumped from the pinned 0.18.20
|
||||||
# image on 2026-10-19), so the files land in the layout the entrypoint
|
# image on 2026-10-30; the 0.16.2 block this replaced named different device
|
||||||
|
# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
|
||||||
|
# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
|
||||||
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
|
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
|
||||||
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
|
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
|
||||||
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
|
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
|
||||||
@@ -57,21 +59,25 @@ manifest() {
|
|||||||
case "$1" in
|
case "$1" in
|
||||||
qwen3-embedding-0.6b)
|
qwen3-embedding-0.6b)
|
||||||
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
|
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
|
||||||
NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
|
NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
|
||||||
FILES=(
|
FILES=(
|
||||||
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
|
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
|
||||||
"devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
|
"devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
|
||||||
"devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
|
"devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
|
||||||
"devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
|
"devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
|
||||||
"devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
|
"devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
|
||||||
"devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
|
"devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
|
||||||
"devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
|
"devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
|
||||||
"devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
|
"devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
|
||||||
"devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
|
"devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
|
||||||
"devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
|
"devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
|
||||||
"devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
|
"devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
|
||||||
"devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
|
"devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
|
||||||
"devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
|
"devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
|
||||||
|
"devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
|
||||||
|
"devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
|
||||||
|
"devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
|
||||||
|
"devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
|
||||||
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
|
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
|
||||||
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
|
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
|
||||||
) ;;
|
) ;;
|
||||||
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
|
|||||||
# 62 GiB fetch hit this on 2026-10-18).
|
# 62 GiB fetch hit this on 2026-10-18).
|
||||||
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|
||||||
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
|
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
|
||||||
curl -fL --retry 5 -C - -o "$path" "$url"
|
# Resume ONLY into a genuinely partial file. A staged file of a
|
||||||
|
# DIFFERENT size is an artifact of another revision (a version
|
||||||
|
# bump re-cuts the device programs), and `-C -` against it
|
||||||
|
# appends the tail to the old bytes: the result is the right
|
||||||
|
# BYTE COUNT with the wrong content, which reads as a corrupt
|
||||||
|
# program, not as a fetch fault (measured 2026-10-30 on
|
||||||
|
# consts.hnpb). Fetch those from zero.
|
||||||
|
resume=()
|
||||||
|
if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
|
||||||
|
resume=(-C -)
|
||||||
|
fi
|
||||||
|
curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
|
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
|
||||||
|
|||||||
@@ -10,8 +10,11 @@
|
|||||||
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
|
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
|
||||||
# engine moves with the payload, never ahead of it.
|
# engine moves with the payload, never ahead of it.
|
||||||
# Image digest-pinned (a tag pull silently re-points; the digest is what was
|
# Image digest-pinned (a tag pull silently re-points; the digest is what was
|
||||||
# qualified — 0.16.2 is the version the model card names as its quickstart pin).
|
# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
|
||||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
# pin) — the vendor has moved two minor versions and the payload lock is what
|
||||||
|
# the deploy's health gate plus the boot ledger decide; rollback is the
|
||||||
|
# previous digest, 0c61bf84…, on this same line.
|
||||||
|
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
|
||||||
# Container identity kept from the 27B era: the unit's ExecStop names it.
|
# Container identity kept from the 27B era: the unit's ExecStop names it.
|
||||||
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||||||
# Predecessor identity the serve script clears if it is still holding the GPUs.
|
# Predecessor identity the serve script clears if it is still holding the GPUs.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
|
||||||
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
||||||
# the refusal-direction adapter applied at load.
|
# the refusal-direction adapter applied at load.
|
||||||
#
|
#
|
||||||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||||||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||||||
# knobs and only the reason a reader editing THIS line cannot recover
|
# knobs and only the reason a reader editing THIS line cannot recover
|
||||||
# elsewhere. Where the two disagree, this file wins on values.
|
# elsewhere. Where the two disagree, this file wins on values.
|
||||||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
|
||||||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||||||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||||||
# metadata dequantizes wrong, silently.
|
# metadata dequantizes wrong, silently.
|
||||||
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
|
|||||||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||||||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||||||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||||||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
|
||||||
|
# disk as the rollback path).
|
||||||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||||||
# every --spec-draft-*); half a set has segfaulted at load.
|
# every --spec-draft-*); half a set has segfaulted at load.
|
||||||
SPEC_TYPE=draft-mtp
|
SPEC_TYPE=draft-mtp
|
||||||
|
|||||||
Executable
+391
@@ -0,0 +1,391 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Metadata-provider keys for Jellyfin on Dankserver, placed from the service's
|
||||||
|
# host env file with the plugin artifact pinned by content hash.
|
||||||
|
#
|
||||||
|
# Jellyfin 12 ships TMDb, OMDb, StudioImages and MusicBrainz as CORE plugins:
|
||||||
|
# they have no DLL, only a configuration file under the data root, and that
|
||||||
|
# file is created by the running container. So the fleet's provider state lives
|
||||||
|
# in two places the repo does not track —
|
||||||
|
#
|
||||||
|
# plugins/configurations/Jellyfin.Plugin.Tmdb.xml <TmdbApiKey /> (empty = keyless)
|
||||||
|
# plugins/configurations/Jellyfin.Plugin.Tvdb.xml <ApiKey /> (needs the plugin)
|
||||||
|
#
|
||||||
|
# and both are owned by the container user. This tool is the only thing that
|
||||||
|
# writes them: it is declarative, idempotent, and it never prints a key, so it
|
||||||
|
# is safe to run while someone watches.
|
||||||
|
#
|
||||||
|
# Keys never enter git and never enter argv (a value in argv is world-readable
|
||||||
|
# through /proc/<pid>/cmdline). They live in the per-service host env file that
|
||||||
|
# is the fleet's convention for container credentials (grafana, caddy, pihole,
|
||||||
|
# copyparty, romm, seerr), mode 600:
|
||||||
|
#
|
||||||
|
# /home/dank/Containers/jellyfin/provider-keys
|
||||||
|
# TMDB_API_KEY=<key>
|
||||||
|
# TVDB_API_KEY=<key>
|
||||||
|
# TVDB_PIN=<pin> # optional: only a subscriber-shaped TVDB key
|
||||||
|
# # needs one; a licensed key does not
|
||||||
|
#
|
||||||
|
# TVDB itself is not core in 12 (the metadata provider was cut in 10.9), so it
|
||||||
|
# is installed from the official catalog, pinned three ways: sha256 of the
|
||||||
|
# artifact, the plugin's own guid, and its version. A matching sha proves only
|
||||||
|
# that the bytes are the bytes fetched — the guid check is what says it is the
|
||||||
|
# official plugin, the version check is what says it is the one this file was
|
||||||
|
# reviewed against.
|
||||||
|
#
|
||||||
|
# Subcommands
|
||||||
|
# plan print the resolved shape and every pending change, touch nothing
|
||||||
|
# install fetch+verify the plugin, then write the keys (backing up first)
|
||||||
|
# verify prove the plugin and both keys are actually in place
|
||||||
|
#
|
||||||
|
# Nothing here restarts Jellyfin: the deployer owns that
|
||||||
|
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
|
||||||
|
# start-up, so a key written while the server runs is not live until then.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Jellyfin's data root on Dankserver (the rootless quadlet's config volume).
|
||||||
|
JF_ROOT="${JF_ROOT:-/home/dank/Containers/jellyfin/config}"
|
||||||
|
PROV_KEYS="${PROV_KEYS:-/home/dank/Containers/jellyfin/provider-keys}"
|
||||||
|
TRASH_DIR="${TRASH_DIR:-$HOME/.AGENTTRASH/$(date +%Y%m%d)-jellyfin-providers}"
|
||||||
|
|
||||||
|
PLUGINS="$JF_ROOT/plugins"
|
||||||
|
CONF="$PLUGINS/configurations"
|
||||||
|
TMDB_XML="$CONF/Jellyfin.Plugin.Tmdb.xml"
|
||||||
|
TVDB_XML="$CONF/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
|
||||||
|
# repo.jellyfin.org catalog entry for TheTVDB (owner: jellyfin), version 24.0.0.0,
|
||||||
|
# targetAbi 12.0.0.0 — published 2026-09-09, verified against Jellyfin 12.4.0.
|
||||||
|
TVDB_PLUGIN_URL="${TVDB_PLUGIN_URL:-https://repo.jellyfin.org/files/plugin/thetvdb/thetvdb_24.0.0.0.zip}"
|
||||||
|
TVDB_PLUGIN_SHA256="${TVDB_PLUGIN_SHA256:-6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319}"
|
||||||
|
TVDB_PLUGIN_GUID="${TVDB_PLUGIN_GUID:-a677c0da-fac5-4cde-941a-7134223f14c8}"
|
||||||
|
TVDB_PLUGIN_VERSION="${TVDB_PLUGIN_VERSION:-24.0.0.0}"
|
||||||
|
PLUGIN_DIR="thetvdb_${TVDB_PLUGIN_VERSION}"
|
||||||
|
TVDB_DLL="Jellyfin.Plugin.Tvdb.dll"
|
||||||
|
|
||||||
|
die() {
|
||||||
|
printf 'jellyfin-providers: %s\n' "$*" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
printf 'usage: %s plan|install|verify\n' "$0" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# check_keys — refuse anything that is not a readable private file. An open
|
||||||
|
# mode is an error, not a warning: a provider key readable by every user on the
|
||||||
|
# box is the failure this check exists to catch.
|
||||||
|
check_keys() {
|
||||||
|
[ -f "$PROV_KEYS" ] ||
|
||||||
|
die "$PROV_KEYS not found (create it with TMDB_API_KEY= and TVDB_API_KEY=, then chmod 600)"
|
||||||
|
local mode
|
||||||
|
mode="$(stat -c '%a' "$PROV_KEYS")"
|
||||||
|
case "$mode" in
|
||||||
|
600 | 400) ;;
|
||||||
|
*) die "$PROV_KEYS is mode $mode; provider keys must be 600" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# keys_value <VAR> — the value of one variable in the env file, or nothing.
|
||||||
|
keys_value() {
|
||||||
|
awk -v k="$1" -F= '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$PROV_KEYS"
|
||||||
|
}
|
||||||
|
|
||||||
|
# keys_len <VAR> — the length, and a refusal naming the variable when absent.
|
||||||
|
# The value itself is never printed.
|
||||||
|
keys_len() {
|
||||||
|
local v
|
||||||
|
v="$(keys_value "$1")"
|
||||||
|
[ -n "$v" ] || die "$PROV_KEYS has no $1= line"
|
||||||
|
printf '%s' "${#v}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# xml_value <file> <element> — the element's text, empty when it is absent or
|
||||||
|
# self-closing.
|
||||||
|
xml_value() {
|
||||||
|
python3 - "$1" "$2" <<'PYEOF'
|
||||||
|
import re, sys
|
||||||
|
src = open(sys.argv[1], encoding="utf-8").read()
|
||||||
|
m = re.search(r"<%s(?:\s*/>|>(.*?)</%s>)" % (sys.argv[2], sys.argv[2]), src, re.S)
|
||||||
|
print(m.group(1).strip() if (m and m.group(1)) else "")
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# xml_secret <file> <element> <var> — set <element> to that variable from the
|
||||||
|
# env file. The python process reads the env file itself, so the value never
|
||||||
|
# appears in argv.
|
||||||
|
xml_secret() {
|
||||||
|
python3 - "$1" "$2" "$3" "$PROV_KEYS" <<'PYEOF'
|
||||||
|
import os, re, sys, tempfile
|
||||||
|
from xml.sax.saxutils import escape
|
||||||
|
|
||||||
|
xml_path, elem, var, keys_path = sys.argv[1:5]
|
||||||
|
val = ""
|
||||||
|
for line in open(keys_path, encoding="utf-8"):
|
||||||
|
key, sep, rest = line.partition("=")
|
||||||
|
if sep and key.strip() == var:
|
||||||
|
val = rest.strip()
|
||||||
|
break
|
||||||
|
if not val:
|
||||||
|
raise SystemExit(f"{keys_path} has no {var}= line")
|
||||||
|
|
||||||
|
src = open(xml_path, encoding="utf-8").read()
|
||||||
|
pat = re.compile(r"([ \t]*)<%s(?:\s*/>|>.*?</%s>)" % (elem, elem), re.S)
|
||||||
|
text = escape(val)
|
||||||
|
if pat.search(src):
|
||||||
|
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
|
||||||
|
else:
|
||||||
|
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
|
||||||
|
if out == src:
|
||||||
|
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
|
||||||
|
|
||||||
|
d, base = os.path.split(xml_path)
|
||||||
|
fd, tmp = tempfile.mkstemp(prefix=base + ".", dir=d)
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||||
|
fh.write(out)
|
||||||
|
os.chmod(tmp, os.stat(xml_path).st_mode & 0o777)
|
||||||
|
os.replace(tmp, xml_path)
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# plugin_meta <zip> — "guid version" out of the artifact's own meta.json.
|
||||||
|
plugin_meta() {
|
||||||
|
python3 - "$1" <<'PYEOF'
|
||||||
|
import json, sys, zipfile
|
||||||
|
meta = json.loads(zipfile.ZipFile(sys.argv[1]).read("meta.json").decode("utf-8"))
|
||||||
|
print(meta.get("guid", ""), meta.get("version", ""))
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
check_root() {
|
||||||
|
# Never fabricate a data root: creating plugins/ beside a typo'd path would
|
||||||
|
# install a plugin Jellyfin never loads and hide the operator's mistake.
|
||||||
|
[ -d "$CONF" ] || die "no Jellyfin configuration directory at $CONF (is JF_ROOT right?)"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_keys_report() {
|
||||||
|
check_keys
|
||||||
|
# A refusal inside a command substitution used as a printf argument would
|
||||||
|
# be swallowed, so the guard runs as its own statement.
|
||||||
|
keys_len TMDB_API_KEY >/dev/null
|
||||||
|
keys_len TVDB_API_KEY >/dev/null
|
||||||
|
printf '= keys read from %s (mode %s)\n' "$PROV_KEYS" "$(stat -c '%a' "$PROV_KEYS")"
|
||||||
|
}
|
||||||
|
|
||||||
|
# The plugin's state, as one line in the shared vocabulary.
|
||||||
|
plugin_state() {
|
||||||
|
if [ -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||||
|
printf '= %s already installed\n' "$PLUGIN_DIR"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local other
|
||||||
|
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
|
||||||
|
if [ -n "$other" ]; then
|
||||||
|
printf '+ install %s (a different version is present: %s)\n' "$PLUGIN_DIR" "$other"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
printf '+ install %s from %s\n' "$PLUGIN_DIR" "${TVDB_PLUGIN_URL#*://}"
|
||||||
|
}
|
||||||
|
|
||||||
|
tmdb_state() {
|
||||||
|
if [ ! -f "$TMDB_XML" ]; then
|
||||||
|
printf '! %s is missing — save the TMDb plugin settings once in the UI, then re-run\n' "$TMDB_XML" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
local now want
|
||||||
|
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
|
||||||
|
want="$(keys_value TMDB_API_KEY)"
|
||||||
|
if [ -n "$now" ] && [ "$now" = "$want" ]; then
|
||||||
|
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
|
||||||
|
elif [ -n "$now" ]; then
|
||||||
|
printf '+ replace the TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
|
||||||
|
else
|
||||||
|
printf '+ set TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
tvdb_state() {
|
||||||
|
local now
|
||||||
|
if [ -f "$TVDB_XML" ]; then
|
||||||
|
now="$(xml_value "$TVDB_XML" ApiKey)"
|
||||||
|
if [ -n "$now" ]; then
|
||||||
|
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
|
||||||
|
else
|
||||||
|
printf '+ fill the empty ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
printf '+ create Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
write_plugin() {
|
||||||
|
local tmp guid version other
|
||||||
|
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
|
||||||
|
[ -z "$other" ] ||
|
||||||
|
die "another TheTVDB version is already installed ($other); move it to the trash first — two copies of one plugin means two loads"
|
||||||
|
|
||||||
|
tmp="$(mktemp "${TMPDIR:-/tmp}/thetvdb.XXXXXX.zip")"
|
||||||
|
# The pins are checked before anything is unpacked, so a bad artifact never
|
||||||
|
# reaches the plugins directory.
|
||||||
|
curl -fsSL --max-time 300 -o "$tmp" "$TVDB_PLUGIN_URL" ||
|
||||||
|
{ rm -f "$tmp"; die "could not fetch $TVDB_PLUGIN_URL"; }
|
||||||
|
local got
|
||||||
|
got="$(sha256sum "$tmp" | cut -d' ' -f1)"
|
||||||
|
if [ "$got" != "$TVDB_PLUGIN_SHA256" ]; then
|
||||||
|
rm -f "$tmp"
|
||||||
|
die "artifact sha256 is $got, pinned $TVDB_PLUGIN_SHA256 — refusing to install"
|
||||||
|
fi
|
||||||
|
read -r guid version <<<"$(plugin_meta "$tmp")"
|
||||||
|
if [ "$guid" != "$TVDB_PLUGIN_GUID" ]; then
|
||||||
|
rm -f "$tmp"
|
||||||
|
die "artifact guid is $guid, expected the official $TVDB_PLUGIN_GUID — refusing to install"
|
||||||
|
fi
|
||||||
|
if [ "$version" != "$TVDB_PLUGIN_VERSION" ]; then
|
||||||
|
rm -f "$tmp"
|
||||||
|
die "artifact version is $version, pinned $TVDB_PLUGIN_VERSION — refusing to install"
|
||||||
|
fi
|
||||||
|
mkdir -p "$PLUGINS/$PLUGIN_DIR"
|
||||||
|
python3 -m zipfile -e "$tmp" "$PLUGINS/$PLUGIN_DIR/"
|
||||||
|
rm -f "$tmp"
|
||||||
|
[ -f "$PLUGINS/$PLUGIN_DIR/$TVDB_DLL" ] ||
|
||||||
|
die "$PLUGIN_DIR has no $TVDB_DLL — the artifact does not match this plugin"
|
||||||
|
printf '+ installed %s\n' "$PLUGIN_DIR"
|
||||||
|
}
|
||||||
|
|
||||||
|
backup_to_trash() {
|
||||||
|
local f="$1"
|
||||||
|
mkdir -p "$TRASH_DIR"
|
||||||
|
local dest="$TRASH_DIR/$(basename "$f").$(date +%H%M%S)"
|
||||||
|
cp -p "$f" "$dest"
|
||||||
|
printf '+ previous copy kept at %s\n' "$dest"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_write_keys() {
|
||||||
|
local now want
|
||||||
|
KEYS_CHANGED=0
|
||||||
|
# TMDb: the running container owns this file and every setting in it. Only
|
||||||
|
# the key element is touched, so country, language and adult settings
|
||||||
|
# survive the edit.
|
||||||
|
[ -f "$TMDB_XML" ] ||
|
||||||
|
die "$TMDB_XML is missing — save the TMDb plugin settings once in the UI, then re-run"
|
||||||
|
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
|
||||||
|
keys_len TMDB_API_KEY >/dev/null
|
||||||
|
want="$(keys_value TMDB_API_KEY)"
|
||||||
|
if [ "$now" = "$want" ]; then
|
||||||
|
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
|
||||||
|
else
|
||||||
|
backup_to_trash "$TMDB_XML"
|
||||||
|
xml_secret "$TMDB_XML" TmdbApiKey TMDB_API_KEY
|
||||||
|
printf '+ set TmdbApiKey (%s chars)\n' "${#want}"
|
||||||
|
KEYS_CHANGED=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# TVDB: never overwrite a config the operator saved through the UI.
|
||||||
|
if [ -f "$TVDB_XML" ]; then
|
||||||
|
now="$(xml_value "$TVDB_XML" ApiKey)"
|
||||||
|
if [ -n "$now" ]; then
|
||||||
|
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
|
||||||
|
else
|
||||||
|
keys_len TVDB_API_KEY >/dev/null
|
||||||
|
backup_to_trash "$TVDB_XML"
|
||||||
|
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
|
||||||
|
if [ -n "$(keys_value TVDB_PIN)" ] && [ -z "$(xml_value "$TVDB_XML" Pin)" ]; then
|
||||||
|
xml_secret "$TVDB_XML" Pin TVDB_PIN
|
||||||
|
fi
|
||||||
|
printf '+ filled the ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
|
||||||
|
KEYS_CHANGED=1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
keys_len TVDB_API_KEY >/dev/null
|
||||||
|
cat >"$TVDB_XML" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<ApiKey />
|
||||||
|
</Configuration>
|
||||||
|
XML
|
||||||
|
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
|
||||||
|
if [ -n "$(keys_value TVDB_PIN)" ]; then
|
||||||
|
xml_secret "$TVDB_XML" Pin TVDB_PIN
|
||||||
|
fi
|
||||||
|
printf '+ created Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
|
||||||
|
KEYS_CHANGED=1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_plan() {
|
||||||
|
check_root
|
||||||
|
check_keys_report
|
||||||
|
printf 'root %s\n' "$JF_ROOT"
|
||||||
|
plugin_state
|
||||||
|
tmdb_state
|
||||||
|
tvdb_state
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_install() {
|
||||||
|
check_root
|
||||||
|
check_keys_report
|
||||||
|
KEYS_CHANGED=0
|
||||||
|
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||||
|
write_plugin
|
||||||
|
KEYS_CHANGED=1
|
||||||
|
else
|
||||||
|
printf '= %s already installed\n' "$PLUGIN_DIR"
|
||||||
|
fi
|
||||||
|
cmd_write_keys
|
||||||
|
if [ "$KEYS_CHANGED" -eq 1 ]; then
|
||||||
|
# Nothing is restarted here on purpose: this host may be playing media
|
||||||
|
# right now, and both the plugin and its keys are read at start-up.
|
||||||
|
printf 'next: scripts/deploy.sh dankserver --service jellyfin (the plugin and keys load at start-up)\n'
|
||||||
|
else
|
||||||
|
printf '= no change\n'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_verify() {
|
||||||
|
check_root
|
||||||
|
local bad=0
|
||||||
|
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
|
||||||
|
printf 'MISSING plugin directory %s (thetvdb not installed)\n' "$PLUGIN_DIR" >&2
|
||||||
|
bad=1
|
||||||
|
else
|
||||||
|
local dlls abi
|
||||||
|
dlls="$(find "$PLUGINS/$PLUGIN_DIR" -maxdepth 1 -name '*.dll' | wc -l)"
|
||||||
|
[ "$dlls" -ge 1 ] || {
|
||||||
|
printf 'MISSING %s inside %s\n' "$TVDB_DLL" "$PLUGIN_DIR" >&2
|
||||||
|
bad=1
|
||||||
|
}
|
||||||
|
abi="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("targetAbi","?"))' \
|
||||||
|
"$PLUGINS/$PLUGIN_DIR/meta.json" 2>/dev/null || echo '?')"
|
||||||
|
printf 'plugin %s: %s dll(s), targetAbi %s\n' "$PLUGIN_DIR" "$dlls" "$abi"
|
||||||
|
fi
|
||||||
|
local pair elem n
|
||||||
|
for pair in "$TMDB_XML|TmdbApiKey" "$TVDB_XML|ApiKey"; do
|
||||||
|
elem="${pair##*|}"
|
||||||
|
if [ ! -f "${pair%%|*}" ]; then
|
||||||
|
printf 'MISSING %s\n' "${pair%%|*}" >&2
|
||||||
|
bad=1
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if ! python3 -c 'import sys, xml.dom.minidom; xml.dom.minidom.parse(sys.argv[1])' "${pair%%|*}"; then
|
||||||
|
printf 'UNPARSEABLE %s\n' "${pair%%|*}" >&2
|
||||||
|
bad=1
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
n="$(xml_value "${pair%%|*}" "$elem")"
|
||||||
|
if [ -z "$n" ]; then
|
||||||
|
printf 'MISSING key: %s has no %s value\n' "$(basename "${pair%%|*}")" "$elem" >&2
|
||||||
|
bad=1
|
||||||
|
else
|
||||||
|
printf '%s %s chars\n' "$elem" "${#n}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[ "$bad" -eq 0 ] || return 1
|
||||||
|
printf 'providers configured\n'
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd="${1:-}"
|
||||||
|
case "$cmd" in
|
||||||
|
plan) cmd_plan ;;
|
||||||
|
install) cmd_install ;;
|
||||||
|
verify) cmd_verify ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
Executable
+454
@@ -0,0 +1,454 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""media-apply.py — the only thing that moves movie-library files.
|
||||||
|
|
||||||
|
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
|
||||||
|
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
|
||||||
|
an interrupted run has destroyed nothing. Every run is resumable and every run
|
||||||
|
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
|
||||||
|
BEFORE the rename it authorises. That ordering is the whole design: a crash
|
||||||
|
leaves either an intent whose move happened or an intent whose move never ran,
|
||||||
|
and both are recognised on the next run. A file that moved with no journal line
|
||||||
|
cannot have been moved by this tool, and saying so is better than guessing.
|
||||||
|
|
||||||
|
Four subcommands:
|
||||||
|
|
||||||
|
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
|
||||||
|
no --apply => report only and write NOTHING, not even folders
|
||||||
|
rollback --journal J [--root ROOT] [--apply]
|
||||||
|
back to the paths the census saw; folders this run created are
|
||||||
|
removed only when empty, never anything else
|
||||||
|
verify --plan PLAN --journal J [--root ROOT]
|
||||||
|
every planned file is at its target with the journalled byte count
|
||||||
|
sweep --journal J [--root ROOT] [--apply]
|
||||||
|
removes the folders THIS RUN emptied — the journal has to show a
|
||||||
|
file that moved out of one, so a folder someone made by hand can
|
||||||
|
never be removed by this tool, and rollback puts them back
|
||||||
|
|
||||||
|
Refusals (exit 2, before the first rename, with nothing touched): a target
|
||||||
|
already occupied by a foreign file, a source that has gone, a duplicate target,
|
||||||
|
a path outside --root, a journal that was already rolled back, a move that would
|
||||||
|
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
|
||||||
|
|
||||||
|
--force relaxes only the guards that are opinion (journal already rolled back,
|
||||||
|
paths outside --root); it never allows overwriting a file.
|
||||||
|
|
||||||
|
--metadata-view names the overlay the media servers read their library through
|
||||||
|
(default /var/lib/media-meta/view). The sidecar follow-up printed after an apply
|
||||||
|
or rollback is conditioned on that path being a mountpoint: where no view is
|
||||||
|
mounted the servers bind the array itself, the sidecars moved with the media,
|
||||||
|
and a reap order would point at a directory that is not there.
|
||||||
|
|
||||||
|
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
|
||||||
|
inside it: a log inside the library is a file Jellyfin will index.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
|
||||||
|
|
||||||
|
|
||||||
|
class Refuse(Exception):
|
||||||
|
"""A gate the operator has to see; every refusal leaves the library alone."""
|
||||||
|
|
||||||
|
|
||||||
|
def journal_default(root):
|
||||||
|
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
|
||||||
|
|
||||||
|
|
||||||
|
class Journal:
|
||||||
|
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
|
||||||
|
|
||||||
|
def __init__(self, path, writing):
|
||||||
|
self.path, self.writing = path, writing
|
||||||
|
self.records = []
|
||||||
|
if os.path.exists(path):
|
||||||
|
for line in open(path):
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
self.records.append(json.loads(line))
|
||||||
|
self._fh = None
|
||||||
|
|
||||||
|
def already(self, dst, src):
|
||||||
|
"""An intent for this exact (src, dst) was written by an earlier run."""
|
||||||
|
return any(r.get("op") == "move" and r.get("dst") == dst
|
||||||
|
and r.get("src") == src for r in self.records)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def rolled_back(self):
|
||||||
|
return any(r.get("op") == "rollback" for r in self.records)
|
||||||
|
|
||||||
|
def record(self, **fields):
|
||||||
|
self.records.append(fields)
|
||||||
|
if not self.writing:
|
||||||
|
return
|
||||||
|
if self._fh is None:
|
||||||
|
self._fh = open(self.path, "a")
|
||||||
|
self._fh.write(json.dumps(fields) + "\n")
|
||||||
|
self._fh.flush()
|
||||||
|
os.fsync(self._fh.fileno())
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
if self._fh:
|
||||||
|
self._fh.close()
|
||||||
|
|
||||||
|
|
||||||
|
def read_plan(path, root, force):
|
||||||
|
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
|
||||||
|
mkdirs, keeps, moves = [], [], []
|
||||||
|
with open(path) as fh:
|
||||||
|
header = fh.readline()
|
||||||
|
if not header.startswith("action\t"):
|
||||||
|
raise Refuse(f"{path} is not a plan (first column is "
|
||||||
|
f"{header.split(chr(9))[0]!r}, expected 'action') —"
|
||||||
|
" regenerate it with scripts/media-plan.py")
|
||||||
|
for line in fh:
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if f[0] == "mkdir":
|
||||||
|
mkdirs.append(f[2])
|
||||||
|
elif f[0] == "keep":
|
||||||
|
keeps.append(f[1])
|
||||||
|
elif f[0] == "move":
|
||||||
|
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
|
||||||
|
if not force:
|
||||||
|
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
|
||||||
|
if p and not (p == root or p.startswith(root + os.sep)):
|
||||||
|
raise Refuse(f"{p} is outside --root {root}"
|
||||||
|
" (a plan built for another library?)")
|
||||||
|
return mkdirs, keeps, moves
|
||||||
|
|
||||||
|
|
||||||
|
# The media servers may read the library through the SSD overlay view
|
||||||
|
# (scripts/media-view.sh). A relayout moves bytes on the raw array, so sidecars
|
||||||
|
# the servers wrote through that view stay at the old coordinates — naming the
|
||||||
|
# follow-up here is the point of use for that fact, but only where the view
|
||||||
|
# exists: where it is not mounted the servers bind the array itself, the
|
||||||
|
# sidecars moved with the media, and a reap order points at a directory that
|
||||||
|
# is not there. See the Media Metadata View page for the host's state.
|
||||||
|
METADATA_VIEW = "/var/lib/media-meta/view"
|
||||||
|
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
|
||||||
|
" these moves — next: scripts/media-view.sh reap")
|
||||||
|
|
||||||
|
|
||||||
|
def metadata_note(args):
|
||||||
|
"""Print the sidecar follow-up only when the overlay is really mounted."""
|
||||||
|
if os.path.ismount(args.metadata_view):
|
||||||
|
print(METADATA_HINT)
|
||||||
|
|
||||||
|
|
||||||
|
def do_apply(args, root, journal, mkdirs, keeps, moves):
|
||||||
|
"""Preflight everything, then folders, then renames. Never a partial start."""
|
||||||
|
problems = []
|
||||||
|
for p in keeps:
|
||||||
|
if not os.path.exists(p):
|
||||||
|
problems.append(f"missing: keep row source is gone: {p}")
|
||||||
|
seen = {}
|
||||||
|
for src, dst, _kind in moves:
|
||||||
|
if src == dst:
|
||||||
|
problems.append(f"no-op: move row whose target is its source: {src}")
|
||||||
|
if dst in seen:
|
||||||
|
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
|
||||||
|
seen[dst] = src
|
||||||
|
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
|
||||||
|
continue # this intent already ran
|
||||||
|
if not os.path.exists(src):
|
||||||
|
if os.path.exists(dst):
|
||||||
|
problems.append(f"missing: {src} is gone and {dst} carries no journal"
|
||||||
|
" line — it moved outside this tool, refusing to guess")
|
||||||
|
else:
|
||||||
|
problems.append(f"missing: {src} is gone and {dst} is not there either")
|
||||||
|
continue
|
||||||
|
if os.path.exists(dst):
|
||||||
|
problems.append(f"occupied: target already holds another file: {dst}")
|
||||||
|
if problems:
|
||||||
|
print(f"refusing: {len(problems)} problem(s), nothing moved")
|
||||||
|
for p in sorted(problems)[:20]:
|
||||||
|
print(" " + p)
|
||||||
|
if len(problems) > 20:
|
||||||
|
print(f" ... and {len(problems) - 20} more")
|
||||||
|
return 2
|
||||||
|
|
||||||
|
if not args.apply:
|
||||||
|
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
|
||||||
|
" — nothing written (pass --apply to run)")
|
||||||
|
for src, dst, _k in moves[:5]:
|
||||||
|
print(f" would move {src} -> {dst}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
|
||||||
|
dirs_new = 0
|
||||||
|
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
|
||||||
|
if d in made or os.path.isdir(d):
|
||||||
|
continue
|
||||||
|
journal.record(op="mkdir", dst=d)
|
||||||
|
os.makedirs(d, exist_ok=True)
|
||||||
|
dirs_new += 1
|
||||||
|
|
||||||
|
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
|
||||||
|
# library this size is not an option anyone would choose.
|
||||||
|
cross = []
|
||||||
|
for src, dst, _k in moves:
|
||||||
|
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
|
||||||
|
continue
|
||||||
|
parent = os.path.dirname(dst)
|
||||||
|
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
|
||||||
|
cross.append(dst)
|
||||||
|
if cross:
|
||||||
|
print(f"refusing: {len(cross)} moves would cross a filesystem"
|
||||||
|
f" (first: {cross[0]}) — this tool only renames")
|
||||||
|
return 2
|
||||||
|
|
||||||
|
moved = done = 0
|
||||||
|
for src, dst, _kind in moves:
|
||||||
|
if journal.already(dst, src):
|
||||||
|
if os.path.exists(dst) and not os.path.exists(src):
|
||||||
|
done += 1
|
||||||
|
continue
|
||||||
|
if os.path.exists(src) and not os.path.exists(dst):
|
||||||
|
os.rename(src, dst) # the intent ran, the move did not
|
||||||
|
moved += 1
|
||||||
|
continue
|
||||||
|
st = os.stat(src)
|
||||||
|
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
|
||||||
|
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
|
||||||
|
os.rename(src, dst)
|
||||||
|
moved += 1
|
||||||
|
print(f"applied: {moved} moved, {done} already done in an earlier run,"
|
||||||
|
f" {dirs_new} folders created, {len(keeps)} files stayed put")
|
||||||
|
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
|
||||||
|
f" --journal {journal.path} --root {root} --apply")
|
||||||
|
metadata_note(args)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def do_verify(root, journal, moves, keeps):
|
||||||
|
"""Every planned file sits at its target carrying the journalled bytes."""
|
||||||
|
journalled = {(r["src"], r["dst"]): r for r in journal.records
|
||||||
|
if r.get("op") == "move"}
|
||||||
|
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
|
||||||
|
if r.get("op") == "move"]
|
||||||
|
bad, missing, bytes_total = [], [], 0
|
||||||
|
for src, dst, _kind in todo:
|
||||||
|
rec = journalled.get((src, dst))
|
||||||
|
if not os.path.exists(dst):
|
||||||
|
missing.append(dst)
|
||||||
|
continue
|
||||||
|
size = os.path.getsize(dst)
|
||||||
|
bytes_total += size
|
||||||
|
if rec and rec.get("bytes") not in (None, size):
|
||||||
|
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
|
||||||
|
if os.path.exists(src) and src != dst:
|
||||||
|
bad.append(f"{src}: still in place as well as at {dst}")
|
||||||
|
missing += [p for p in keeps if not os.path.exists(p)]
|
||||||
|
if bad or missing:
|
||||||
|
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
|
||||||
|
for line in (missing + bad)[:20]:
|
||||||
|
print(" " + line)
|
||||||
|
return 2
|
||||||
|
empty = [r["dst"] for r in journal.records
|
||||||
|
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
|
||||||
|
and not os.listdir(r["dst"])]
|
||||||
|
print(f"verified: {len(todo)} files at their targets"
|
||||||
|
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
|
||||||
|
if empty:
|
||||||
|
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def sweep_targets(root, journal):
|
||||||
|
"""The folders this run emptied, deepest first — never one it did not touch.
|
||||||
|
|
||||||
|
Eligibility is evidence, not appearance. A folder is a target only when the
|
||||||
|
journal shows a file moved out of it or out of something beneath it, it holds
|
||||||
|
nothing but empty directories now, it is not a folder this run created, and no
|
||||||
|
journalled destination lives under it. A symlink anywhere inside disqualifies
|
||||||
|
the whole subtree: this tool never follows a link, and deleting the folder
|
||||||
|
that holds one would delete someone's mount point.
|
||||||
|
|
||||||
|
Proof carries downward: once a folder qualifies, every empty directory inside
|
||||||
|
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
|
||||||
|
a move source, so without that rule it holds its emptied parent on disk
|
||||||
|
forever — the parent can never be removed while the child stands.
|
||||||
|
"""
|
||||||
|
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||||
|
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
|
||||||
|
# An rmdir INTENT that failed left its folder standing, and the next sweep
|
||||||
|
# must be able to finish it: only a folder that is really gone is out of
|
||||||
|
# scope. Counting every intent as done made 146 live survivors permanent.
|
||||||
|
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
|
||||||
|
and not os.path.exists(r["path"])}
|
||||||
|
dsts = [r["dst"] for r in moves]
|
||||||
|
marked = set()
|
||||||
|
for r in moves:
|
||||||
|
d = os.path.dirname(r["src"])
|
||||||
|
while d.startswith(root + os.sep):
|
||||||
|
marked.add(d)
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
|
||||||
|
def removable(d):
|
||||||
|
if d in made or d in gone or os.path.islink(d):
|
||||||
|
return False
|
||||||
|
if any(p == d or p.startswith(d + os.sep) for p in dsts):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
entries = os.listdir(d)
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
for e in entries:
|
||||||
|
c = os.path.join(d, e)
|
||||||
|
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
roots = [d for d in marked if removable(d)]
|
||||||
|
targets = set(roots)
|
||||||
|
for d in roots:
|
||||||
|
for dirpath, dirnames, _ in os.walk(d):
|
||||||
|
for c in dirnames:
|
||||||
|
p = os.path.join(dirpath, c)
|
||||||
|
if removable(p):
|
||||||
|
targets.add(p)
|
||||||
|
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
|
||||||
|
|
||||||
|
|
||||||
|
def do_sweep(args, root, journal):
|
||||||
|
"""Remove the empty husks an apply left standing at the library root."""
|
||||||
|
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||||
|
if not moves:
|
||||||
|
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
|
||||||
|
" does not guess at emptiness")
|
||||||
|
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
|
||||||
|
if not (p == root or p.startswith(root + os.sep))})
|
||||||
|
if outside and not args.force:
|
||||||
|
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
|
||||||
|
f" (first: {outside[0]}) — this journal belongs to another"
|
||||||
|
" library")
|
||||||
|
targets = sweep_targets(root, journal)
|
||||||
|
if not targets:
|
||||||
|
print("swept: 0 folders — nothing this run emptied is still standing empty")
|
||||||
|
return 0
|
||||||
|
if not args.apply:
|
||||||
|
print(f"dry run: {len(targets)} folders this run emptied would be removed"
|
||||||
|
" (pass --apply to run)")
|
||||||
|
for d in targets[:5]:
|
||||||
|
print(f" would remove {d}")
|
||||||
|
return 0
|
||||||
|
removed = 0
|
||||||
|
for d in targets:
|
||||||
|
journal.record(op="rmdir", path=d)
|
||||||
|
try:
|
||||||
|
os.rmdir(d)
|
||||||
|
except OSError as e:
|
||||||
|
print(f"left in place: {d} ({e})")
|
||||||
|
continue
|
||||||
|
removed += 1
|
||||||
|
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
|
||||||
|
print(f"journal: {journal.path} — rollback recreates them with their files")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def do_rollback(args, root, journal):
|
||||||
|
"""Undo every journalled move, newest first, after a full preflight."""
|
||||||
|
moves = [r for r in journal.records if r.get("op") == "move"]
|
||||||
|
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
|
||||||
|
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
|
||||||
|
problems = []
|
||||||
|
for r in reversed(moves):
|
||||||
|
src, dst = r["src"], r["dst"]
|
||||||
|
if not args.force and not (src.startswith(root + os.sep)
|
||||||
|
or dst.startswith(root + os.sep)):
|
||||||
|
problems.append(f"outside --root: {dst}")
|
||||||
|
continue
|
||||||
|
if os.path.exists(dst) and os.path.exists(src):
|
||||||
|
problems.append(f"occupied: both paths exist, refusing to choose:"
|
||||||
|
f" {src} | {dst}")
|
||||||
|
elif os.path.exists(dst):
|
||||||
|
pass
|
||||||
|
elif os.path.exists(src):
|
||||||
|
continue # already undone
|
||||||
|
else:
|
||||||
|
problems.append(f"missing: {dst} is gone and {src} never came back")
|
||||||
|
if problems:
|
||||||
|
print(f"refusing: {len(problems)} problem(s), nothing moved back")
|
||||||
|
for p in problems[:20]:
|
||||||
|
print(" " + p)
|
||||||
|
return 2
|
||||||
|
if not args.apply:
|
||||||
|
print(f"dry run: would move {len(moves)} files back to their census paths"
|
||||||
|
f" and recreate {len(set(swept))} swept folders (pass --apply)")
|
||||||
|
return 0
|
||||||
|
undone = 0
|
||||||
|
recreated = 0
|
||||||
|
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
|
||||||
|
if not os.path.isdir(d):
|
||||||
|
os.makedirs(d, exist_ok=True)
|
||||||
|
recreated += 1
|
||||||
|
for r in reversed(moves):
|
||||||
|
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
|
||||||
|
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
|
||||||
|
os.rename(r["dst"], r["src"])
|
||||||
|
undone += 1
|
||||||
|
dirs_gone = 0
|
||||||
|
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
|
||||||
|
if d == root or not os.path.isdir(d) or os.listdir(d):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
os.rmdir(d)
|
||||||
|
dirs_gone += 1
|
||||||
|
except OSError as e:
|
||||||
|
print(f"left in place: {d} ({e})")
|
||||||
|
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
|
||||||
|
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
|
||||||
|
print(f"rolled back: {undone} files back where the census found them,"
|
||||||
|
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
|
||||||
|
metadata_note(args)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
|
||||||
|
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
|
||||||
|
ap.add_argument("--plan")
|
||||||
|
ap.add_argument("--root", default=DEFAULT_ROOT)
|
||||||
|
ap.add_argument("--journal")
|
||||||
|
ap.add_argument("--apply", action="store_true", help="actually change the library")
|
||||||
|
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
|
||||||
|
ap.add_argument("--metadata-view", default=METADATA_VIEW,
|
||||||
|
help="overlay the media servers read through; the sidecar"
|
||||||
|
" note prints only when this path is a mountpoint")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
root = os.path.abspath(args.root)
|
||||||
|
if not os.path.isdir(root):
|
||||||
|
raise Refuse(f"no library at {root}")
|
||||||
|
journal_path = args.journal or journal_default(root)
|
||||||
|
journal = Journal(journal_path, args.apply)
|
||||||
|
try:
|
||||||
|
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
|
||||||
|
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
|
||||||
|
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
|
||||||
|
raise Refuse("this journal has already been rolled back"
|
||||||
|
" (--force to replay it)")
|
||||||
|
if args.cmd in ("apply", "verify") and not args.plan:
|
||||||
|
raise Refuse(f"{args.cmd} needs --plan")
|
||||||
|
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
|
||||||
|
if args.plan else ([], [], []))
|
||||||
|
if args.cmd == "apply":
|
||||||
|
return do_apply(args, root, journal, mkdirs, keeps, moves)
|
||||||
|
if args.cmd == "verify":
|
||||||
|
return do_verify(root, journal, moves, keeps)
|
||||||
|
if args.cmd == "sweep":
|
||||||
|
return do_sweep(args, root, journal)
|
||||||
|
return do_rollback(args, root, journal)
|
||||||
|
finally:
|
||||||
|
journal.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
sys.exit(main())
|
||||||
|
except Refuse as r:
|
||||||
|
print(f"refusing: {r}")
|
||||||
|
sys.exit(2)
|
||||||
Executable
+425
@@ -0,0 +1,425 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
r"""media-audiobooks.py — a plan that re-files the audiobook library into one folder per book.
|
||||||
|
|
||||||
|
media-audiobooks.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
|
||||||
|
|
||||||
|
Plex reads this library and Jellyfin does not: its music scanner has no audiobook
|
||||||
|
concept, and the tree is indexed entirely by Plex, which matched 58,975 of the
|
||||||
|
62,879 files here into its Artist/Album/Track shape. So the authority is
|
||||||
|
`scripts/media-census.py audiobooks` — the BOOK (album row), its NARRATOR (that
|
||||||
|
book's single artist row) and its CHAPTERS (track rows) — and never the folder a
|
||||||
|
release pack made. The plan is:
|
||||||
|
|
||||||
|
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>.<ext>
|
||||||
|
... the same folder, " (2)" on the name, when two chapters
|
||||||
|
of one book carry the same name
|
||||||
|
|
||||||
|
What that buys is the one property both media servers' book scanners assume: a
|
||||||
|
folder holds exactly one book. Today 212 top-level folders are a mix of
|
||||||
|
author-named shelves and collection packs, and the books inside the packs are
|
||||||
|
split across `… [Part 1]`, `… [Part 2]` folders three or four deep.
|
||||||
|
|
||||||
|
Three laws here are the opposite of what a music shelf does, and each is forced
|
||||||
|
by what the rows actually hold:
|
||||||
|
|
||||||
|
* A chapter is NEVER renumbered. 7,556 of the 10,485 books contain at least one
|
||||||
|
chapter whose index Plex does not know, so a number written into a name would
|
||||||
|
be invented — and it would buy nothing, because both servers order playback by
|
||||||
|
the embedded track tag, not by the file name. A wrong number in a name is worse
|
||||||
|
than none: it is the number a human later trusts. A file already carrying its
|
||||||
|
own number keeps it, untouched.
|
||||||
|
* The shelf is the NARRATOR, not the author. Every book has exactly one artist row
|
||||||
|
and it is the narrator; the author survives only inside 10% of the album titles
|
||||||
|
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving the
|
||||||
|
author would mean parsing titles for one book in ten and inventing it for the
|
||||||
|
rest. One rule that always resolves beats a shelf that is half author and half
|
||||||
|
narrator; the author stays visible in the book folder's own name, which is the
|
||||||
|
title Plex shows.
|
||||||
|
* Nothing is guessed at. A chapter row with no book or no narrator parks — a
|
||||||
|
tag-scan placeholder (`[Unknown Album]`) names neither, since unrelated books
|
||||||
|
share it and filing on one merges them, and so does a chapter number the ripper
|
||||||
|
mistagged as the artist (`0.4`). An audio file Plex never matched parks, a
|
||||||
|
video container parks, and a loose file in a folder holding two books parks.
|
||||||
|
|
||||||
|
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
|
||||||
|
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
|
||||||
|
already claimed and whose journal is the rollback.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
|
||||||
|
# What a chapter file can be. .m4b is the audiobook container; the rest are the
|
||||||
|
# audio a pack ships when it was ripped as music.
|
||||||
|
AUDIO_EXT = {".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav", ".flac",
|
||||||
|
".wma", ".aiff", ".aif", ".m4v", ".aac", ".mka"}
|
||||||
|
# Never an audiobook, whatever the library it sits in says: Plex reads these as
|
||||||
|
# video, and renaming one into an audio extension would claim it is a chapter.
|
||||||
|
VIDEO_EXT = {".mkv", ".mp4", ".webm", ".avi", ".mov", ".mpg", ".mpeg", ".ts",
|
||||||
|
".m2ts", ".vob", ".ifo", ".wmv", ".iso", ".img", ".bdmv", ".ogm",
|
||||||
|
".divx"}
|
||||||
|
|
||||||
|
# Tag-scan placeholders. These strings are not identities: 1,106 chapters from
|
||||||
|
# 14 unrelated top-level folders carry the album `[Unknown Album]`, so filing on
|
||||||
|
# them would merge books that have nothing to do with each other — the exact
|
||||||
|
# shape this relayout exists to make impossible. Such a chapter parks with the
|
||||||
|
# path it has today and the fix belongs in its tags. `Various Artists` is NOT a
|
||||||
|
# placeholder: that is what a multi-cast read carries, and the one book here
|
||||||
|
# with 1,189 chapters under it is genuinely one book.
|
||||||
|
PLACEHOLDER_BOOK = {"[unknown album]", "unknown album", "no title", "untitled",
|
||||||
|
"[untitled]", "unknown title", "n/a", "-"}
|
||||||
|
PLACEHOLDER_NARRATOR = {"[unknown artist]", "unknown artist", "unknown author",
|
||||||
|
"<unknown>", "[unknown]", "unknown"}
|
||||||
|
|
||||||
|
|
||||||
|
def identity(value, placeholders, needs_letter=False):
|
||||||
|
"""The tag value Plex matched, when it names something; '' when it does not.
|
||||||
|
|
||||||
|
A trailing parenthesized group is stripped for the test only, because rippers
|
||||||
|
glue timestamps to the placeholder (`Unknown album (03/03/2015 21:39:55)`).
|
||||||
|
|
||||||
|
`needs_letter` is for a NARRATOR alone: a shelf is a person's name, and 272
|
||||||
|
chapters here carry a chapter number in the artist tag (`0.4`, `31`). The same
|
||||||
|
test must never judge a BOOK title, which may honestly be `1984`.
|
||||||
|
"""
|
||||||
|
v = (value or "").strip()
|
||||||
|
bare = re.sub(r"\s*\([^()]*\)\s*$", "", v).strip().casefold()
|
||||||
|
if bare in placeholders or (needs_letter and not any(c.isalpha() for c in v)):
|
||||||
|
return ""
|
||||||
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
# The characters Plex documents as breaking playback. They are dropped, not
|
||||||
|
# substituted: a replacement character is a guess at what the title meant.
|
||||||
|
BAD_CHARS = set('<>:"/\\|?*')
|
||||||
|
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
|
||||||
|
r"|lost\+found)(/|$)")
|
||||||
|
|
||||||
|
ARGS = None
|
||||||
|
HELD = {} # path -> (bytes, ext)
|
||||||
|
ROWS = {} # path -> the chapter row Plex matched
|
||||||
|
PLACED = {} # chapter path -> the destination the plan gave it
|
||||||
|
CLAIMED = set() # every destination path the plan has handed out
|
||||||
|
MOVES, KEEPS = [], []
|
||||||
|
PARKED = defaultdict(list) # reason -> [path]
|
||||||
|
NOTES = defaultdict(list) # section -> [line]
|
||||||
|
AUDIO_BY_DIR = defaultdict(set) # source dir -> chapter files a row names
|
||||||
|
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding chapters
|
||||||
|
FOLDERS_SEEN = {} # (dir, depth) -> book folders placed below it
|
||||||
|
SUBTREE_DEPTH = 2 # how far below a folder its chapters may sit
|
||||||
|
BOOKS = {} # (narrator, book, year) -> the book group
|
||||||
|
|
||||||
|
AMBIGUOUS = object()
|
||||||
|
|
||||||
|
|
||||||
|
def dominant(values):
|
||||||
|
"""The spelling the rows mostly agree on: the most frequent value, ties broken
|
||||||
|
by sort so the answer never depends on the order rows arrived in."""
|
||||||
|
counts = Counter(values)
|
||||||
|
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
|
||||||
|
|
||||||
|
|
||||||
|
def rel(path):
|
||||||
|
"""The path as seen from the library root."""
|
||||||
|
return os.path.relpath(path, ARGS.root)
|
||||||
|
|
||||||
|
|
||||||
|
def safe(name):
|
||||||
|
"""A name every client can carry: no reserved character, nothing unprintable,
|
||||||
|
no trailing dot or space, and short enough to rename."""
|
||||||
|
out = "".join(c for c in (name or "")
|
||||||
|
if c not in BAD_CHARS and (c.isprintable() or c == " "))
|
||||||
|
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
|
||||||
|
while out and len(out.encode("utf-8", "ignore")) > 200:
|
||||||
|
out = out[:-4].rstrip()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def assign(src, folder, name, ext, kind, note):
|
||||||
|
"""Book one file into `folder` under `name`, renamed if that exact destination
|
||||||
|
is already taken — by another file in the plan, or by a file standing there
|
||||||
|
today. Two files claiming one path is the fault that costs a re-run of a 60k
|
||||||
|
plan, and so is a plan that only works if the renames happen in the right
|
||||||
|
order: the executor refuses any target that is occupied, so an occupied path
|
||||||
|
is never a destination, even when its occupant is scheduled to leave.
|
||||||
|
A file already standing where the contract puts it is a keep, not a move."""
|
||||||
|
stem = safe(name) or "file"
|
||||||
|
dst = os.path.join(folder, stem + ext)
|
||||||
|
if dst == src:
|
||||||
|
CLAIMED.add(dst.lower()) # it is standing on its own target
|
||||||
|
KEEPS.append((src, dst, kind))
|
||||||
|
return dst
|
||||||
|
# A file already in this folder under this name — or under the numbered
|
||||||
|
# variant the collision rule gives it — keeps the name it has. Renumbering it
|
||||||
|
# every run would turn the plan into a shuffle instead of a fixed point.
|
||||||
|
base = os.path.splitext(os.path.basename(src))[0]
|
||||||
|
if os.path.dirname(src) == folder and (base == stem
|
||||||
|
or base.startswith(stem + " (")):
|
||||||
|
CLAIMED.add(src.lower())
|
||||||
|
KEEPS.append((src, src, kind))
|
||||||
|
return src
|
||||||
|
n = 1
|
||||||
|
while dst.lower() in CLAIMED:
|
||||||
|
n += 1
|
||||||
|
dst = os.path.join(folder, f"{stem} ({n}){ext}")
|
||||||
|
if n > 1:
|
||||||
|
NOTES[note].append(rel(dst))
|
||||||
|
CLAIMED.add(dst.lower())
|
||||||
|
MOVES.append((src, dst, kind))
|
||||||
|
return dst
|
||||||
|
|
||||||
|
|
||||||
|
def park(path, reason):
|
||||||
|
"""Hold a file where it is, under the park folder, keeping its relative path."""
|
||||||
|
PARKED[reason].append(path)
|
||||||
|
dst = os.path.join(ARGS.park_dir, rel(path))
|
||||||
|
if dst != path:
|
||||||
|
CLAIMED.add(dst.lower())
|
||||||
|
MOVES.append((path, dst, "park"))
|
||||||
|
|
||||||
|
|
||||||
|
def under_park(path):
|
||||||
|
"""True for a file already parked: that pile is for a human to review, and
|
||||||
|
re-parking it every run would only push it deeper."""
|
||||||
|
return path.startswith(ARGS.park_dir + os.sep)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------ books
|
||||||
|
def group_books():
|
||||||
|
"""Every matched chapter into one group per (narrator, book, year).
|
||||||
|
|
||||||
|
The key is the identity Plex asserts in its own database, not the folder the
|
||||||
|
file sits in: one book arrives as four part folders inside a collection pack,
|
||||||
|
and two copies of one book in one pack are one book to Plex, whose album item
|
||||||
|
IS that narrator, title and year. The year is part of the key so a 1965
|
||||||
|
recording and a 2020 recording that share a title stay two books.
|
||||||
|
"""
|
||||||
|
for path, r in sorted(ROWS.items()):
|
||||||
|
if path not in HELD or under_park(path) or TRASH_DIR.search(path):
|
||||||
|
continue
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
AUDIO_BY_DIR[d].add(path)
|
||||||
|
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above Part 1/Part 2 sees both
|
||||||
|
narrator = identity(r["narrator"], PLACEHOLDER_NARRATOR, needs_letter=True)
|
||||||
|
book = identity(r["book"], PLACEHOLDER_BOOK)
|
||||||
|
if not book:
|
||||||
|
park(path, "a chapter row that names no book — Plex's album title is a "
|
||||||
|
"placeholder like [Unknown Album], so filing on it would "
|
||||||
|
"merge unrelated books onto one shelf")
|
||||||
|
continue
|
||||||
|
if not narrator:
|
||||||
|
park(path, "a chapter row that names no narrator — the artist row is a "
|
||||||
|
"placeholder like [Unknown Artist] or a chapter number a "
|
||||||
|
"ripper mistagged as the artist, so no shelf owns it")
|
||||||
|
continue
|
||||||
|
key = (narrator.casefold(), book.casefold(), (r["year"] or "").strip())
|
||||||
|
g = BOOKS.setdefault(key, {"narrators": [], "books": [], "year": key[2],
|
||||||
|
"members": [], "indexed": 0})
|
||||||
|
g["members"].append(path)
|
||||||
|
g["narrators"].append(narrator)
|
||||||
|
g["books"].append(book)
|
||||||
|
if str(r["track"] or "").isdigit():
|
||||||
|
g["indexed"] += 1
|
||||||
|
|
||||||
|
|
||||||
|
def book_folder(g):
|
||||||
|
"""The shelf Plex's own rows name: <narrator>/<book> (<year>). None when a
|
||||||
|
name is made of characters no path can hold — inventing a shelf is filing it
|
||||||
|
wrong, so those chapters park instead."""
|
||||||
|
narrator = safe(dominant(g["narrators"]))
|
||||||
|
book = safe(dominant(g["books"]))
|
||||||
|
if not narrator or not book:
|
||||||
|
return None
|
||||||
|
return os.path.join(ARGS.root, narrator,
|
||||||
|
f"{book} ({g['year']})" if g["year"] else book)
|
||||||
|
|
||||||
|
|
||||||
|
def place_books():
|
||||||
|
"""One folder per book; every chapter keeps the name it has today."""
|
||||||
|
for key in sorted(BOOKS):
|
||||||
|
g = BOOKS[key]
|
||||||
|
folder = book_folder(g)
|
||||||
|
if folder is None:
|
||||||
|
for path in sorted(g["members"]):
|
||||||
|
park(path, "a book or narrator Plex spells with characters no "
|
||||||
|
"folder name can hold")
|
||||||
|
continue
|
||||||
|
for path in sorted(g["members"]):
|
||||||
|
ext = HELD[path][1]
|
||||||
|
base = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
# The name the ripper wrote, unchanged: the track tag inside the file
|
||||||
|
# is what orders playback, and 72% of the books here have chapters no
|
||||||
|
# row numbers, so any number added here would be invented.
|
||||||
|
PLACED[path] = assign(path, folder, base, ext, "chapter",
|
||||||
|
"two chapters of one book carried the same name")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------- sidecars
|
||||||
|
def folders_below(d, depth):
|
||||||
|
"""The book folders the chapters in `d`, or within `depth` directories below
|
||||||
|
it, were placed in. Memoised, because every loose file in a big collection
|
||||||
|
folder asks the same question about it."""
|
||||||
|
if (d, depth) not in FOLDERS_SEEN:
|
||||||
|
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
|
||||||
|
if a in PLACED}
|
||||||
|
if depth:
|
||||||
|
for child in CHILD_OF.get(d, ()):
|
||||||
|
seen |= folders_below(child, depth - 1)
|
||||||
|
FOLDERS_SEEN[(d, depth)] = seen
|
||||||
|
return FOLDERS_SEEN[(d, depth)]
|
||||||
|
|
||||||
|
|
||||||
|
def owner_folder(d):
|
||||||
|
"""The book folder that owns this directory — the first place up the tree
|
||||||
|
where chapters sit, in the directory itself or within SUBTREE_DEPTH levels
|
||||||
|
below it, which is how a pack whose parts are `… [Part 1]`/`… [Part 2]` owns
|
||||||
|
the cover art lying above them. None when no chapter is near enough to say,
|
||||||
|
AMBIGUOUS when the chapters there belong to more than one book."""
|
||||||
|
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
|
||||||
|
folders = folders_below(d, SUBTREE_DEPTH)
|
||||||
|
if folders:
|
||||||
|
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def place_other():
|
||||||
|
"""Everything that is not a chapter Plex matched: covers, booklets, EPUBs,
|
||||||
|
logs, checksums, playlists — and the clutter that is none of those."""
|
||||||
|
for path, (_size, ext) in sorted(HELD.items()):
|
||||||
|
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
|
||||||
|
continue
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
base = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
if ext in VIDEO_EXT:
|
||||||
|
park(path, "a video container inside the audiobook library: Plex reads "
|
||||||
|
"it as video, and renaming it would claim it is a chapter")
|
||||||
|
elif ext in AUDIO_EXT:
|
||||||
|
park(path, "an audio file no chapter row names — it is not in the "
|
||||||
|
"library Plex matched, so nothing says which book it is")
|
||||||
|
else:
|
||||||
|
owner = owner_folder(d)
|
||||||
|
if owner is AMBIGUOUS:
|
||||||
|
park(path, "a file whose folder holds more than one book, so "
|
||||||
|
"nothing says which one it belongs to")
|
||||||
|
elif owner:
|
||||||
|
assign(path, owner, base, ext,
|
||||||
|
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
|
||||||
|
"one book folder would have carried two files of the same"
|
||||||
|
" sidecar name")
|
||||||
|
else:
|
||||||
|
park(path, "not an audiobook file, and no book's folder lies with"
|
||||||
|
" it to carry it")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- outputs
|
||||||
|
def write_plan():
|
||||||
|
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
||||||
|
Parents first: the executor creates directories in the order they are listed."""
|
||||||
|
chain = set()
|
||||||
|
for _s, dst, _k in MOVES:
|
||||||
|
d = os.path.dirname(dst)
|
||||||
|
while d and d != ARGS.root and d != os.path.dirname(d):
|
||||||
|
chain.add(d)
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
||||||
|
with open(ARGS.out, "w") as fh:
|
||||||
|
fh.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in dirs:
|
||||||
|
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
|
||||||
|
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||||
|
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
|
||||||
|
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
||||||
|
parked = [m for m in MOVES if m[2] == "park"]
|
||||||
|
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
|
||||||
|
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
|
||||||
|
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
|
||||||
|
f" {len(dirs)} folders -> {ARGS.out}")
|
||||||
|
|
||||||
|
|
||||||
|
def write_exceptions():
|
||||||
|
with open(ARGS.exceptions, "w") as fh:
|
||||||
|
fh.write(f"Audiobook relayout exceptions for {ARGS.root}\n\n")
|
||||||
|
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
|
||||||
|
"relative path it has today. Nothing is deleted and nothing is\n"
|
||||||
|
"overwritten: a parked file is one rename from wherever the plan is\n"
|
||||||
|
"pointed next.\n\n")
|
||||||
|
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
|
||||||
|
rs = PARKED[reason]
|
||||||
|
fh.write(f"{len(rs):>6} files {reason}\n")
|
||||||
|
for p in sorted(rs, key=rel)[:400]:
|
||||||
|
fh.write(f" {rel(p)}\n")
|
||||||
|
if len(rs) > 400:
|
||||||
|
fh.write(f" … and {len(rs) - 400} more\n")
|
||||||
|
fh.write("\n")
|
||||||
|
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
|
||||||
|
lines = sorted(set(NOTES[section]))
|
||||||
|
fh.write(f"{len(lines):>6} {section}\n")
|
||||||
|
for line in lines[:200]:
|
||||||
|
fh.write(f" {line}\n")
|
||||||
|
if len(lines) > 200:
|
||||||
|
fh.write(f" … and {len(lines) - 200} more\n")
|
||||||
|
fh.write("\n")
|
||||||
|
unindexed = sorted(f"{safe(dominant(g['narrators']))}/{safe(dominant(g['books']))}"
|
||||||
|
for g in BOOKS.values()
|
||||||
|
if 0 < g["indexed"] < len(g["members"]))
|
||||||
|
if unindexed:
|
||||||
|
fh.write(f"{len(unindexed):>6} books have chapters Plex does not number. Their\n")
|
||||||
|
fh.write(" files keep the names they carry: playback order comes from the\n"
|
||||||
|
" track tag inside each file, and a number written here would be\n"
|
||||||
|
" invented for the chapters with no index. Fix it in the tags, not\n"
|
||||||
|
" in the names — Plex re-reads tags on a scan and never names files.\n\n")
|
||||||
|
fh.write(" Playlists move with their book and their contents are never\n"
|
||||||
|
" rewritten: the paths inside an .m3u go stale.\n")
|
||||||
|
print(f"exceptions -> {ARGS.exceptions}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
global ARGS
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
|
||||||
|
"/mnt/entropy/Audiobooks"))
|
||||||
|
ap.add_argument("--enum", default="/tmp/audiobooks-enum.tsv",
|
||||||
|
help="the enumerate TSV (bytes, ext, path)")
|
||||||
|
ap.add_argument("--rows", default="/tmp/plex-audiobooks.tsv",
|
||||||
|
help="the chapter rows scripts/media-census.py audiobooks wrote")
|
||||||
|
ap.add_argument("--out", default="/tmp/audiobooks-plan.tsv")
|
||||||
|
ap.add_argument("--exceptions", default="/tmp/audiobooks-exceptions.txt")
|
||||||
|
ap.add_argument("--park", default="_unsorted",
|
||||||
|
help="the folder inside the root that holds what cannot be filed")
|
||||||
|
ARGS = ap.parse_args()
|
||||||
|
ARGS.root = os.path.normpath(ARGS.root)
|
||||||
|
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
|
||||||
|
read_inputs()
|
||||||
|
# Every path a file occupies today is claimed before a single destination is
|
||||||
|
# handed out: the executor refuses an occupied target, so a plan that aimed at
|
||||||
|
# one would only fail after the renames that were supposed to clear it.
|
||||||
|
CLAIMED.update(p.lower() for p in HELD)
|
||||||
|
group_books()
|
||||||
|
place_books()
|
||||||
|
place_other()
|
||||||
|
write_plan()
|
||||||
|
write_exceptions()
|
||||||
|
|
||||||
|
|
||||||
|
def read_inputs():
|
||||||
|
with open(ARGS.enum) as fh:
|
||||||
|
for line in fh:
|
||||||
|
parts = line.rstrip("\n").split("\t")
|
||||||
|
if len(parts) >= 3:
|
||||||
|
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
|
||||||
|
with open(ARGS.rows, newline="") as fh:
|
||||||
|
for row in csv.DictReader(fh, delimiter="\t"):
|
||||||
|
if row.get("file"):
|
||||||
|
ROWS[row["file"]] = row
|
||||||
|
gone = [p for p in ROWS if p not in HELD]
|
||||||
|
if gone:
|
||||||
|
NOTES["a chapter row whose file the census no longer sees"].extend(gone)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+518
@@ -0,0 +1,518 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
r"""Read-only census of a media library — the inputs a relayout plan is built from.
|
||||||
|
|
||||||
|
media-census.py enumerate [--root DIR] [--out TSV]
|
||||||
|
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
|
||||||
|
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
|
||||||
|
media-census.py jellyfin [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
|
||||||
|
media-census.py audiobooks [--root DIR] [--db PATH] [--out TSV]
|
||||||
|
|
||||||
|
Run it on the machine that mounts the library. Nothing here writes inside the
|
||||||
|
library and nothing is deleted; the only outputs are the TSVs below, and the
|
||||||
|
scripts/media-plan.py and scripts/media-shows.py read them instead of touching media.
|
||||||
|
|
||||||
|
enum bytes <TAB> ext <TAB> path every file under the root
|
||||||
|
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
|
||||||
|
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
|
||||||
|
one row per video-container file; `audio` is "codec:lang+codec:lang",
|
||||||
|
`subs` the subtitle-stream count, `spare` the audio streams carrying
|
||||||
|
neither a default nor a forced tag
|
||||||
|
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
|
||||||
|
the titles Plex already matched — the authority over any filename guess
|
||||||
|
|
||||||
|
jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
|
||||||
|
<TAB> tmdb <TAB> title <TAB> tvdb (see JF_COLS)
|
||||||
|
one row per episode Jellyfin has matched and numbered, its path mapped
|
||||||
|
from the container's view back to the host; `year`, `tmdb` and `tvdb`
|
||||||
|
are the SERIES' values, `title` is the episode title the row carries,
|
||||||
|
`extra` is Jellyfin's own extra type when the row is bonus material
|
||||||
|
rather than an episode
|
||||||
|
|
||||||
|
music file <TAB> artist <TAB> album <TAB> year <TAB> disc <TAB> track <TAB> title
|
||||||
|
<TAB> mb <TAB> performer (see MUSIC_COLS)
|
||||||
|
one row per track Jellyfin has tagged: the tag read is what the row IS,
|
||||||
|
so `artist` is the album artist (the track artist when that is empty) and
|
||||||
|
`album`/`year` are the album item's provider values when the folder is an
|
||||||
|
album item and the tags' otherwise; `mb` is the album's MusicBrainz
|
||||||
|
release group, `performer` the track artist the tags carry
|
||||||
|
|
||||||
|
audiobooks file <TAB> narrator <TAB> book <TAB> year <TAB> track <TAB> chapter
|
||||||
|
(see AB_COLS)
|
||||||
|
one row per chapter Plex matched under the audiobook root: `book` is the
|
||||||
|
album row Plex built from the files' own tags, `narrator` that book's
|
||||||
|
artist row — Plex keeps exactly one artist per book, which is why it, and
|
||||||
|
not an author parsed out of a title, names the shelf. `track` is empty
|
||||||
|
wherever Plex has no chapter number, which is most of this library.
|
||||||
|
|
||||||
|
`probe` is the slow step (every container is opened for its header, ~10 min for
|
||||||
|
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
|
||||||
|
live service must not starve it. `enumerate` is a stat pass, so re-censusing
|
||||||
|
before a plan is cheap — and a plan is only ever as good as these files.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
|
||||||
|
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
|
||||||
|
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
|
||||||
|
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
||||||
|
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
|
||||||
|
# Trash and thumbnail caches are not library content, and this repo's trash policy
|
||||||
|
# means .AGENTTRASH can sit inside a movie folder at any time.
|
||||||
|
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
|
||||||
|
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
|
||||||
|
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
|
||||||
|
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
|
||||||
|
"parent_title", "parent_orig", "parent_year", "parent_edition",
|
||||||
|
"container", "vcodec", "acodec", "width", "height"]
|
||||||
|
PLEX_SQL = """
|
||||||
|
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
|
||||||
|
coalesce(mi.edition_title,''), coalesce(mi.year,0),
|
||||||
|
coalesce(par.title,''), coalesce(par.original_title,''),
|
||||||
|
coalesce(par.year,0), coalesce(par.edition_title,''),
|
||||||
|
m.container, m.video_codec, m.audio_codec, m.width, m.height
|
||||||
|
from media_parts p
|
||||||
|
join media_items m on m.id = p.media_item_id
|
||||||
|
join metadata_items mi on mi.id = m.metadata_item_id
|
||||||
|
left join metadata_items par on par.id = mi.parent_id
|
||||||
|
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
|
||||||
|
"""
|
||||||
|
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
|
||||||
|
JF_COLS = ["file", "series", "year", "season", "episode", "extra",
|
||||||
|
"tmdb", "title", "tvdb"]
|
||||||
|
# Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
|
||||||
|
JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
|
||||||
|
"MediaBrowser.Controller.Entities.TV.Season",
|
||||||
|
"MediaBrowser.Controller.Entities.TV.Series")
|
||||||
|
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType,
|
||||||
|
IsVirtualItem, Name from BaseItems where Type = ? and Path like ?""")
|
||||||
|
|
||||||
|
# The music tree is these two entity classes. The track row carries the tags
|
||||||
|
# Jellyfin read out of the file; the album row carries what the provider matched,
|
||||||
|
# which is the better folder name of the two.
|
||||||
|
JF_AUDIO = "MediaBrowser.Controller.Entities.Audio.Audio"
|
||||||
|
JF_ALBUM = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
|
||||||
|
MUSIC_COLS = ["file", "artist", "album", "year", "disc", "track", "title", "mb",
|
||||||
|
"performer"]
|
||||||
|
MUSIC_SQL = ("""select b.Path, b.AlbumArtists, b.Artists, b.Album, b.ProductionYear,
|
||||||
|
b.ParentIndexNumber, b.IndexNumber, b.Name, b.Id,
|
||||||
|
a.Name, a.ProductionYear, a.Id
|
||||||
|
from BaseItems b
|
||||||
|
left join BaseItems a on a.Id = b.ParentId and a.Type = ?
|
||||||
|
where b.Type = ? and b.Path like ?""")
|
||||||
|
# The album's identity, in preference order. A release group is the album; a bare
|
||||||
|
# release id is one reissue of it.
|
||||||
|
MUSIC_PROVIDERS = ("MusicBrainzReleaseGroup", "MusicBrainzAlbum")
|
||||||
|
|
||||||
|
# Plex's audiobook tree is the same Artist/Album/Track shape as its music tree:
|
||||||
|
# the leaf (10) is a CHAPTER, its parent (9) is the BOOK, and that book's artist
|
||||||
|
# (8) is the NARRATOR. The track's own `index` is exported because the exceptions
|
||||||
|
# report counts how many books have chapters Plex does not number — the reason the
|
||||||
|
# relayout refuses to invent numbers into file names.
|
||||||
|
PLEX_ARTIST, PLEX_ALBUM, PLEX_TRACK = 8, 9, 10
|
||||||
|
AB_COLS = ["file", "narrator", "book", "year", "track", "chapter"]
|
||||||
|
AB_SQL = f"""
|
||||||
|
select p.file, coalesce(ar.title,''), coalesce(al.title,''), coalesce(al.year,0),
|
||||||
|
coalesce(mi."index",-1), coalesce(mi.title,'')
|
||||||
|
from media_parts p
|
||||||
|
join media_items m on m.id = p.media_item_id
|
||||||
|
join metadata_items mi on mi.id = m.metadata_item_id
|
||||||
|
join metadata_items al on al.id = mi.parent_id and al.metadata_type = {PLEX_ALBUM}
|
||||||
|
left join metadata_items ar on ar.id = al.parent_id
|
||||||
|
and ar.metadata_type = {PLEX_ARTIST}
|
||||||
|
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
|
||||||
|
and al.deleted_at is null
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def enumerate_lib(root, out):
|
||||||
|
"""One row per real file under root: size, extension, path."""
|
||||||
|
rows = []
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
dirnames[:] = [d for d in dirnames
|
||||||
|
if not SKIP_DIR.search(os.path.join(dirpath, d))]
|
||||||
|
for fn in filenames:
|
||||||
|
p = os.path.join(dirpath, fn)
|
||||||
|
if SKIP_DIR.search(p):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
st = os.lstat(p)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
if not os.path.isfile(p) or os.path.islink(p):
|
||||||
|
continue # links and sockets are not library content
|
||||||
|
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
for size, ext, p in rows:
|
||||||
|
fh.write(f"{size}\t{ext}\t{p}\n")
|
||||||
|
vid = [r for r in rows if r[1] in PROBE_EXT]
|
||||||
|
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
|
||||||
|
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def probe_one(path):
|
||||||
|
"""ffprobe header read -> the probed TSV row (never the whole file)."""
|
||||||
|
try:
|
||||||
|
os.nice(19) # a census must never outrank the media server
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
size = os.stat(path).st_size
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
|
||||||
|
"-show_format", "-show_streams", path]
|
||||||
|
try:
|
||||||
|
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
|
||||||
|
data = json.loads(done.stdout or "{}")
|
||||||
|
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
|
||||||
|
json.JSONDecodeError, OSError):
|
||||||
|
data = {}
|
||||||
|
st = data.get("streams", [])
|
||||||
|
fmt = data.get("format", {})
|
||||||
|
v = next((s for s in st if s.get("codec_type") == "video"), None)
|
||||||
|
aud = [s for s in st if s.get("codec_type") == "audio"]
|
||||||
|
subs = [s for s in st if s.get("codec_type") == "subtitle"]
|
||||||
|
fps = "-"
|
||||||
|
if v and v.get("r_frame_rate"):
|
||||||
|
try:
|
||||||
|
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
|
||||||
|
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def num(x):
|
||||||
|
try:
|
||||||
|
return round(float(x))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
return [size, path, num(fmt.get("duration")),
|
||||||
|
(v or {}).get("codec_name", "-"),
|
||||||
|
f"{v['width']}x{v['height']}" if v else "-",
|
||||||
|
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
|
||||||
|
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
|
||||||
|
len(subs),
|
||||||
|
sum(1 for a in aud if not re.search(
|
||||||
|
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
|
||||||
|
fmt.get("format_name", "-") or "-"]
|
||||||
|
|
||||||
|
|
||||||
|
def probe(enum, out, jobs):
|
||||||
|
"""Probe every video-container row of the enum TSV, in parallel."""
|
||||||
|
paths = []
|
||||||
|
with open(enum) as fh:
|
||||||
|
for line in fh:
|
||||||
|
f = line.rstrip("\n").split("\t", 2)
|
||||||
|
if len(f) == 3 and f[1] in PROBE_EXT:
|
||||||
|
paths.append(f[2])
|
||||||
|
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
|
||||||
|
done = unreadable = 0
|
||||||
|
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
|
||||||
|
for row in pool.map(probe_one, paths):
|
||||||
|
if row is None:
|
||||||
|
continue
|
||||||
|
if row[3] == "-":
|
||||||
|
unreadable += 1 # header unreadable: the row still gets written
|
||||||
|
done += 1
|
||||||
|
fh.write("\t".join(str(x) for x in row) + "\n")
|
||||||
|
if done % 250 == 0:
|
||||||
|
print(f" {done}/{len(paths)}", flush=True)
|
||||||
|
print(f"probed={done} unreadable={unreadable} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def plex(root, db, out):
|
||||||
|
"""Plex's matched titles for files under root, from its DB, read-only."""
|
||||||
|
uri = f"file:{db}?mode=ro&immutable=1"
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(uri, uri=True)
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
|
||||||
|
"Plex runs rootless here: run this as the container's owner, never "
|
||||||
|
"with sudo — rootless storage is per-user, so root sees nothing.")
|
||||||
|
with open(out, "w", newline="") as fh:
|
||||||
|
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||||
|
w.writerow(PLEX_COLS)
|
||||||
|
n = 0
|
||||||
|
for r in con.execute(PLEX_SQL, (root + "/%",)):
|
||||||
|
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
|
||||||
|
n += 1
|
||||||
|
print(f"plex-matched={n} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def jellyfin(root, db, container, out):
|
||||||
|
"""The episodes Jellyfin has already numbered, for the library at root.
|
||||||
|
|
||||||
|
Jellyfin stores each path as the CONTAINER sees it (its quadlet binds the
|
||||||
|
share at /data), so `container` is the in-container prefix that stands for
|
||||||
|
root and every exported path is mapped back to the host before it is written
|
||||||
|
— the planner reads host paths, because it plans moves on the array.
|
||||||
|
|
||||||
|
The numbering is what matters, not the name: a release group's file name says
|
||||||
|
nothing about which episode a file is, and Jellyfin has already answered that
|
||||||
|
question against a provider. Season/episode and the episode's own title come
|
||||||
|
from the episode row; the series name, its year and its TMDb and Tvdb ids come
|
||||||
|
from the Series row the episode climbs to, because those are the only place
|
||||||
|
Jellyfin keeps them.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
|
||||||
|
"It is a WAL database: copy it with its -wal and -shm beside it and"
|
||||||
|
" point --db at the copy. Never sudo it — the data dir belongs to"
|
||||||
|
" the container's owner, and rootless storage is per-user.")
|
||||||
|
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
|
||||||
|
need = {"Id", "ParentId", "Type", "Path", "IndexNumber", "ParentIndexNumber",
|
||||||
|
"ExtraType", "IsVirtualItem", "ProductionYear", "Data"}
|
||||||
|
if not need <= have:
|
||||||
|
sys.exit(f"{db} is not a Jellyfin library database (BaseItems has no"
|
||||||
|
f" {','.join(sorted(need - have))})")
|
||||||
|
node = {}
|
||||||
|
for i, pid, typ, name, year, data in con.execute(
|
||||||
|
"select Id, ParentId, Type, Name, ProductionYear, Data from BaseItems"
|
||||||
|
" where Type in (?,?,?)", JF_TV):
|
||||||
|
node[i] = (pid, typ, name, year, data)
|
||||||
|
|
||||||
|
# Jellyfin 12 keeps provider ids in their own table; older builds kept them in
|
||||||
|
# the row's Data blob. Both shapes are read, because the tvdb id the series
|
||||||
|
# naming depends on is wherever this build happened to write it.
|
||||||
|
prov = {}
|
||||||
|
if "BaseItemProviders" in {r[0] for r in con.execute(
|
||||||
|
"select name from sqlite_master where type='table'")}:
|
||||||
|
for iid, pid, val in con.execute(
|
||||||
|
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
|
||||||
|
" where ProviderId in ('Tmdb', 'Tvdb')"):
|
||||||
|
prov[(iid, pid)] = val
|
||||||
|
|
||||||
|
def provider_of(item_id, want):
|
||||||
|
got = prov.get((item_id, want))
|
||||||
|
if got:
|
||||||
|
return str(got)
|
||||||
|
try:
|
||||||
|
blob = json.loads(node[item_id][4] or "{}").get("ProviderIds") or {}
|
||||||
|
except (KeyError, ValueError, TypeError):
|
||||||
|
return ""
|
||||||
|
return str(blob.get(want, "") or "")
|
||||||
|
|
||||||
|
def series_of(item_id):
|
||||||
|
"""Climb episode -> season -> series; empty when the chain is broken."""
|
||||||
|
for _ in range(4):
|
||||||
|
if item_id not in node:
|
||||||
|
return "", "", "", ""
|
||||||
|
pid, typ = node[item_id][0], node[item_id][1]
|
||||||
|
if typ.endswith(".Series"):
|
||||||
|
return (node[item_id][2] or "", node[item_id][3] or "",
|
||||||
|
provider_of(item_id, "Tmdb"), provider_of(item_id, "Tvdb"))
|
||||||
|
item_id = pid
|
||||||
|
return "", "", "", ""
|
||||||
|
|
||||||
|
# The fleet's quadlets bind the share at /data, so the container's view of
|
||||||
|
# this library is /data/<the library's own folder name> unless told otherwise.
|
||||||
|
if not container:
|
||||||
|
container = "/data/" + os.path.basename(os.path.normpath(root))
|
||||||
|
prefix = container.rstrip("/") + "/"
|
||||||
|
written = skipped = 0
|
||||||
|
with open(out, "w", newline="") as fh:
|
||||||
|
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||||
|
w.writerow(JF_COLS)
|
||||||
|
for iid, path, idx, pidx, extra, virt, epname in con.execute(
|
||||||
|
JF_SQL, (JF_TV[0], prefix + "%")):
|
||||||
|
if virt or not path or not path.startswith(prefix):
|
||||||
|
skipped += 1 # a virtual item is a variant, not a file
|
||||||
|
continue
|
||||||
|
sname, syear, tmdb, tvdb = series_of(iid)
|
||||||
|
w.writerow([root + "/" + path[len(prefix):],
|
||||||
|
sname.replace("\t", " "), syear,
|
||||||
|
"" if pidx is None else pidx,
|
||||||
|
"" if idx is None else idx,
|
||||||
|
(extra or "").replace("\t", " "), tmdb,
|
||||||
|
(epname or "").replace("\t", " "), tvdb])
|
||||||
|
written += 1
|
||||||
|
print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def music(root, db, container, out):
|
||||||
|
"""The tracks Jellyfin has already tagged, for the library at root.
|
||||||
|
|
||||||
|
Jellyfin reads music metadata out of the file itself, so an Audio row IS the
|
||||||
|
tag read, done once, by the vendor's own parser: its album-artist, album,
|
||||||
|
year, disc and track columns are what the embedded tags say and its Name is
|
||||||
|
the track title. That makes this export the whole authority for a music
|
||||||
|
relayout — the path a ripper chose is not evidence of anything, and the
|
||||||
|
folder Jellyfin put a track under is often a release pack, not an album.
|
||||||
|
|
||||||
|
Two gaps this export answers from the row rather than the tree. Many tracks
|
||||||
|
sit under a folder that is not an album item at all (a disc folder, or a
|
||||||
|
mixed one), so the album name and year fall back to the tags when the album
|
||||||
|
row is absent. And an album's MusicBrainz ids are not in the row's own Data
|
||||||
|
— the current schema leaves that empty for music — they are rows of
|
||||||
|
BaseItemProviders keyed by the item's Id.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
|
||||||
|
"It is a WAL database: copy it with its -wal and -shm beside it and"
|
||||||
|
" point --db at the copy. Never sudo it — the data dir belongs to"
|
||||||
|
" the container's owner, and rootless storage is per-user.")
|
||||||
|
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
|
||||||
|
need = {"Id", "ParentId", "Type", "Path", "Name", "IndexNumber",
|
||||||
|
"ParentIndexNumber", "ProductionYear", "Album", "AlbumArtists",
|
||||||
|
"Artists"}
|
||||||
|
if not need <= have:
|
||||||
|
sys.exit(f"{db} is not a Jellyfin library database with music columns"
|
||||||
|
f" (BaseItems has no {','.join(sorted(need - have))})")
|
||||||
|
# Provider ids, best identity first: the release group is the album, a bare
|
||||||
|
# release id is one reissue of it and never overrides a group.
|
||||||
|
mb = {}
|
||||||
|
if "BaseItemProviders" in {r[0] for r in con.execute(
|
||||||
|
"select name from sqlite_master where type='table'")}:
|
||||||
|
rank = {p: i for i, p in enumerate(MUSIC_PROVIDERS)}
|
||||||
|
best = {}
|
||||||
|
for iid, pid, val in con.execute(
|
||||||
|
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
|
||||||
|
" where ProviderId in (?, ?)", MUSIC_PROVIDERS):
|
||||||
|
if iid not in best or rank[pid] < best[iid][0]:
|
||||||
|
best[iid] = (rank[pid], val)
|
||||||
|
mb = {k: v[1] for k, v in best.items()}
|
||||||
|
|
||||||
|
# The fleet's quadlets bind the share at /data, so the container's view of
|
||||||
|
# this library is /data/<the library's own folder name> unless told otherwise.
|
||||||
|
if not container:
|
||||||
|
container = "/data/" + os.path.basename(os.path.normpath(root))
|
||||||
|
prefix = container.rstrip("/") + "/"
|
||||||
|
written = skipped = 0
|
||||||
|
with open(out, "w", newline="") as fh:
|
||||||
|
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||||
|
w.writerow(MUSIC_COLS)
|
||||||
|
for (path, aal, arts, tag_album, tag_year, disc, track, name, iid,
|
||||||
|
alb_name, alb_year, alb_id) in con.execute(
|
||||||
|
MUSIC_SQL, (JF_ALBUM, JF_AUDIO, prefix + "%")):
|
||||||
|
if not path or not path.startswith(prefix):
|
||||||
|
skipped += 1 # a row for a file outside this library
|
||||||
|
continue
|
||||||
|
# The album artist owns the folder; the track artist is the fallback
|
||||||
|
# for the compilations whose album-artist tag was left empty.
|
||||||
|
w.writerow([root + "/" + path[len(prefix):],
|
||||||
|
((aal or "").strip() or (arts or "").strip()),
|
||||||
|
(alb_name or tag_album or "").replace("\t", " "),
|
||||||
|
alb_year or tag_year or "",
|
||||||
|
"" if disc is None else disc,
|
||||||
|
"" if track is None else track,
|
||||||
|
(name or "").replace("\t", " "),
|
||||||
|
mb.get(alb_id or "") or mb.get(iid or "") or "",
|
||||||
|
(arts or "").replace("\t", " ")])
|
||||||
|
written += 1
|
||||||
|
print(f"tracks={written} outside-root-or-blank={skipped} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def audiobooks(root, db, out):
|
||||||
|
"""The chapters Plex matched inside the audiobook library, read-only.
|
||||||
|
|
||||||
|
Plex stores HOST paths here (unlike Jellyfin, which stores the container's
|
||||||
|
view), so no prefix mapping is needed: the part's `file` is the path the
|
||||||
|
operator has today. Rows for files outside `root` are skipped, because one
|
||||||
|
Plex library DB carries every library on the box.
|
||||||
|
|
||||||
|
The chapter's `index` is exported as an empty field when Plex has none. That
|
||||||
|
is not a gap to fill in here: three quarters of the books on this box contain
|
||||||
|
at least one un-indexed chapter, so the relayout that reads this file must
|
||||||
|
not renumber anything from it.
|
||||||
|
"""
|
||||||
|
uri = f"file:{db}?mode=ro&immutable=1"
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(uri, uri=True)
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
|
||||||
|
"Plex runs rootless here: run this as the container's owner, never "
|
||||||
|
"with sudo — rootless storage is per-user, so root sees nothing.")
|
||||||
|
tables = {r[0] for r in con.execute(
|
||||||
|
"select name from sqlite_master where type='table'")}
|
||||||
|
missing = {"media_parts", "media_items", "metadata_items"} - tables
|
||||||
|
if missing:
|
||||||
|
sys.exit(f"{db} is not a Plex library database (no "
|
||||||
|
f"{', '.join(sorted(missing))})")
|
||||||
|
written = skipped = 0
|
||||||
|
with open(out, "w", newline="") as fh:
|
||||||
|
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
|
||||||
|
w.writerow(AB_COLS)
|
||||||
|
for file, narrator, book, year, track, chapter in con.execute(
|
||||||
|
AB_SQL, (root.rstrip("/") + "/%",)):
|
||||||
|
if not file or not file.startswith(root.rstrip("/") + "/"):
|
||||||
|
skipped += 1 # a row for a file outside this library
|
||||||
|
continue
|
||||||
|
w.writerow([file.replace("\t", " "), narrator.replace("\t", " "),
|
||||||
|
book.replace("\t", " "), "" if not year else year,
|
||||||
|
"" if track is None or int(track) < 0 else track,
|
||||||
|
chapter.replace("\t", " ")])
|
||||||
|
written += 1
|
||||||
|
print(f"chapters={written} outside-root={skipped} -> {out}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||||
|
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||||
|
"help": "library root"}
|
||||||
|
e = sub.add_parser("enumerate", help="stat every file under the root")
|
||||||
|
e.add_argument("--root", **root)
|
||||||
|
e.add_argument("--out", default="/tmp/media-enum.tsv")
|
||||||
|
p = sub.add_parser("probe", help="ffprobe every video container")
|
||||||
|
p.add_argument("--enum", default="/tmp/media-enum.tsv")
|
||||||
|
p.add_argument("--out", default="/tmp/media-probed.tsv")
|
||||||
|
p.add_argument("--jobs", type=int, default=6)
|
||||||
|
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
|
||||||
|
x.add_argument("--root", **root)
|
||||||
|
x.add_argument("--db", default=PLEX_DB)
|
||||||
|
x.add_argument("--out", default="/tmp/plex-movies.tsv")
|
||||||
|
j = sub.add_parser("jellyfin",
|
||||||
|
help="export the episodes Jellyfin has numbered for the root")
|
||||||
|
j.add_argument("--root", default=os.environ.get("MEDIA_ROOT",
|
||||||
|
"/mnt/entropy/TVShows"),
|
||||||
|
help="TV library root on the host")
|
||||||
|
j.add_argument("--db", default=JF_DB)
|
||||||
|
j.add_argument("--container", default=None,
|
||||||
|
help="the in-container prefix that stands for --root"
|
||||||
|
" (default /data/<root's folder name>)")
|
||||||
|
j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
|
||||||
|
m = sub.add_parser("music",
|
||||||
|
help="export the tracks Jellyfin has tagged for the root")
|
||||||
|
m.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
|
||||||
|
"/mnt/entropy/Music"),
|
||||||
|
help="music library root on the host")
|
||||||
|
m.add_argument("--db", default=JF_DB)
|
||||||
|
m.add_argument("--container", default=None,
|
||||||
|
help="the in-container prefix that stands for --root"
|
||||||
|
" (default /data/<root's folder name>)")
|
||||||
|
m.add_argument("--out", default="/tmp/jellyfin-music.tsv")
|
||||||
|
b = sub.add_parser("audiobooks",
|
||||||
|
help="export the chapters Plex matched for the audiobook root")
|
||||||
|
b.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
|
||||||
|
"/mnt/entropy/Audiobooks"),
|
||||||
|
help="audiobook library root on the host")
|
||||||
|
b.add_argument("--db", default=PLEX_DB)
|
||||||
|
b.add_argument("--out", default="/tmp/plex-audiobooks.tsv")
|
||||||
|
a = ap.parse_args()
|
||||||
|
if a.cmd == "enumerate":
|
||||||
|
enumerate_lib(a.root, a.out)
|
||||||
|
elif a.cmd == "probe":
|
||||||
|
probe(a.enum, a.out, a.jobs)
|
||||||
|
elif a.cmd == "plex":
|
||||||
|
plex(a.root, a.db, a.out)
|
||||||
|
elif a.cmd == "music":
|
||||||
|
music(a.root, a.db, a.container, a.out)
|
||||||
|
elif a.cmd == "audiobooks":
|
||||||
|
audiobooks(a.root, a.db, a.out)
|
||||||
|
else:
|
||||||
|
jellyfin(a.root, a.db, a.container, a.out)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+445
@@ -0,0 +1,445 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
r"""media-music.py — a plan that re-files the music library into Jellyfin's own shape.
|
||||||
|
|
||||||
|
media-music.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
|
||||||
|
|
||||||
|
Jellyfin's music contract (https://jellyfin.org/docs/general/server/media/music/)
|
||||||
|
is not the movie one. Jellyfin reads the metadata out of the file, so a folder is
|
||||||
|
an ALBUM and nothing else, every disc of an album belongs in that one folder, the
|
||||||
|
tags decide the track order, and a file name only matters when it carries a
|
||||||
|
character known to break playback (`< > : " / \ | ? *`) or when a lyric file has
|
||||||
|
to match the basename of the track it belongs to. The plan is therefore:
|
||||||
|
|
||||||
|
Music/<album artist>/<Album> (<year>)/NN - <Title>.<ext>
|
||||||
|
... the same folder, D.NN - <Title> when the album has discs
|
||||||
|
Music/Various Artists/... when its tracks name
|
||||||
|
more than one artist
|
||||||
|
|
||||||
|
The authority is the tag row Jellyfin already read — `scripts/media-census.py
|
||||||
|
music` — never the release-pack folder a ripper made. A file with no row is not
|
||||||
|
guessed at: it parks under `<root>/_unsorted` with the relative path it has today.
|
||||||
|
|
||||||
|
The rules below exist because of what a downloaded discography pack actually
|
||||||
|
contains:
|
||||||
|
|
||||||
|
* A cuesheet beside exactly one audio file means the album is one image with a
|
||||||
|
track list inside it. That file keeps its own name: numbering it `01` would
|
||||||
|
claim it is one track, and Jellyfin shows one track either way.
|
||||||
|
* Sidecars — images, cuesheets, EAC logs, checksums, booklets, playlists —
|
||||||
|
follow the album the audio in their own folder belongs to. A folder holding
|
||||||
|
two albums gives a sidecar no single owner, so it parks instead of being
|
||||||
|
guessed into one.
|
||||||
|
* A video container (.mkv, .vob, .ifo, …) inside a music library is not music.
|
||||||
|
Renaming it to .mka/.m4a would tell Jellyfin to index video as audio, so it
|
||||||
|
parks for a human decision.
|
||||||
|
* Playlists move with their album and their CONTENTS are never rewritten: the
|
||||||
|
paths inside an .m3u go stale. Editing a file the operator owns to keep a
|
||||||
|
convenience file tidy is not this tool's job.
|
||||||
|
|
||||||
|
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
|
||||||
|
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
|
||||||
|
already claimed and whose journal is the rollback.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
|
||||||
|
# What Jellyfin will index as music. .mp4/.mkv/.webm are deliberately NOT here:
|
||||||
|
# the vendor doc says an audio-only stream in those containers is never read as
|
||||||
|
# music, and this tool renames for a reason it can prove, not to guess at one.
|
||||||
|
AUDIO_EXT = {".flac", ".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav",
|
||||||
|
".wma", ".aiff", ".aif", ".ape", ".dsf", ".dff", ".wv", ".mka",
|
||||||
|
".mpc", ".aac"}
|
||||||
|
VIDEO_EXT = {".mkv", ".vob", ".ifo", ".bup", ".mpg", ".mpeg", ".avi", ".mov",
|
||||||
|
".ts", ".m2ts", ".wmv", ".webm", ".mp4", ".m4v", ".iso", ".img",
|
||||||
|
".bdmv", ".divx", ".ogm"}
|
||||||
|
# A lyric has to carry the basename of its track, so it is named after the audio
|
||||||
|
# file and not after itself. .txt counts only when its stem matches an audio stem:
|
||||||
|
# a loose .txt in an album folder is a note.
|
||||||
|
LYRIC_EXT = {".lrc", ".elrc", ".txt"}
|
||||||
|
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tiff"}
|
||||||
|
# The album-artist strings that mean "compilation", so such an album is not filed
|
||||||
|
# under one artist who does not own it.
|
||||||
|
VARIOUS = {"various", "various artists", "unknown", "unknown artist", "unknow"}
|
||||||
|
# The characters Jellyfin documents as breaking playback. They are dropped, not
|
||||||
|
# substituted: a replacement character is a guess at what the title meant.
|
||||||
|
BAD_CHARS = set('<>:"/\\|?*')
|
||||||
|
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
|
||||||
|
r"|lost\+found)(/|$)")
|
||||||
|
|
||||||
|
ARGS = None
|
||||||
|
HELD = {} # path -> (bytes, ext)
|
||||||
|
ROWS = {} # path -> the tag row
|
||||||
|
PLACED = {} # audio path -> the destination the plan gave it
|
||||||
|
CLAIMED = set() # every destination path the plan has handed out
|
||||||
|
MOVES, KEEPS = [], []
|
||||||
|
PARKED = defaultdict(list) # reason -> [path]
|
||||||
|
NOTES = defaultdict(list) # section -> [line]
|
||||||
|
CUE_DIRS = set() # source dirs holding a .cue
|
||||||
|
AUDIO_BY_DIR = defaultdict(set) # source dir -> audio files a row names
|
||||||
|
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding audio
|
||||||
|
FOLDERS_SEEN = {} # (dir, depth) -> album folders placed below it
|
||||||
|
SUBTREE_DEPTH = 2 # how far below a folder its audio may sit
|
||||||
|
ALBUMS = {} # (artist, album, year) -> the album group
|
||||||
|
|
||||||
|
AMBIGUOUS = object()
|
||||||
|
|
||||||
|
|
||||||
|
def dominant(values):
|
||||||
|
"""The spelling the tags mostly agree on: the most frequent value, ties broken
|
||||||
|
by sort so the answer never depends on the order rows arrived in."""
|
||||||
|
counts = Counter(values)
|
||||||
|
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
|
||||||
|
|
||||||
|
|
||||||
|
def rel(path):
|
||||||
|
"""The path as seen from the library root."""
|
||||||
|
return os.path.relpath(path, ARGS.root)
|
||||||
|
|
||||||
|
|
||||||
|
def safe(name):
|
||||||
|
"""A name every client can carry: no reserved character, nothing
|
||||||
|
unprintable, no trailing dot or space, and short enough to rename."""
|
||||||
|
out = "".join(c for c in (name or "")
|
||||||
|
if c not in BAD_CHARS and (c.isprintable() or c == " "))
|
||||||
|
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
|
||||||
|
while out and len(out.encode("utf-8", "ignore")) > 200:
|
||||||
|
out = out[:-4].rstrip()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def assign(src, folder, name, ext, kind, note):
|
||||||
|
"""Book one file into `folder` under `name`, renamed if that exact destination
|
||||||
|
is already taken — by another file in the plan, or by a file standing there
|
||||||
|
today. Two files claiming one path is the fault that costs a re-run of a 90k
|
||||||
|
plan, and so is a plan that only works if the renames happen in the right
|
||||||
|
order: the executor refuses any target that is occupied, so an occupied path
|
||||||
|
is never a destination, even when its occupant is scheduled to leave.
|
||||||
|
A file already standing where the contract puts it is a keep, not a move."""
|
||||||
|
stem = safe(name) or "file"
|
||||||
|
dst = os.path.join(folder, stem + ext)
|
||||||
|
if dst == src:
|
||||||
|
CLAIMED.add(dst.lower()) # it is standing on its own target
|
||||||
|
KEEPS.append((src, dst, kind))
|
||||||
|
return dst
|
||||||
|
# A file already in this folder under this name — or under the numbered
|
||||||
|
# variant the collision rule gives it — keeps the name it has. Renumbering it
|
||||||
|
# every run would turn the plan into a shuffle instead of a fixed point.
|
||||||
|
base = os.path.splitext(os.path.basename(src))[0]
|
||||||
|
if os.path.dirname(src) == folder and (base == stem
|
||||||
|
or base.startswith(stem + " (")):
|
||||||
|
CLAIMED.add(src.lower())
|
||||||
|
KEEPS.append((src, src, kind))
|
||||||
|
return src
|
||||||
|
n = 1
|
||||||
|
while dst.lower() in CLAIMED:
|
||||||
|
n += 1
|
||||||
|
dst = os.path.join(folder, f"{stem} ({n}){ext}")
|
||||||
|
if n > 1:
|
||||||
|
NOTES[note].append(rel(dst))
|
||||||
|
CLAIMED.add(dst.lower())
|
||||||
|
MOVES.append((src, dst, kind))
|
||||||
|
return dst
|
||||||
|
|
||||||
|
|
||||||
|
def park(path, reason):
|
||||||
|
"""Hold a file where it is, under the park folder, keeping its relative path."""
|
||||||
|
PARKED[reason].append(path)
|
||||||
|
dst = os.path.join(ARGS.park_dir, rel(path))
|
||||||
|
if dst != path:
|
||||||
|
CLAIMED.add(dst.lower())
|
||||||
|
MOVES.append((path, dst, "park"))
|
||||||
|
|
||||||
|
|
||||||
|
def under_park(path):
|
||||||
|
"""True for a file already parked: that pile is for a human to review, and
|
||||||
|
re-parking it every run would only push it deeper."""
|
||||||
|
return path.startswith(ARGS.park_dir + os.sep)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- albums
|
||||||
|
def group_albums():
|
||||||
|
"""Every tagged track into one group per (album-artist tag, album, year).
|
||||||
|
|
||||||
|
The key is the identity the tags assert, not the folder the file sits in: one
|
||||||
|
album arrives as three discs in three folders, and two copies of an album in
|
||||||
|
one pack are one album to Jellyfin, whose album item is that name, artist and
|
||||||
|
year. The artist in the key is the ALBUM-artist tag and never the performer,
|
||||||
|
because a compilation is one album whose tracks name many performers — keying
|
||||||
|
on the performer would file one album once per artist.
|
||||||
|
"""
|
||||||
|
for path, r in sorted(ROWS.items()):
|
||||||
|
if path not in HELD or under_park(path):
|
||||||
|
continue
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
AUDIO_BY_DIR[d].add(path)
|
||||||
|
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above CD1/CD2 sees both
|
||||||
|
tag = (r["artist"] or "").strip()
|
||||||
|
album = (r["album"] or "").strip()
|
||||||
|
performer = (r["performer"] or "").strip()
|
||||||
|
if not album:
|
||||||
|
park(path, "a track no album names")
|
||||||
|
continue
|
||||||
|
if not tag and not performer:
|
||||||
|
park(path, "a track no artist names")
|
||||||
|
continue
|
||||||
|
key = (tag.casefold(), album.casefold(), (r["year"] or "").strip())
|
||||||
|
g = ALBUMS.setdefault(key, {"tags": [], "albums": [], "year": key[2],
|
||||||
|
"members": [], "performers": set(),
|
||||||
|
"mb": set()})
|
||||||
|
g["members"].append(path)
|
||||||
|
g["tags"].append(tag)
|
||||||
|
g["albums"].append(album)
|
||||||
|
g["performers"].add(performer or tag)
|
||||||
|
if r.get("mb"):
|
||||||
|
g["mb"].add(r["mb"])
|
||||||
|
|
||||||
|
|
||||||
|
def folder_artist(g):
|
||||||
|
"""Whose shelf the album sits on. The album-artist tag owns it when it names
|
||||||
|
one; when the tags name only performers, a single performer names the album
|
||||||
|
and several make it a compilation, which belongs to no one of them. The key is
|
||||||
|
case-folded, so the spelling that reaches the folder is the one the tracks
|
||||||
|
mostly carry — see dominant()."""
|
||||||
|
tags = [t for t in g["tags"] if t]
|
||||||
|
if tags:
|
||||||
|
name = dominant(tags)
|
||||||
|
return "Various Artists" if name.casefold() in VARIOUS else name
|
||||||
|
if len({p.casefold() for p in g["performers"]}) == 1:
|
||||||
|
return dominant(g["performers"])
|
||||||
|
return "Various Artists"
|
||||||
|
|
||||||
|
|
||||||
|
def album_folder(g):
|
||||||
|
name = safe(dominant(g["albums"])) or "Untitled"
|
||||||
|
return os.path.join(ARGS.root, safe(folder_artist(g)) or "Unknown Artist",
|
||||||
|
f"{name} ({g['year']})" if g["year"] else name)
|
||||||
|
|
||||||
|
|
||||||
|
def is_cue_image(path):
|
||||||
|
"""A folder holding this one audio file and a cuesheet is an album image."""
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
return len(AUDIO_BY_DIR.get(d, ())) == 1 and d in CUE_DIRS
|
||||||
|
|
||||||
|
|
||||||
|
def number_of(value, width):
|
||||||
|
return f"{int(value):0{width}d}" if str(value or "").isdigit() else ""
|
||||||
|
|
||||||
|
|
||||||
|
def place_albums():
|
||||||
|
for key in sorted(ALBUMS):
|
||||||
|
g = ALBUMS[key]
|
||||||
|
folder = album_folder(g)
|
||||||
|
discs = {g0["disc"] for g0 in (ROWS[p] for p in g["members"]) if g0["disc"]}
|
||||||
|
tracks = [int(ROWS[p]["track"]) for p in g["members"]
|
||||||
|
if str(ROWS[p]["track"] or "").isdigit()]
|
||||||
|
# More than one disc, or a disc numbered past the first: the track number
|
||||||
|
# alone cannot name a file uniquely, so the disc goes in front of it.
|
||||||
|
multi = len(discs) > 1 or any(str(d).isdigit() and int(d) > 1 for d in discs)
|
||||||
|
width = 3 if multi or (max(tracks, default=0) > 99) else 2
|
||||||
|
for path in sorted(g["members"]):
|
||||||
|
r = ROWS[path]
|
||||||
|
ext = HELD[path][1]
|
||||||
|
base = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
if is_cue_image(path):
|
||||||
|
name = safe(base) or "image"
|
||||||
|
else:
|
||||||
|
title = safe(r["title"] or base) or "untitled"
|
||||||
|
num = number_of(r["track"], 2 if multi else width)
|
||||||
|
if multi and num:
|
||||||
|
num = f"{r['disc'] or '1'}.{num}"
|
||||||
|
name = f"{num} - {title}" if num else title
|
||||||
|
PLACED[path] = assign(path, folder, name, ext, "track",
|
||||||
|
"two tracks claimed one name inside one album")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------- sidecars
|
||||||
|
def folders_below(d, depth):
|
||||||
|
"""The album folders the audio in `d`, or within `depth` directories below it,
|
||||||
|
was placed in. Memoised, because every loose file in a big discography folder
|
||||||
|
asks the same question about it."""
|
||||||
|
if (d, depth) not in FOLDERS_SEEN:
|
||||||
|
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
|
||||||
|
if a in PLACED}
|
||||||
|
if depth:
|
||||||
|
for child in CHILD_OF.get(d, ()):
|
||||||
|
seen |= folders_below(child, depth - 1)
|
||||||
|
FOLDERS_SEEN[(d, depth)] = seen
|
||||||
|
return FOLDERS_SEEN[(d, depth)]
|
||||||
|
|
||||||
|
|
||||||
|
def owner_folder(d):
|
||||||
|
"""The album folder that owns this directory — the first place up the tree
|
||||||
|
where audio sits, in the directory itself or within SUBTREE_DEPTH levels below
|
||||||
|
it, which is how a box with its discs in CD1/CD2 owns the cover lying above
|
||||||
|
them. None when no audio is near enough to say, AMBIGUOUS when the audio there
|
||||||
|
belongs to more than one album."""
|
||||||
|
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
|
||||||
|
folders = folders_below(d, SUBTREE_DEPTH)
|
||||||
|
if folders:
|
||||||
|
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def place_sidecars():
|
||||||
|
"""Everything that is not a tagged track: lyrics, images, cuesheets, logs,
|
||||||
|
booklets, playlists — and the clutter that is none of those."""
|
||||||
|
# The lyric rule needs the audio's stem as it is named TODAY, before the plan
|
||||||
|
# renames it, because that is what the sidecar's own name was copied from.
|
||||||
|
stem = {(d, os.path.splitext(os.path.basename(a))[0]): a
|
||||||
|
for d, audios in AUDIO_BY_DIR.items() for a in audios}
|
||||||
|
for path, (_size, ext) in sorted(HELD.items()):
|
||||||
|
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
|
||||||
|
continue
|
||||||
|
d, base = os.path.dirname(path), os.path.splitext(os.path.basename(path))[0]
|
||||||
|
if ext in VIDEO_EXT:
|
||||||
|
park(path, "a video container inside the music library: Jellyfin will "
|
||||||
|
"not read it as music, and renaming it would claim it is")
|
||||||
|
elif ext in AUDIO_EXT:
|
||||||
|
park(path, "an audio file no provider row names — its tags are not in "
|
||||||
|
"the library Jellyfin has matched")
|
||||||
|
elif ext in (LYRIC_EXT | IMAGE_EXT) and (d, base) in stem:
|
||||||
|
# A lyric, or a picture named after its track: it follows that track
|
||||||
|
# and takes the name the track now has, or Jellyfin cannot find it.
|
||||||
|
audio = stem[(d, base)]
|
||||||
|
dst = PLACED.get(audio)
|
||||||
|
if dst:
|
||||||
|
folder, name = os.path.dirname(dst), os.path.splitext(
|
||||||
|
os.path.basename(dst))[0]
|
||||||
|
assign(path, folder, name, ext, "lyric",
|
||||||
|
"a sidecar whose name collided with another in its album")
|
||||||
|
else:
|
||||||
|
park(path, "a sidecar of a track the plan could not place")
|
||||||
|
else:
|
||||||
|
owner = owner_folder(d)
|
||||||
|
if owner is AMBIGUOUS:
|
||||||
|
park(path, "a sidecar whose folder holds more than one album, so "
|
||||||
|
"nothing says which one it belongs to")
|
||||||
|
elif owner:
|
||||||
|
assign(path, owner, base, ext,
|
||||||
|
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
|
||||||
|
"one album folder would have carried two files of the same "
|
||||||
|
"sidecar name")
|
||||||
|
else:
|
||||||
|
park(path, "not music, or music with nothing beside it to belong to")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- outputs
|
||||||
|
def write_plan():
|
||||||
|
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
||||||
|
Parents first: the executor creates directories in the order they are listed."""
|
||||||
|
chain = set()
|
||||||
|
for _s, dst, _k in MOVES:
|
||||||
|
d = os.path.dirname(dst)
|
||||||
|
while d and d != ARGS.root and d != os.path.dirname(d):
|
||||||
|
chain.add(d)
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
||||||
|
with open(ARGS.out, "w") as fh:
|
||||||
|
fh.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in dirs:
|
||||||
|
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
|
||||||
|
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||||
|
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
|
||||||
|
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
||||||
|
parked = [m for m in MOVES if m[2] == "park"]
|
||||||
|
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
|
||||||
|
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
|
||||||
|
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
|
||||||
|
f" {len(dirs)} folders -> {ARGS.out}")
|
||||||
|
|
||||||
|
|
||||||
|
def write_exceptions():
|
||||||
|
with open(ARGS.exceptions, "w") as fh:
|
||||||
|
fh.write(f"Music relayout exceptions for {ARGS.root}\n\n")
|
||||||
|
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
|
||||||
|
"relative path it has today. Nothing is deleted and nothing is\n"
|
||||||
|
"overwritten: a parked file is one rename from wherever the plan is\n"
|
||||||
|
"pointed next.\n\n")
|
||||||
|
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
|
||||||
|
rs = PARKED[reason]
|
||||||
|
fh.write(f"{len(rs):>6} files {reason}\n")
|
||||||
|
for p in sorted(rs, key=rel)[:400]:
|
||||||
|
fh.write(f" {rel(p)}\n")
|
||||||
|
if len(rs) > 400:
|
||||||
|
fh.write(f" … and {len(rs) - 400} more\n")
|
||||||
|
fh.write("\n")
|
||||||
|
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
|
||||||
|
lines = sorted(set(NOTES[section]))
|
||||||
|
fh.write(f"{len(lines):>6} {section}\n")
|
||||||
|
for line in lines[:200]:
|
||||||
|
fh.write(f" {line}\n")
|
||||||
|
if len(lines) > 200:
|
||||||
|
fh.write(f" … and {len(lines) - 200} more\n")
|
||||||
|
fh.write("\n")
|
||||||
|
no_year = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
|
||||||
|
for g in ALBUMS.values() if not g["year"])
|
||||||
|
if no_year:
|
||||||
|
fh.write(f"{len(no_year):>6} albums carry no year in any tag, so their folders\n"
|
||||||
|
" are named without one; Jellyfin still matches them.\n")
|
||||||
|
fh.write(" " + ", ".join(no_year[:300]) + "\n\n")
|
||||||
|
clash = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
|
||||||
|
for g in ALBUMS.values() if len(g["mb"]) > 1)
|
||||||
|
if clash:
|
||||||
|
fh.write(f"{len(clash):>6} albums put one folder name on more than one\n"
|
||||||
|
" MusicBrainz release group — different releases, one name.\n"
|
||||||
|
" They share the folder Jellyfin's own album item names.\n")
|
||||||
|
fh.write(" " + ", ".join(clash[:200]) + "\n\n")
|
||||||
|
fh.write(" Playlists move with their album and their contents are never\n"
|
||||||
|
" rewritten: the paths inside an .m3u go stale.\n")
|
||||||
|
print(f"exceptions -> {ARGS.exceptions}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
global ARGS
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
|
||||||
|
"/mnt/entropy/Music"))
|
||||||
|
ap.add_argument("--enum", default="/tmp/music-enum.tsv",
|
||||||
|
help="the enumerate TSV (bytes, ext, path)")
|
||||||
|
ap.add_argument("--rows", default="/tmp/jellyfin-music.tsv",
|
||||||
|
help="the tag rows scripts/media-census.py music wrote")
|
||||||
|
ap.add_argument("--out", default="/tmp/music-plan.tsv")
|
||||||
|
ap.add_argument("--exceptions", default="/tmp/music-exceptions.txt")
|
||||||
|
ap.add_argument("--park", default="_unsorted",
|
||||||
|
help="the folder inside the root that holds what cannot be filed")
|
||||||
|
ARGS = ap.parse_args()
|
||||||
|
ARGS.root = os.path.normpath(ARGS.root)
|
||||||
|
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
|
||||||
|
read_inputs()
|
||||||
|
# Every path a file occupies today is claimed before a single destination is
|
||||||
|
# handed out: the executor refuses an occupied target, so a plan that aimed at
|
||||||
|
# one would only fail after the renames that were supposed to clear it.
|
||||||
|
CLAIMED.update(p.lower() for p in HELD)
|
||||||
|
for path, (_size, ext) in HELD.items():
|
||||||
|
if ext == ".cue":
|
||||||
|
CUE_DIRS.add(os.path.dirname(path))
|
||||||
|
group_albums()
|
||||||
|
place_albums()
|
||||||
|
place_sidecars()
|
||||||
|
write_plan()
|
||||||
|
write_exceptions()
|
||||||
|
|
||||||
|
|
||||||
|
def read_inputs():
|
||||||
|
with open(ARGS.enum) as fh:
|
||||||
|
for line in fh:
|
||||||
|
parts = line.rstrip("\n").split("\t")
|
||||||
|
if len(parts) >= 3:
|
||||||
|
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
|
||||||
|
with open(ARGS.rows, newline="") as fh:
|
||||||
|
for row in csv.DictReader(fh, delimiter="\t"):
|
||||||
|
if row.get("file"):
|
||||||
|
ROWS[row["file"]] = row
|
||||||
|
gone = [p for p in ROWS if p not in HELD]
|
||||||
|
if gone:
|
||||||
|
NOTES["a tag row whose file the census no longer sees"].extend(gone)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+881
@@ -0,0 +1,881 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
|
||||||
|
|
||||||
|
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
|
||||||
|
[--out TSV] [--exceptions TXT]
|
||||||
|
|
||||||
|
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
|
||||||
|
metadata-cache export whose clean titles outrank the filename parser; the title
|
||||||
|
authority for everything Plex never matched is scripts/media-titles.py beside this
|
||||||
|
file. No media file is opened, so planning the whole library costs seconds.
|
||||||
|
|
||||||
|
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
|
||||||
|
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
|
||||||
|
parents exist before children. EXCEPTIONS is the human review list: every file the
|
||||||
|
plan could not name a film for, every collision it refused, the READMEs the
|
||||||
|
executor must write into the piles it created.
|
||||||
|
|
||||||
|
Target contract (jellyfin.org/docs/general/server/media/movies):
|
||||||
|
* one folder per movie, named Title (year)
|
||||||
|
* the video inside is named exactly like the folder
|
||||||
|
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
|
||||||
|
match the folder name character-for-character or Jellyfin splits them apart
|
||||||
|
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
|
||||||
|
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
|
||||||
|
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
|
||||||
|
* <>:"/\|?* break the matcher, so they never survive here
|
||||||
|
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import collections
|
||||||
|
import csv
|
||||||
|
import importlib.util
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
|
||||||
|
# this file is the title authority. Nothing here opens a media file.
|
||||||
|
def parse_args():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||||
|
help="library root the plan moves files inside")
|
||||||
|
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
|
||||||
|
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
|
||||||
|
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
|
||||||
|
help="Plex metadata export (clean titles); absent = parser only")
|
||||||
|
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
|
||||||
|
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
|
||||||
|
return ap.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
ARGS = parse_args()
|
||||||
|
ROOT = ARGS.root
|
||||||
|
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
|
||||||
|
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
|
||||||
|
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||||
|
eng = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(eng)
|
||||||
|
|
||||||
|
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
||||||
|
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
|
||||||
|
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
|
||||||
|
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
|
||||||
|
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
|
||||||
|
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
|
||||||
|
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
|
||||||
|
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
|
||||||
|
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
|
||||||
|
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
|
||||||
|
r"|(additional|extra) languages|languages|resources")
|
||||||
|
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
|
||||||
|
r"|(additional|extra) languages|commentar(y|ies)"
|
||||||
|
r"|audio commentary|audio tracks?")
|
||||||
|
|
||||||
|
|
||||||
|
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
|
||||||
|
".opus", ".ogg", ".wma"}
|
||||||
|
NFO_EXT = {".nfo"}
|
||||||
|
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
|
||||||
|
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
|
||||||
|
"banner": "banner", "logo": "logo", "clearlogo": "logo",
|
||||||
|
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
|
||||||
|
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
|
||||||
|
"discart": "poster", "disc": "poster"}
|
||||||
|
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
|
||||||
|
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
|
||||||
|
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
|
||||||
|
# library holds six of those films), so leaving it in a name turns ONE movie folder
|
||||||
|
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
|
||||||
|
# folders, none of them a movie.
|
||||||
|
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
||||||
|
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
|
||||||
|
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
|
||||||
|
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
|
||||||
|
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
|
||||||
|
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
|
||||||
|
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
|
||||||
|
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
|
||||||
|
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
|
||||||
|
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
|
||||||
|
"eng", "english", "italian", "dual", "audio", "forced", "default"}
|
||||||
|
# Canonical title where the library only ever carries a localised name.
|
||||||
|
OVERRIDES = {"28 giorni dopo": "28 Days Later",
|
||||||
|
"28 settimane dopo": "28 Weeks Later"}
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for line in open(ENUM):
|
||||||
|
size, ext, path = line.rstrip("\n").split("\t")
|
||||||
|
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
|
||||||
|
ext_by = {r["path"]: r["ext"] for r in rows}
|
||||||
|
size_by = {r["path"]: r["bytes"] for r in rows}
|
||||||
|
|
||||||
|
|
||||||
|
def rel(p):
|
||||||
|
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
|
||||||
|
|
||||||
|
|
||||||
|
def parts_of(p):
|
||||||
|
return [q for q in rel(p).split("/") if q]
|
||||||
|
|
||||||
|
|
||||||
|
def is_qb(p):
|
||||||
|
return ".!q" in p.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def disc_root(p, fold=True):
|
||||||
|
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
|
||||||
|
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
|
||||||
|
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
|
||||||
|
ps = parts_of(p)
|
||||||
|
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
|
||||||
|
for i, q in enumerate(ps):
|
||||||
|
if q.upper() in kinds:
|
||||||
|
return ROOT + "/" + "/".join(ps[:i])
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def row_name(r):
|
||||||
|
"""The name a disc-tree entry should be identified by: the engine stores the
|
||||||
|
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
|
||||||
|
return r.get("name_src") or r["path"]
|
||||||
|
|
||||||
|
|
||||||
|
def sanitize(name):
|
||||||
|
name = BAD_CHARS.sub("-", name).replace("\\", "-")
|
||||||
|
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
|
||||||
|
return (name[:180] or "untitled")
|
||||||
|
|
||||||
|
|
||||||
|
def ok_year(y):
|
||||||
|
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
|
||||||
|
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
|
||||||
|
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
|
||||||
|
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
|
||||||
|
|
||||||
|
|
||||||
|
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
|
||||||
|
|
||||||
|
|
||||||
|
def words_of(text):
|
||||||
|
"""The matching words of a name — or of a whole path, which is how callers
|
||||||
|
pass a file together with the folders it sits in.
|
||||||
|
|
||||||
|
Split on everything that is not a letter or a digit: eng.norm does not touch
|
||||||
|
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
|
||||||
|
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
|
||||||
|
film could not be found by the folder that holds it — which is how one box
|
||||||
|
set's featurettes ended up in another film's folder. Both sides of every
|
||||||
|
comparison below go through here. A bare number counts only when it could be
|
||||||
|
a release year, so release junk ("1080", "2160", "2", "7") never poses as
|
||||||
|
agreement between two names.
|
||||||
|
"""
|
||||||
|
out = set()
|
||||||
|
for w in WORD_SPLIT.split(eng.norm(text)):
|
||||||
|
if not w or w in eng.STOP:
|
||||||
|
continue
|
||||||
|
if w.isdigit() and not 1870 <= int(w) <= 2100:
|
||||||
|
continue
|
||||||
|
out.add(w)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ Plex titles
|
||||||
|
# Plex has matched a large part of the library's media files to real titles
|
||||||
|
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
|
||||||
|
# holds a MATCH, its title wins; the filename parser carries everything else
|
||||||
|
# (extras, disc trees, TV, and whatever Plex never matched).
|
||||||
|
def plex_titles():
|
||||||
|
if not os.path.exists(PLEX):
|
||||||
|
return {}
|
||||||
|
out = {}
|
||||||
|
with open(PLEX, newline="") as fh:
|
||||||
|
for r in csv.DictReader(fh, delimiter="\t"):
|
||||||
|
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
|
||||||
|
# Only a match is an authority, and a match carries a year: an
|
||||||
|
# unmatched row echoes the filename, which the parser reads better
|
||||||
|
# than a copied release name. The junk filter stays for the matches
|
||||||
|
# whose title still carries the release.
|
||||||
|
if t and ok_year(y) and not re.search(
|
||||||
|
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
|
||||||
|
out[r["file"]] = (t, y)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
PLEX_T = plex_titles()
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- classification
|
||||||
|
erows = eng.load(PROBE)
|
||||||
|
for r in erows:
|
||||||
|
for k, v in OVERRIDES.items():
|
||||||
|
if k in r["title"]:
|
||||||
|
r["title"] = v
|
||||||
|
r["key"] = eng.key_of(r["title"])
|
||||||
|
r["base"] = eng.base_words(r["title"], r["key"])
|
||||||
|
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
|
||||||
|
extras = {r["path"] for r in erows
|
||||||
|
if r["path"] not in marks
|
||||||
|
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
|
||||||
|
matched=r["path"] in PLEX_T)}
|
||||||
|
eps = collections.defaultdict(list)
|
||||||
|
for r in erows:
|
||||||
|
if r["path"] in marks | extras:
|
||||||
|
continue
|
||||||
|
ep = eng.ep_of(r["path"])
|
||||||
|
if ep:
|
||||||
|
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
|
||||||
|
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
|
||||||
|
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
|
||||||
|
# A folder INSIDE a movie's folder is bonus territory even when its own name is
|
||||||
|
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
|
||||||
|
# misspelled Extras folder, so the engine's folder rule never fired and the
|
||||||
|
# featurette became a movie folder of its own — whose words then out-scored the
|
||||||
|
# real Saboteur and drew that box set's other extras into it. The rule needs no
|
||||||
|
# metadata: the folder holding the longest film owns the shorter video filed
|
||||||
|
# under it, provided that shorter file's own name speaks the film's name — a
|
||||||
|
# whole other film filed there stays a film, exactly as the engine's folder rule
|
||||||
|
# exempts it. A different year in the file's own name IS that other film
|
||||||
|
# speaking, so it vetoes the rule.
|
||||||
|
def longest_of(rows):
|
||||||
|
"""The longest video in each folder: that folder's feature."""
|
||||||
|
out = {}
|
||||||
|
for r in rows:
|
||||||
|
if r["dur"] > 0:
|
||||||
|
d = os.path.dirname(r["path"])
|
||||||
|
if d not in out or r["dur"] > out[d]["dur"]:
|
||||||
|
out[d] = r
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def not_disc(r):
|
||||||
|
"""A disc tree's innards belong to the tree, never to the folder's film."""
|
||||||
|
return not any(c in DISC_DIRS for c in r["path"].split("/"))
|
||||||
|
|
||||||
|
|
||||||
|
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
|
||||||
|
|
||||||
|
|
||||||
|
def under_a_feature(path, dur):
|
||||||
|
"""True when a folder above this file holds a longer film that this file's
|
||||||
|
own name names."""
|
||||||
|
d = os.path.dirname(os.path.dirname(path))
|
||||||
|
while len(d) > len(ROOT):
|
||||||
|
f = FEATURE_OF.get(d)
|
||||||
|
if f:
|
||||||
|
if f["dur"] < dur:
|
||||||
|
return False
|
||||||
|
theirs = (words_of(os.path.basename(f["path"]))
|
||||||
|
| words_of(os.path.basename(d)))
|
||||||
|
if not words_of(os.path.basename(path)) & theirs:
|
||||||
|
return False
|
||||||
|
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
|
||||||
|
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
|
||||||
|
return not (y1 and y2 and y1.group() != y2.group())
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
extras |= {r["path"] for r in erows
|
||||||
|
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
|
||||||
|
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
|
||||||
|
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
|
||||||
|
|
||||||
|
buckets = collections.defaultdict(list)
|
||||||
|
for r in film_rows:
|
||||||
|
buckets[eng.key_of(r["title"])].append(r)
|
||||||
|
|
||||||
|
|
||||||
|
def res_label(r):
|
||||||
|
m = RES_RE.search(r["res"] or "")
|
||||||
|
if m:
|
||||||
|
n, kind = m.group(1).lower(), m.group(2).lower()
|
||||||
|
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
|
||||||
|
try:
|
||||||
|
w = int((r["res"] or "0x0").split("x")[0])
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
w = 0
|
||||||
|
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
|
||||||
|
|
||||||
|
|
||||||
|
def version_label(r):
|
||||||
|
bits = [res_label(r)]
|
||||||
|
vc = (r["vcodec"] or "").lower()
|
||||||
|
if vc in ("av1", "libaom", "libsvtav1"):
|
||||||
|
bits.append("AV1")
|
||||||
|
elif vc in ("vc1", "wmv3"):
|
||||||
|
bits.append("VC1")
|
||||||
|
if r["bytes"] > 40 * 2**30:
|
||||||
|
bits.append("REMUX")
|
||||||
|
if "imax" in r["path"].lower():
|
||||||
|
bits.append("IMAX")
|
||||||
|
return sanitize(" ".join(bits))
|
||||||
|
|
||||||
|
|
||||||
|
def group_name(items):
|
||||||
|
titles = collections.Counter(r["title"] for r in items)
|
||||||
|
ascii_t = [t for t in titles if t.isascii()]
|
||||||
|
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
|
||||||
|
years = [r["year"] for r in items if r["year"] != "?"] or \
|
||||||
|
[r["soft"] for r in items if r["soft"]]
|
||||||
|
year = collections.Counter(years).most_common(1)[0][0] if years else None
|
||||||
|
return sanitize(f"{title} ({year})" if year else title)
|
||||||
|
|
||||||
|
|
||||||
|
groups = []
|
||||||
|
for key, items in buckets.items():
|
||||||
|
for c in eng.dur_clusters(items):
|
||||||
|
g = {"rows": c, "base": group_name(c)}
|
||||||
|
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
|
||||||
|
if known:
|
||||||
|
# one film, one name: a Plex match for any copy names every copy
|
||||||
|
t, yr = collections.Counter(known).most_common(1)[0][0]
|
||||||
|
g["plex"] = t
|
||||||
|
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
|
||||||
|
groups.append(g)
|
||||||
|
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
|
||||||
|
for g in groups:
|
||||||
|
if g["base"] in clash:
|
||||||
|
# two genuinely different films share title+year: separate folders, and
|
||||||
|
# say so in the name, because Jellyfin can only match one item per folder
|
||||||
|
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
|
||||||
|
g["flag"] = "same title and year as another film"
|
||||||
|
|
||||||
|
moves, mkdirs, problems, readmes = [], set(), [], []
|
||||||
|
# --------------------------------------------------------------- unnamed groups
|
||||||
|
# A movie folder has to READ as a title. The engine's runtime veto says any file
|
||||||
|
# over 45 minutes that is not named like an extra is a film whatever it is called
|
||||||
|
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
|
||||||
|
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
|
||||||
|
# with no identity whose numeric keys went on to capture every subtitle named
|
||||||
|
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
|
||||||
|
# real film), so the gate is: a year, or a word that is not release noise. With
|
||||||
|
# neither, the rows are bonus material until a human names them.
|
||||||
|
def named(g):
|
||||||
|
if re.search(r"\(\d{4}\)$", g["base"]):
|
||||||
|
return True
|
||||||
|
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
|
||||||
|
|
||||||
|
|
||||||
|
unnamed = [g for g in groups if not named(g)]
|
||||||
|
groups = [g for g in groups if named(g)]
|
||||||
|
for g in unnamed:
|
||||||
|
for r in g["rows"]:
|
||||||
|
extras.add(r["path"])
|
||||||
|
problems.append(
|
||||||
|
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
|
||||||
|
f"bonus pile, not a folder — give the file a real name and re-plan")
|
||||||
|
|
||||||
|
|
||||||
|
group_of = {}
|
||||||
|
for g in groups:
|
||||||
|
g["dir"] = f"{ROOT}/{g['base']}"
|
||||||
|
for r in g["rows"]:
|
||||||
|
group_of[r["path"]] = g
|
||||||
|
dr = disc_root(r["path"])
|
||||||
|
if dr:
|
||||||
|
group_of[dr] = g
|
||||||
|
by_path = {r["path"]: r for r in erows}
|
||||||
|
|
||||||
|
# -------------------------------------------------------------- bookkeeping
|
||||||
|
claimed = {}
|
||||||
|
existing = set(rows_map := {r["path"] for r in rows})
|
||||||
|
|
||||||
|
|
||||||
|
at_target, booked_src = set(), set()
|
||||||
|
|
||||||
|
|
||||||
|
def book(src, dst, kind):
|
||||||
|
if src == dst:
|
||||||
|
at_target.add(src)
|
||||||
|
booked_src.add(src)
|
||||||
|
return
|
||||||
|
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
|
||||||
|
why = "rebook" if src in booked_src else "collision"
|
||||||
|
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
|
||||||
|
if src in booked_src: # one source may only ever move once
|
||||||
|
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
|
||||||
|
return
|
||||||
|
if dst.lower() in claimed:
|
||||||
|
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
|
||||||
|
return
|
||||||
|
claimed[dst.lower()] = src
|
||||||
|
booked_src.add(src)
|
||||||
|
comp = dst.split("/")
|
||||||
|
for i in range(1, len(comp)):
|
||||||
|
mkdirs.add("/".join(comp[:i]))
|
||||||
|
moves.append((src, dst, kind))
|
||||||
|
|
||||||
|
|
||||||
|
def uniq_dir(d, stem, ext, hint=""):
|
||||||
|
"""A destination that no other source already took."""
|
||||||
|
cand = f"{d}/{stem}{ext}"
|
||||||
|
if cand.lower() not in claimed:
|
||||||
|
return cand
|
||||||
|
n = 2
|
||||||
|
while True:
|
||||||
|
cand = f"{d}/{stem} {n}{ext}"
|
||||||
|
if cand.lower() not in claimed:
|
||||||
|
return cand
|
||||||
|
n += 1
|
||||||
|
|
||||||
|
|
||||||
|
def media_dir(p):
|
||||||
|
dr = disc_root(p)
|
||||||
|
return os.path.dirname(dr) if dr else os.path.dirname(p)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ films
|
||||||
|
# film / disc moves: the largest copy carries the folder's own name, the rest
|
||||||
|
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
|
||||||
|
|
||||||
|
|
||||||
|
def vext(p):
|
||||||
|
return ext_by.get(p) or os.path.splitext(p)[1].lower()
|
||||||
|
disc_children = collections.defaultdict(list)
|
||||||
|
for q in rows:
|
||||||
|
dr = disc_root(q["path"])
|
||||||
|
if dr:
|
||||||
|
disc_children[dr].append(q["path"])
|
||||||
|
for g in groups:
|
||||||
|
primary_taken = False
|
||||||
|
disc_kinds = set()
|
||||||
|
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
|
||||||
|
dr = disc_root(r["path"])
|
||||||
|
if dr:
|
||||||
|
inner = os.path.basename(dr).upper()
|
||||||
|
if inner in disc_kinds:
|
||||||
|
problems.append(
|
||||||
|
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
|
||||||
|
f"share the folder — discs have no multiple versions")
|
||||||
|
continue
|
||||||
|
disc_kinds.add(inner)
|
||||||
|
for src in disc_children[dr]:
|
||||||
|
tail = os.path.relpath(src, dr)
|
||||||
|
fold = disc_root(src, fold=False)
|
||||||
|
if fold != dr and os.path.dirname(src) != dr:
|
||||||
|
tail = os.path.basename(dr) + "/" + tail
|
||||||
|
book(src, f"{g['dir']}/{tail}", "disc-file")
|
||||||
|
continue
|
||||||
|
label = "" if not primary_taken else f" - {version_label(r)}"
|
||||||
|
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
|
||||||
|
n = 2
|
||||||
|
while dst.lower() in claimed:
|
||||||
|
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
|
||||||
|
n += 1
|
||||||
|
r["dst"] = dst
|
||||||
|
primary_taken = True
|
||||||
|
book(r["path"], dst, "film")
|
||||||
|
g["primary"] = primary_taken
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ extras
|
||||||
|
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
|
||||||
|
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
|
||||||
|
("interviews", r"interview|cast and crew|session piece|commentary"),
|
||||||
|
("featurettes", r"featurette|behind the scenes|making of|documentary"
|
||||||
|
r"|heritage|archival|video archive|music video"),
|
||||||
|
("samples", r"sample"),
|
||||||
|
("shorts", r"\bshort\b|short film"),
|
||||||
|
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
|
||||||
|
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
|
||||||
|
r"|audition|test footage|watermark|workprint|poster"
|
||||||
|
r"|picture in picture|singalong|karaoke|original score")]
|
||||||
|
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
|
||||||
|
r"|\bost\b|music|song", re.I)
|
||||||
|
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
|
||||||
|
|
||||||
|
|
||||||
|
def extras_kind(path):
|
||||||
|
low = path.lower()
|
||||||
|
if THEME_RE.search(low):
|
||||||
|
return "theme-music"
|
||||||
|
if BACKDROP_RE.search(low):
|
||||||
|
return "backdrops"
|
||||||
|
for name, rx in EXTRA_KIND:
|
||||||
|
if re.search(rx, low):
|
||||||
|
return name
|
||||||
|
return "extras"
|
||||||
|
|
||||||
|
|
||||||
|
def attachable_title(path):
|
||||||
|
t, _y = eng.title_of(path)
|
||||||
|
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
|
||||||
|
|
||||||
|
|
||||||
|
for g in groups:
|
||||||
|
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
|
||||||
|
top_groups = collections.defaultdict(list)
|
||||||
|
for g in groups:
|
||||||
|
for t in g["tops"]:
|
||||||
|
top_groups[t].append(g)
|
||||||
|
group_by_key = collections.defaultdict(list)
|
||||||
|
for g in groups:
|
||||||
|
for r in g["rows"]:
|
||||||
|
group_by_key[eng.key_of(r["title"])].append(g)
|
||||||
|
group_by_dir = {}
|
||||||
|
for g in groups:
|
||||||
|
for r in g["rows"]:
|
||||||
|
group_by_dir.setdefault(media_dir(r["path"]), g)
|
||||||
|
dr = disc_root(r["path"])
|
||||||
|
if dr:
|
||||||
|
group_by_dir.setdefault(os.path.dirname(dr), g)
|
||||||
|
|
||||||
|
|
||||||
|
def group_words(g):
|
||||||
|
"""The words that identify a film group: its titles and the names of the
|
||||||
|
folders its files live in."""
|
||||||
|
if "words" not in g:
|
||||||
|
w = set()
|
||||||
|
if g.get("plex"):
|
||||||
|
w |= words_of(g["plex"])
|
||||||
|
for r in g["rows"]:
|
||||||
|
w |= words_of(r["title"])
|
||||||
|
w |= words_of(os.path.basename(media_dir(r["path"])))
|
||||||
|
g["words"] = w
|
||||||
|
return g["words"]
|
||||||
|
|
||||||
|
|
||||||
|
def score_words(text, groups):
|
||||||
|
"""Ranks film groups by how many of their words a path carries."""
|
||||||
|
words = words_of(text)
|
||||||
|
out = [(len(words & group_words(g)), g) for g in groups]
|
||||||
|
out.sort(key=lambda x: -x[0])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def group_for_orphan(path):
|
||||||
|
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
|
||||||
|
several films must be resolved by WORDS, never by position: taking the first
|
||||||
|
film of the folder attached every loose subtitle in a box set to whichever
|
||||||
|
movie happened to sort first."""
|
||||||
|
stem = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
key = attachable_title(path)
|
||||||
|
if key and group_by_key.get(key):
|
||||||
|
cands = group_by_key[key]
|
||||||
|
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
|
||||||
|
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
|
||||||
|
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
|
||||||
|
pool = same_top or cands
|
||||||
|
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
|
||||||
|
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
|
||||||
|
pool = top_groups.get(gtop, [])
|
||||||
|
if len(pool) > 1: # a loose collection folder (box set, bonus
|
||||||
|
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
|
||||||
|
return g if n >= 2 else None # album) is resolved by words, not position
|
||||||
|
if len(pool) == 1:
|
||||||
|
return pool[0]
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
while d and d != ROOT:
|
||||||
|
if d in group_by_dir:
|
||||||
|
return group_by_dir[d]
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
tv_tops = {parts_of(p)[0] for p in tv_set}
|
||||||
|
def film_folder_for(path):
|
||||||
|
"""A movie folder for a lone feature that has no group: derived from its own
|
||||||
|
title when a year proves it, else None (goes to the review pile)."""
|
||||||
|
t, y = eng.title_of(path)
|
||||||
|
if not t or y == "?":
|
||||||
|
return None
|
||||||
|
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
|
||||||
|
mkdirs.add(d)
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
for p in sorted(extras):
|
||||||
|
if disc_root(p):
|
||||||
|
continue
|
||||||
|
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
|
||||||
|
continue # a special inside a TV set: staged with it
|
||||||
|
g = group_for_orphan(p)
|
||||||
|
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
|
||||||
|
stem = os.path.splitext(os.path.basename(p))[0]
|
||||||
|
if g is None:
|
||||||
|
solo = film_folder_for(p)
|
||||||
|
if solo:
|
||||||
|
dst = f"{solo}/{sanitize(stem)}{ext}"
|
||||||
|
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
|
||||||
|
f"{os.path.basename(solo)}")
|
||||||
|
else:
|
||||||
|
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
|
||||||
|
problems.append(f"NO-FILM extra {rel(p)}")
|
||||||
|
else:
|
||||||
|
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
|
||||||
|
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
|
||||||
|
ext, os.path.dirname(p)), "extra")
|
||||||
|
|
||||||
|
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
|
||||||
|
# they belong to the film of the folder they sit in, kept as a named subtree.
|
||||||
|
for q in rows:
|
||||||
|
pth = q["path"]
|
||||||
|
dr = disc_root(pth)
|
||||||
|
if not dr or pth in by_path or pth in booked_src:
|
||||||
|
continue
|
||||||
|
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
|
||||||
|
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
|
||||||
|
if g is None:
|
||||||
|
continue
|
||||||
|
inner = os.path.basename(dr)
|
||||||
|
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
|
||||||
|
f"(a second master: compare before keeping both)")
|
||||||
|
for src in disc_children[dr]:
|
||||||
|
if src not in booked_src:
|
||||||
|
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- sidecars
|
||||||
|
media_by_dir = collections.defaultdict(list)
|
||||||
|
for r in erows:
|
||||||
|
if r["path"] in marks:
|
||||||
|
continue
|
||||||
|
dst = r.get("dst")
|
||||||
|
if dst:
|
||||||
|
media_by_dir[media_dir(r["path"])].append(r)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def sidecar_target(path, e):
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
stem = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
if d == ROOT:
|
||||||
|
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
|
||||||
|
# set of root-level MOVIE files, and scoring a stray name against it
|
||||||
|
# attached every root-level leftover to whichever root-level film was
|
||||||
|
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
|
||||||
|
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
|
||||||
|
return None, None
|
||||||
|
cands = media_by_dir.get(d, [])
|
||||||
|
if not cands:
|
||||||
|
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
|
||||||
|
while nd and nd != ROOT and not media_by_dir.get(nd):
|
||||||
|
nd = os.path.dirname(nd)
|
||||||
|
if media_by_dir.get(nd):
|
||||||
|
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
|
||||||
|
g = group_for_orphan(path) # box set's own root) carries evidence
|
||||||
|
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
|
||||||
|
if g is None or g["dir"] not in dirs:
|
||||||
|
# ...but only a STRONG word match outranks the folder the file
|
||||||
|
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
|
||||||
|
# against a junk group named "3" and escaped its own movie's
|
||||||
|
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
|
||||||
|
# and genuinely belongs to the other film of the set.
|
||||||
|
n = score_words(stem, [g])[0][0] if g else 0
|
||||||
|
if n >= 2:
|
||||||
|
return (g, None)
|
||||||
|
cands = media_by_dir[nd]
|
||||||
|
if not cands:
|
||||||
|
g = group_for_orphan(path)
|
||||||
|
return (g, None) if g else (None, None)
|
||||||
|
if len(cands) > 1:
|
||||||
|
w = [t for t in eng.norm(stem).split() if t not in NOISE]
|
||||||
|
cands = sorted(cands, key=lambda r: (
|
||||||
|
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
|
||||||
|
-r["bytes"]))
|
||||||
|
return (None, cands[0])
|
||||||
|
|
||||||
|
|
||||||
|
for q in rows:
|
||||||
|
p, e = q["path"], q["ext"]
|
||||||
|
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
|
||||||
|
continue
|
||||||
|
if os.path.basename(p).startswith("._"):
|
||||||
|
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
|
||||||
|
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
|
||||||
|
# dead files through four movie folders (and they are video extensions in
|
||||||
|
# the AppleDouble case, so they also skipped every other filter). They
|
||||||
|
# claim nothing: one pile, and they stay in it — nothing is deleted here.
|
||||||
|
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
|
||||||
|
"", os.path.dirname(p)), "junk")
|
||||||
|
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
|
||||||
|
continue
|
||||||
|
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
|
||||||
|
g = group_for_orphan(p)
|
||||||
|
stem = os.path.splitext(os.path.basename(p))[0]
|
||||||
|
if g is not None:
|
||||||
|
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
|
||||||
|
else f"{g['dir']}/{g['base']}{e}", "iso")
|
||||||
|
else:
|
||||||
|
solo = film_folder_for(p)
|
||||||
|
if solo:
|
||||||
|
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
|
||||||
|
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
|
||||||
|
f"folder -> {os.path.basename(solo)}")
|
||||||
|
else:
|
||||||
|
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
|
||||||
|
problems.append(f"NO-FILM disc image {rel(p)}")
|
||||||
|
continue
|
||||||
|
if e in VIDEO_EXT:
|
||||||
|
# A video the engine never probed (unreadable header) and not an AppleDouble
|
||||||
|
# remnant: the plan does not guess at it, so it stays where it is.
|
||||||
|
continue
|
||||||
|
stem = os.path.splitext(os.path.basename(p))[0]
|
||||||
|
g, media = sidecar_target(p, e)
|
||||||
|
if media is not None:
|
||||||
|
base = os.path.splitext(os.path.basename(media["dst"]))[0]
|
||||||
|
d = os.path.dirname(media["dst"])
|
||||||
|
elif g is not None:
|
||||||
|
base, d = g["base"], g["dir"]
|
||||||
|
else:
|
||||||
|
# Nothing names this file's film. NEVER scatter it: an archive set or an
|
||||||
|
# album tree must move as the unit it is, so the whole loose folder goes
|
||||||
|
# to the review pile and a film folder's own leftovers stay with it.
|
||||||
|
ps = parts_of(p)
|
||||||
|
if len(ps) < 2:
|
||||||
|
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
|
||||||
|
elif len(top_groups.get(ps[0], [])):
|
||||||
|
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
|
||||||
|
else:
|
||||||
|
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
|
||||||
|
book(dst=dst, src=p, kind="orphan")
|
||||||
|
problems.append(f"NO-OWNER sidecar {rel(p)}")
|
||||||
|
continue
|
||||||
|
if e in IMG_EXT:
|
||||||
|
low = eng.norm(stem)
|
||||||
|
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
|
||||||
|
kind = "poster" if word else next(
|
||||||
|
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
|
||||||
|
if re.search(r"\btheme\b", low):
|
||||||
|
dst = f"{d}/theme{e}"
|
||||||
|
else:
|
||||||
|
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
|
||||||
|
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
|
||||||
|
e, os.path.dirname(p)), "image")
|
||||||
|
elif e in NFO_EXT:
|
||||||
|
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
|
||||||
|
elif e in SUB_EXT | AUDIO_EXT:
|
||||||
|
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
|
||||||
|
langs = [t.lower() for t in toks if t.lower() in LANGS]
|
||||||
|
flags = [t.lower() for t in toks if t.lower() in FLAGS]
|
||||||
|
keep = [sanitize(t) for t in toks
|
||||||
|
if t.lower() not in langs + flags + list(NOISE)
|
||||||
|
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
|
||||||
|
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
|
||||||
|
if e in SUB_EXT:
|
||||||
|
# subtitles: <movie>/srt[/language]
|
||||||
|
d = f"{d}/srt"
|
||||||
|
pd = os.path.basename(os.path.dirname(p))
|
||||||
|
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
|
||||||
|
d = f"{d}/{sanitize(pd)}" # a real language folder survives
|
||||||
|
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
|
||||||
|
d = f"{d}/theme-music" # bonus audio: one canonical folder
|
||||||
|
pd = os.path.basename(os.path.dirname(p))
|
||||||
|
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
|
||||||
|
d = f"{d}/{sanitize(pd)}"
|
||||||
|
else:
|
||||||
|
pd = os.path.basename(os.path.dirname(p))
|
||||||
|
if not AUDIO_DIR_OK.match(pd):
|
||||||
|
d = f"{d}/{sanitize(pd)}"
|
||||||
|
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
|
||||||
|
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
|
||||||
|
else:
|
||||||
|
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
|
||||||
|
os.path.dirname(p)), "misc")
|
||||||
|
|
||||||
|
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
|
||||||
|
"one-folder-per-movie layout; `_release-group watermarks/`, "
|
||||||
|
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
|
||||||
|
for key in tv_keys:
|
||||||
|
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
|
||||||
|
"these are TV episodes found in the MOVIES library; move the folder "
|
||||||
|
"to a TV library source, else Jellyfin groups them wrongly as one "
|
||||||
|
"movie.")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------- TV
|
||||||
|
for key in tv_keys:
|
||||||
|
items = eps[key]
|
||||||
|
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
|
||||||
|
yr = year.most_common(1)[0][0] if year else None
|
||||||
|
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
|
||||||
|
tops = {parts_of(r["path"])[0] for r in items}
|
||||||
|
members = sorted([r["path"] for r in items]
|
||||||
|
+ [q for q in extras | marks if len(parts_of(q)) > 1
|
||||||
|
and parts_of(q)[0] in tops],
|
||||||
|
key=lambda q: q)
|
||||||
|
for src in members:
|
||||||
|
rel_parts = rel(src).split("/")
|
||||||
|
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
|
||||||
|
book(src, f"{ROOT}/{base}/{tail}", "tv")
|
||||||
|
for p in sorted(marks):
|
||||||
|
stem, ext = os.path.splitext(os.path.basename(p))
|
||||||
|
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
|
||||||
|
os.path.dirname(p)), "watermark")
|
||||||
|
for q in rows:
|
||||||
|
if is_qb(q["path"]):
|
||||||
|
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
|
||||||
|
book(q["path"], q["path"], "qb-partial") # stays: resume data
|
||||||
|
|
||||||
|
# --------------------------------------------------------------- debug hook
|
||||||
|
if os.environ.get("JF_ISO"):
|
||||||
|
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
|
||||||
|
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
|
||||||
|
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
|
||||||
|
raise SystemExit
|
||||||
|
if os.environ.get("JF_DEBUG"):
|
||||||
|
import json
|
||||||
|
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
|
||||||
|
print("at_target", len(at_target))
|
||||||
|
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
|
||||||
|
for up in unp[:20]:
|
||||||
|
d = os.path.dirname(up)
|
||||||
|
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
|
||||||
|
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
|
||||||
|
|
||||||
|
# -------------------------------------------------------------- validation
|
||||||
|
srcs = [m[0] for m in moves]
|
||||||
|
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
|
||||||
|
for s in dup_src:
|
||||||
|
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
|
||||||
|
all_src = set(srcs)
|
||||||
|
untouched = {r["path"] for r in rows} - all_src
|
||||||
|
bad_names, into_alive, lost = [], [], []
|
||||||
|
for src, dst, kind in moves:
|
||||||
|
for comp in dst[len(ROOT) + 1:].split("/"):
|
||||||
|
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
|
||||||
|
or len(comp) > 200):
|
||||||
|
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
|
||||||
|
if dst in untouched: # something staying put is there
|
||||||
|
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
|
||||||
|
if dst.lower() in {s.lower() for s in untouched}:
|
||||||
|
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
|
||||||
|
if dst.startswith(src + "/") and src in all_src:
|
||||||
|
lost.append(f"{rel(src)} moves into its own subtree")
|
||||||
|
# every accounted-for source must exist exactly once in the plan
|
||||||
|
covered = {r["path"] for r in rows}
|
||||||
|
not_planned = sorted(covered - all_src - at_target)
|
||||||
|
|
||||||
|
with open(PLAN, "w") as out:
|
||||||
|
out.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
|
||||||
|
if d.startswith(ROOT + "/") and d != ROOT:
|
||||||
|
out.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for p in sorted(booked_src - {m[0] for m in moves}):
|
||||||
|
out.write(f"keep\t{p}\t{p}\n")
|
||||||
|
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
|
||||||
|
out.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||||||
|
|
||||||
|
kinds = collections.Counter(k for _s, _d, k in moves)
|
||||||
|
films_n = sum(1 for _s, _d, k in moves if k == "film")
|
||||||
|
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
|
||||||
|
with open(EXCEPTS, "w") as out:
|
||||||
|
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
|
||||||
|
out.write(f"kinds={dict(kinds)}\n")
|
||||||
|
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
|
||||||
|
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
|
||||||
|
out.write("\n".join(sorted(set(readmes))) + "\n")
|
||||||
|
out.write("\n=== PROBLEMS ===\n")
|
||||||
|
out.write("\n".join(sorted(set(problems))) + "\n")
|
||||||
|
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
|
||||||
|
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
|
||||||
|
+ "\n".join(sorted(set(into_alive))) + "\n")
|
||||||
|
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
|
||||||
|
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
|
||||||
|
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
|
||||||
|
f"mkdirs={len(mkdirs)} film-files={films_n}")
|
||||||
|
print(f"kinds={dict(kinds)}")
|
||||||
|
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
|
||||||
|
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
|
||||||
|
print(f"left-unmoved={len(not_planned)}")
|
||||||
|
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
|
||||||
|
f"disc-trees={len(disc_children)}")
|
||||||
Executable
+1009
File diff suppressed because it is too large.
Load diff
Executable
+579
@@ -0,0 +1,579 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
r"""The fleet's release-filename title parser: the grouping authority the relayout
|
||||||
|
plan uses for every file the media server never matched.
|
||||||
|
|
||||||
|
report: media-titles.py report [--probed TSV] [--out TSV]
|
||||||
|
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
|
||||||
|
ffprobe census scripts/media-census.py writes.
|
||||||
|
parse: media-titles.py parse NAME [NAME…]
|
||||||
|
what title_of() makes of a name — the way to check a naming accident
|
||||||
|
before it merges two films into one folder.
|
||||||
|
|
||||||
|
scripts/media-plan.py imports this file; over there the paths are arguments.
|
||||||
|
|
||||||
|
Method — three independent signals, so a naming accident cannot merge two films:
|
||||||
|
* franchise KEY : first two significant title tokens. Stable across parts of a
|
||||||
|
collection ("resident evil", "jump street") but NOT film-specific — it only
|
||||||
|
selects candidates.
|
||||||
|
* DURATION : the identity check. Same master encodes differ by at most a
|
||||||
|
couple of seconds; different films / cuts do not collide.
|
||||||
|
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
|
||||||
|
are byte-twins (the hash pass confirms); same duration but different sizes are
|
||||||
|
two different releases of one film (deletable redundancy).
|
||||||
|
|
||||||
|
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
|
||||||
|
would otherwise collapse a whole series into one "duplicate" group.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import collections, os, re, sys, unicodedata
|
||||||
|
|
||||||
|
# Library root: the folder directly under this one names the film, and every
|
||||||
|
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
|
||||||
|
# their own; nothing here reads media, only the census TSVs about it.
|
||||||
|
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
|
||||||
|
|
||||||
|
# Release-tag vocabulary: never part of a title, so the title stops here.
|
||||||
|
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
|
||||||
|
JUNK_RE = re.compile(
|
||||||
|
r"^("
|
||||||
|
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
|
||||||
|
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
|
||||||
|
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
|
||||||
|
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
|
||||||
|
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
|
||||||
|
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
|
||||||
|
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
|
||||||
|
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
|
||||||
|
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
|
||||||
|
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
|
||||||
|
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
|
||||||
|
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
|
||||||
|
r")$", re.I)
|
||||||
|
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
|
||||||
|
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
|
||||||
|
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
|
||||||
|
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
|
||||||
|
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
|
||||||
|
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
|
||||||
|
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
|
||||||
|
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
|
||||||
|
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
|
||||||
|
"vs", "avec", "con"}
|
||||||
|
|
||||||
|
|
||||||
|
def norm(s):
|
||||||
|
s = unicodedata.normalize("NFKD", s)
|
||||||
|
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
|
||||||
|
|
||||||
|
|
||||||
|
def fold_sep(s):
|
||||||
|
"""norm(), plus: every character that is left over is not a letter, digit or
|
||||||
|
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
|
||||||
|
of the word. Release groups file titles with lookalikes for punctuation the
|
||||||
|
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
|
||||||
|
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
|
||||||
|
those (no decomposition mapping), and the scrubber inside title_of DELETES
|
||||||
|
what it does not recognise, so five different V/H/S films squashed into the one
|
||||||
|
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
|
||||||
|
they separate. Accents are gone by then — norm() removed the combining marks —
|
||||||
|
so "Amélie" stays "amelie" and never splits into "ame lie"."""
|
||||||
|
return " ".join("".join(c if c.isalnum() or c == "'" else " "
|
||||||
|
for c in norm(s)).split())
|
||||||
|
|
||||||
|
def title_of(path):
|
||||||
|
"""(title, year) from the basename.
|
||||||
|
|
||||||
|
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
|
||||||
|
(2) Find the boundary: the first release-noise token (resolution / codec /
|
||||||
|
source / audio / language / group vocabulary) ends the title, so trailing
|
||||||
|
tags can never leak in. (3) Collect up to that boundary: leading index
|
||||||
|
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
|
||||||
|
are skipped while nothing is collected; once a real word appears a sentinel
|
||||||
|
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
|
||||||
|
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
|
||||||
|
"""
|
||||||
|
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
|
||||||
|
# docstring): the tokens below are then plain alnum/apostrophe words.
|
||||||
|
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
|
||||||
|
raw = [x for x in SEP_RE.split(name) if x]
|
||||||
|
years = [t for t in raw if YEAR_RE.match(t)]
|
||||||
|
two_year = len(years) > 1 # "2001…1968": one of them is the title
|
||||||
|
toks, year = [], None
|
||||||
|
for t in raw:
|
||||||
|
if YEAR_RE.match(t):
|
||||||
|
year = t # last year wins
|
||||||
|
toks.append("\x00")
|
||||||
|
else:
|
||||||
|
toks.append(t)
|
||||||
|
stop, seen_word = len(toks), False
|
||||||
|
for i, t in enumerate(toks):
|
||||||
|
if t == "\x00":
|
||||||
|
continue
|
||||||
|
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||||
|
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
|
||||||
|
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
|
||||||
|
stop = i
|
||||||
|
break
|
||||||
|
if c.isalpha():
|
||||||
|
seen_word = True
|
||||||
|
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
|
||||||
|
# it is three digits or more ("007", "000", "1408") or the name carries no year
|
||||||
|
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
|
||||||
|
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
|
||||||
|
# loses its zero to the parser — Plex's matched title overrides it anyway.
|
||||||
|
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
|
||||||
|
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
|
||||||
|
keep, num_title = [], None
|
||||||
|
for t in toks[:stop]:
|
||||||
|
if t == "\x00":
|
||||||
|
if keep:
|
||||||
|
break # year closes the title
|
||||||
|
if two_year and t is not None and years[0] != year:
|
||||||
|
# two DIFFERENT years ("2001…1968"): the leading one is the title;
|
||||||
|
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
|
||||||
|
num_title = num_title or years[0]
|
||||||
|
continue # leading year: not the title
|
||||||
|
if track_idx and IDX_RE.match(t) and not keep and t == lead:
|
||||||
|
continue # "03." / "1c." ordering
|
||||||
|
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||||
|
if not keep and not c.isalpha():
|
||||||
|
num_title = num_title or (c or None)
|
||||||
|
continue # numeric title candidate
|
||||||
|
keep.append(c)
|
||||||
|
if len(keep) >= 8:
|
||||||
|
break
|
||||||
|
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
|
||||||
|
title = " ".join(keep)
|
||||||
|
if not keep:
|
||||||
|
# nothing collected: a pure-numeric leading token is the title even when it
|
||||||
|
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
|
||||||
|
for t in raw:
|
||||||
|
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
|
||||||
|
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
|
||||||
|
return c, year or "?" # never "2160p" / "1080": they are not pure digits
|
||||||
|
if (num_title and title and not keep[0].isdigit()
|
||||||
|
and (two_year or not track_idx)):
|
||||||
|
# a leading number that only echoes the release year ("1917.2019.2160p…") is
|
||||||
|
# not a title; one that stands alone ("300.2006…", "1900") is
|
||||||
|
title = f"{num_title} {title}"
|
||||||
|
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
|
||||||
|
return title, year or "?"
|
||||||
|
|
||||||
|
|
||||||
|
# Bonus-material vocabulary: names that say nothing about the film, so an extras
|
||||||
|
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
|
||||||
|
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
|
||||||
|
# Clips) — and the running-time guard in is_extra() carries the rest.
|
||||||
|
SEP_CLS = r"[\s._-]+"
|
||||||
|
|
||||||
|
|
||||||
|
def _bonus_rx(word):
|
||||||
|
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
|
||||||
|
|
||||||
|
|
||||||
|
EXTRA_WORDS = [
|
||||||
|
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
|
||||||
|
"tv spot", "advert", "commercial", "behind the scenes", "making of",
|
||||||
|
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
|
||||||
|
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
|
||||||
|
"menus", "gallery", "still gallery", "stills", "poster", "posters",
|
||||||
|
"montage", "preview", "previews", "epilogue", "visual effects",
|
||||||
|
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
|
||||||
|
"test footage", "screen test", "audition", "cast and crew", "isolated score",
|
||||||
|
"picture in picture", "original score", "session piece", "sample",
|
||||||
|
"watermark",
|
||||||
|
]
|
||||||
|
# Words that describe bonus material which can run LONG (a feature-length
|
||||||
|
# commentary-track cut, a documentary about the film). Trusted only below
|
||||||
|
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
|
||||||
|
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
|
||||||
|
"archival", "documentary"]
|
||||||
|
EXTRA_NAME_RE = re.compile(
|
||||||
|
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
|
||||||
|
EXTRA_LONG_NAME_RE = re.compile(
|
||||||
|
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
|
||||||
|
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
|
||||||
|
# search = it only mentions extras ("... + Extras" box sets hold feature films).
|
||||||
|
EXTRA_FOLDER_RE = re.compile(
|
||||||
|
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
|
||||||
|
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
|
||||||
|
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
|
||||||
|
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
|
||||||
|
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
|
||||||
|
|
||||||
|
# Anything longer than this is a feature film whatever it is called: a 118-minute
|
||||||
|
# file named "The Menu" or "Trailer" is a movie, not bonus material.
|
||||||
|
SHORT_MAX = 45 * 60
|
||||||
|
|
||||||
|
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
|
||||||
|
# commentary cut is an alternate version of the film, not a featurette.
|
||||||
|
LONG_BONUS_MAX = 25 * 60
|
||||||
|
|
||||||
|
|
||||||
|
def name_words(x):
|
||||||
|
"""Bare words of a file or folder name: no release metadata, no year."""
|
||||||
|
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
|
||||||
|
return [w for w in SEP_RE.split(fold_sep(x))
|
||||||
|
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
|
||||||
|
and not JUNK_RE.match(w) and w not in JUNK]
|
||||||
|
|
||||||
|
|
||||||
|
def mentions(short, long_):
|
||||||
|
"""True when one word list appears in the other, anywhere: a release names
|
||||||
|
the film at the front ("Alien (1979) - Trailer") or in the middle
|
||||||
|
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
|
||||||
|
a, b = sorted([short, long_], key=len)
|
||||||
|
if not a:
|
||||||
|
return False
|
||||||
|
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def folder_is_bonus(folder):
|
||||||
|
"""True when a folder name carries bonus vocabulary as WHOLE words: the
|
||||||
|
vocabulary is matched against a padded word string, so "Extras" decides and
|
||||||
|
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
|
||||||
|
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
|
||||||
|
|
||||||
|
|
||||||
|
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
|
||||||
|
"""True for bonus material, which must never be counted as a film: a file
|
||||||
|
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
|
||||||
|
extras otherwise pair with each other across the whole library.
|
||||||
|
|
||||||
|
Evidence, strongest first:
|
||||||
|
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
|
||||||
|
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
|
||||||
|
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
|
||||||
|
unless it sits in an explicit bonus folder and metadata never matched it.
|
||||||
|
Bonus words that describe long-form material (commentary, documentary,
|
||||||
|
interview) are believed only
|
||||||
|
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
|
||||||
|
* The file's OWN name carries bonus vocabulary -> bonus.
|
||||||
|
* ANY folder of the path below the film's own root folder carries bonus
|
||||||
|
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
|
||||||
|
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
|
||||||
|
is exempt because it names the film, and a film may be titled with a
|
||||||
|
bonus word; and a file whose own name IS that folder's film name is the
|
||||||
|
feature, not the bonus ("Ghost in the Shell Movies/Extras/
|
||||||
|
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
|
||||||
|
there stays a film).
|
||||||
|
Bonus vocabulary is matched as whole words only, so the film
|
||||||
|
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
|
||||||
|
the folder rule, and "…Extras" as a fragment of a word is never evidence.
|
||||||
|
"""
|
||||||
|
parts = [q for q in path.split("/") if q]
|
||||||
|
if not parts:
|
||||||
|
return False
|
||||||
|
name = (os.path.splitext(parts[-1])[0]
|
||||||
|
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
|
||||||
|
# A bonus folder anywhere BELOW the film's own folder decides for its
|
||||||
|
# contents; the film's own folder is the film's name, whatever words it
|
||||||
|
# contains, so it is exempt. It is the first component AFTER ROOT's own
|
||||||
|
# components — those are the mount path, not folders of the library.
|
||||||
|
head = len([q for q in root.split("/") if q]) + 1
|
||||||
|
bonus_folder = False
|
||||||
|
for folder in parts[head:-1]:
|
||||||
|
if folder_is_bonus(folder):
|
||||||
|
# The folder's words are often the film's own name ("Angel's Egg
|
||||||
|
# Extras/"), and then they are not bonus evidence. Only WORDS can name
|
||||||
|
# a film: "extras" is release junk that name_words drops, so a folder
|
||||||
|
# like "01. Extras" reduces to its index — and an index matching an
|
||||||
|
# index ("NCOP 01") is a coincidence, not a title.
|
||||||
|
nw = [w for w in name_words(name) if not w.isdigit()]
|
||||||
|
fw = [w for w in name_words(folder) if not w.isdigit()]
|
||||||
|
bonus_folder = not fw or not mentions(nw, fw)
|
||||||
|
break
|
||||||
|
if dur and dur > SHORT_MAX:
|
||||||
|
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
|
||||||
|
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
|
||||||
|
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
|
||||||
|
# it as a film. Long-form bonus words still describe a film, so a
|
||||||
|
# feature-length commentary cut or documentary keeps its runtime veto.
|
||||||
|
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
if EXTRA_NAME_RE.search(name):
|
||||||
|
return True
|
||||||
|
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
|
||||||
|
return True
|
||||||
|
return bonus_folder
|
||||||
|
|
||||||
|
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
|
||||||
|
# "RARBG.com.mp4") into many different films, so its name carries no film
|
||||||
|
# identity at all and it would otherwise pair with itself across the library.
|
||||||
|
WATERMARK_MAX = 20 * 2**20
|
||||||
|
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
|
||||||
|
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
|
||||||
|
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
|
||||||
|
|
||||||
|
|
||||||
|
def is_watermark(path, size):
|
||||||
|
"""True for a release group's own watermark video, not a film or an extra."""
|
||||||
|
return (size < WATERMARK_MAX
|
||||||
|
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def key_of(title):
|
||||||
|
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
|
||||||
|
Deliberately NOT film-specific — it only selects candidates for the duration
|
||||||
|
check, so a two-token collision (two films starting "star trek") cannot by
|
||||||
|
itself create a duplicate group."""
|
||||||
|
toks = [t for t in title.split() if t]
|
||||||
|
words = [t for t in toks if not t.isdigit()]
|
||||||
|
if not words:
|
||||||
|
return " ".join(toks)
|
||||||
|
sig = [t for t in words if t not in STOP] or words
|
||||||
|
return " ".join(sig[:2])
|
||||||
|
|
||||||
|
|
||||||
|
def ep_of(path):
|
||||||
|
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
|
||||||
|
|
||||||
|
The series is the name BEFORE the marker. The box-set convention writes
|
||||||
|
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
|
||||||
|
follows the marker is not the show's name: keying on the whole title gave
|
||||||
|
every episode of one series its own group, so a 124-episode box set never
|
||||||
|
reached the size that calls something a series and each episode was filed
|
||||||
|
as its own film. A file named only by its marker has nothing before it, so
|
||||||
|
its whole title stands in."""
|
||||||
|
name = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
m = EP_RE.search(name)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
head = name[:m.start()].strip(" ._\u2013-")
|
||||||
|
series = title_of(head)[0] if head else None
|
||||||
|
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
|
||||||
|
|
||||||
|
|
||||||
|
def base_words(title, key):
|
||||||
|
"""Title tokens beyond the franchise key — the FILM identity part.
|
||||||
|
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
|
||||||
|
kt = set(key.split())
|
||||||
|
return frozenset(t for t in title.split() if t not in kt and t not in STOP
|
||||||
|
and not t.isdigit() and not YEAR_RE.match(t))
|
||||||
|
|
||||||
|
|
||||||
|
def folder_year(path):
|
||||||
|
"""Soft year evidence: a 19xx/20xx token in any parent directory name
|
||||||
|
(collection folders are written both "(2002)" and "[2009]"; a range such as
|
||||||
|
"(1998-2000)" deliberately yields nothing)."""
|
||||||
|
for d in reversed(os.path.dirname(path).split("/")):
|
||||||
|
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
|
||||||
|
if len(ys) == 1:
|
||||||
|
return ys[0]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def same_film(a, b):
|
||||||
|
"""Two files are copies of ONE film only when the film-identity tokens agree
|
||||||
|
(one side may carry none) AND no year evidence contradicts them. The key says
|
||||||
|
the franchise matches; this says the actual film matches — it is what keeps
|
||||||
|
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
|
||||||
|
Powers apart when their runtimes happen to agree."""
|
||||||
|
ba, bb = a["base"], b["base"]
|
||||||
|
if ba and bb and ba != bb:
|
||||||
|
return False
|
||||||
|
ya = a["year"] if a["year"] != "?" else a["soft"]
|
||||||
|
yb = b["year"] if b["year"] != "?" else b["soft"]
|
||||||
|
if ya and yb and ya != yb:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def dur_close(a, b):
|
||||||
|
if a <= 0 or b <= 0:
|
||||||
|
return True # unknown duration: let titles decide
|
||||||
|
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
|
||||||
|
|
||||||
|
|
||||||
|
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
|
||||||
|
# is folded into ONE entry, described by its longest main-feature clip (measured
|
||||||
|
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
|
||||||
|
BDMV_MAIN = {
|
||||||
|
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
|
||||||
|
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
|
||||||
|
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
|
||||||
|
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load(probed):
|
||||||
|
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
|
||||||
|
BDMV tree collapses to one entry carrying the tree's total size."""
|
||||||
|
trees = {}
|
||||||
|
rows = []
|
||||||
|
for line in open(probed):
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if len(f) < 12:
|
||||||
|
continue
|
||||||
|
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
|
||||||
|
base = os.path.basename(p)
|
||||||
|
if base.startswith("._") or base == ".DS_Store":
|
||||||
|
continue # macOS sidecar, not media
|
||||||
|
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
|
||||||
|
root = p[:m.start()] if m else None
|
||||||
|
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
|
||||||
|
if not root:
|
||||||
|
continue # disc bookkeeping: folds into its tree, never a film
|
||||||
|
if root:
|
||||||
|
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
|
||||||
|
t["bytes"] += int(b)
|
||||||
|
t["audio"].update(a for a in aud.split("+") if a)
|
||||||
|
t["subs"] += int(subs or 0)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
dur = float(dur)
|
||||||
|
except ValueError:
|
||||||
|
dur = 0.0
|
||||||
|
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
|
||||||
|
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
|
||||||
|
for root, t in trees.items():
|
||||||
|
folder = os.path.basename(root)
|
||||||
|
m = BDMV_MAIN.get(folder, {})
|
||||||
|
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
|
||||||
|
name_src=folder, dur=m.get("dur", 0.0),
|
||||||
|
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
|
||||||
|
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
|
||||||
|
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
|
||||||
|
fmt="BDMV tree"))
|
||||||
|
for r in rows:
|
||||||
|
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
|
||||||
|
r["key"] = key_of(r["title"])
|
||||||
|
r["base"] = base_words(r["title"], r["key"])
|
||||||
|
r["soft"] = folder_year(r["path"])
|
||||||
|
r["ep"] = ep_of(r["path"])
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def dur_clusters(items):
|
||||||
|
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
|
||||||
|
buckets come from the franchise key; same_film then rejects the different
|
||||||
|
installments whose runtimes happen to agree."""
|
||||||
|
clusters = []
|
||||||
|
for r in sorted(items, key=lambda r: -r["dur"]):
|
||||||
|
for c in clusters:
|
||||||
|
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
|
||||||
|
c.append(r)
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
clusters.append([r])
|
||||||
|
return clusters
|
||||||
|
|
||||||
|
|
||||||
|
def tier(groups):
|
||||||
|
"""Split same-film/same-duration clusters: DUP = different releases of one
|
||||||
|
film (real, deletable redundancy); REVIEW = identical size AND duration
|
||||||
|
(byte-twins — the hash pass is the authority) or clashing year evidence."""
|
||||||
|
dup, review = [], []
|
||||||
|
for g in groups:
|
||||||
|
g.sort(key=lambda r: -r["bytes"])
|
||||||
|
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
|
||||||
|
years.discard(None)
|
||||||
|
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
|
||||||
|
if len(years) > 1 or spread < 0.005:
|
||||||
|
review.append(g)
|
||||||
|
else:
|
||||||
|
dup.append(g)
|
||||||
|
return dup, review
|
||||||
|
|
||||||
|
|
||||||
|
def redun(g):
|
||||||
|
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
|
||||||
|
|
||||||
|
|
||||||
|
def line(r):
|
||||||
|
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
|
||||||
|
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
|
||||||
|
|
||||||
|
|
||||||
|
def dump(tag, groups, out, note=""):
|
||||||
|
for gi, g in enumerate(groups, 1):
|
||||||
|
g.sort(key=lambda r: -r["bytes"])
|
||||||
|
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
|
||||||
|
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
|
||||||
|
for rank, r in enumerate(g):
|
||||||
|
print(f" {line(r)} [{r['year']}] "
|
||||||
|
f"{r['path'].replace(ROOT + '/', '')}")
|
||||||
|
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
|
||||||
|
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
|
||||||
|
f"{r['path']}\n")
|
||||||
|
|
||||||
|
|
||||||
|
def report(probed, out_path):
|
||||||
|
rows = load(probed)
|
||||||
|
for r in rows:
|
||||||
|
r["mark"] = is_watermark(r["path"], r["bytes"])
|
||||||
|
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
|
||||||
|
marks = [r for r in rows if r["mark"]]
|
||||||
|
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
|
||||||
|
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
|
||||||
|
extras = [r for r in rows if r["extra"]]
|
||||||
|
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
|
||||||
|
f"TV episodes + {len(extras)} bonus/trailers "
|
||||||
|
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
|
||||||
|
f"+ {len(marks)} release-group watermarks")
|
||||||
|
|
||||||
|
out = open(out_path, "w")
|
||||||
|
|
||||||
|
buckets = collections.defaultdict(list)
|
||||||
|
for r in movies:
|
||||||
|
buckets[r["key"]].append(r)
|
||||||
|
cand = {k: v for k, v in buckets.items() if len(v) > 1}
|
||||||
|
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
|
||||||
|
dup, review = tier(clusters)
|
||||||
|
dup.sort(key=lambda g: -redun(g))
|
||||||
|
review.sort(key=lambda g: -redun(g))
|
||||||
|
dump("DUP", dup, out)
|
||||||
|
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
|
||||||
|
|
||||||
|
|
||||||
|
ebuf = collections.defaultdict(list)
|
||||||
|
for r in episodes:
|
||||||
|
ebuf[(r["ep"][0], r["ep"][1])].append(r)
|
||||||
|
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
|
||||||
|
for c in dur_clusters(v) if len(c) > 1])
|
||||||
|
edup.sort(key=lambda g: -redun(g))
|
||||||
|
dump("EP-DUP", edup, out)
|
||||||
|
|
||||||
|
shows = collections.Counter()
|
||||||
|
for r in episodes:
|
||||||
|
shows[r["ep"][0]] += 1
|
||||||
|
multi = {s: n for s, n in shows.items() if n > 4}
|
||||||
|
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
|
||||||
|
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
|
||||||
|
print(f" {n:4d} episodes {s}")
|
||||||
|
out.close()
|
||||||
|
|
||||||
|
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
|
||||||
|
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
|
||||||
|
print(f"REVIEW groups={len(review)} "
|
||||||
|
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
|
||||||
|
print(f"EP-DUP groups={len(edup)} "
|
||||||
|
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def parse(names):
|
||||||
|
for n in names:
|
||||||
|
t, y = title_of(n)
|
||||||
|
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||||
|
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
|
||||||
|
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
|
||||||
|
rep.add_argument("--out", default="/tmp/media-similar.tsv")
|
||||||
|
par = sub.add_parser("parse", help="show the parsed title/year of a name")
|
||||||
|
par.add_argument("names", nargs="+")
|
||||||
|
a = ap.parse_args()
|
||||||
|
if a.cmd == "report":
|
||||||
|
report(a.probed, a.out)
|
||||||
|
else:
|
||||||
|
parse(a.names)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+281
@@ -0,0 +1,281 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""media-tv.py — move TV episodes out of the movie library into a TV library.
|
||||||
|
|
||||||
|
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
|
||||||
|
`_TV needs a TV library - <series>` pile for a human. This is the other half of
|
||||||
|
that refusal. It walks the movie library, groups the files whose own name
|
||||||
|
carries an episode marker by SERIES, and writes a plan that places each series
|
||||||
|
in the shape both Jellyfin and Plex read without help:
|
||||||
|
|
||||||
|
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
|
||||||
|
|
||||||
|
It writes a plan and nothing else. Every rename belongs to
|
||||||
|
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
|
||||||
|
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
|
||||||
|
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
|
||||||
|
the executor's own path guard still has teeth, and give it its own journal —
|
||||||
|
one journal is one undo, and a TV move must not be welded to a movie relayout:
|
||||||
|
|
||||||
|
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
|
||||||
|
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
|
||||||
|
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||||
|
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
|
||||||
|
python3 scripts/media-apply.py sweep --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||||
|
|
||||||
|
Guards, and why each one is here:
|
||||||
|
|
||||||
|
* The same five-file threshold as the planner. An episode-shaped name on one
|
||||||
|
file is not a series — `The Godfather Part 2` is a film — so a smaller group
|
||||||
|
is reported and left exactly where it is.
|
||||||
|
* An episode whose name carries no season number is reported and left, never
|
||||||
|
filed into an invented `Season 01`.
|
||||||
|
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
|
||||||
|
episode; an episode without its subtitles is a half-move.
|
||||||
|
* A group that would claim one destination twice is not resolved here. The
|
||||||
|
executor refuses the whole plan before its first rename, which is the right
|
||||||
|
outcome: nobody but Frank decides which copy of an episode is the keeper.
|
||||||
|
* An episode the destination library already holds at the same byte count is
|
||||||
|
left where it is. The TV library has held a complete Monk box set for years
|
||||||
|
while the movie library carried a second copy of all 124 episodes; a mover
|
||||||
|
that did not look where it was filing would have put both inside one library.
|
||||||
|
* A symlink is never followed and never moved.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import collections
|
||||||
|
import importlib.util
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
|
||||||
|
help="the movie library to empty of TV")
|
||||||
|
ap.add_argument("--library", default="/mnt/entropy/TVShows",
|
||||||
|
help="the TV library to fill (a different share or folder)")
|
||||||
|
ap.add_argument("--out", default=None,
|
||||||
|
help="plan TSV for scripts/media-apply.py"
|
||||||
|
" (default: media-tv-plan.tsv beside --root)")
|
||||||
|
ap.add_argument("--min", type=int, default=5,
|
||||||
|
help="episode files that make something a series")
|
||||||
|
a = ap.parse_args()
|
||||||
|
if not a.out:
|
||||||
|
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
|
||||||
|
"media-tv-plan.tsv")
|
||||||
|
return a
|
||||||
|
|
||||||
|
|
||||||
|
ARGS = parse_args()
|
||||||
|
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||||||
|
spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||||||
|
eng = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(eng)
|
||||||
|
|
||||||
|
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
||||||
|
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
|
||||||
|
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
|
||||||
|
# Sidecars that follow an episode; macOS forks and resume data do not.
|
||||||
|
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
|
||||||
|
# A box set that stamps the show's year into every file name; the folder's year
|
||||||
|
# must not be added on top of it.
|
||||||
|
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
|
||||||
|
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
|
||||||
|
"by", "de", "la", "vs"}
|
||||||
|
|
||||||
|
|
||||||
|
def clean(name):
|
||||||
|
"""A name the filesystem and both matchers can read (same law as the plan)."""
|
||||||
|
name = BAD_CHARS.sub("-", name).replace("\\", "-")
|
||||||
|
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
|
||||||
|
return name[:180] or "untitled"
|
||||||
|
|
||||||
|
|
||||||
|
def episode_name(path):
|
||||||
|
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
|
||||||
|
|
||||||
|
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
|
||||||
|
already carries spaces is left exactly as it was written."""
|
||||||
|
stem, ext = os.path.splitext(os.path.basename(path))
|
||||||
|
if " " not in stem and stem.count(".") >= 2:
|
||||||
|
stem = stem.replace(".", " ")
|
||||||
|
return clean(stem) + ext.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def head_of(path):
|
||||||
|
"""The publisher's series name: the text BEFORE the episode marker.
|
||||||
|
|
||||||
|
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
|
||||||
|
after the marker is not the show's name, and case is kept because it is the
|
||||||
|
library's own spelling, not a normalised key."""
|
||||||
|
name = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
m = eng.EP_RE.search(name)
|
||||||
|
head = name[:m.start()] if m else name
|
||||||
|
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def title_series(head):
|
||||||
|
"""The series folder in the case a TV library is written in.
|
||||||
|
|
||||||
|
Only a name with no capital at all is retitled: a box set that was
|
||||||
|
lower-cased on the way in would otherwise file `monk (2002)` among
|
||||||
|
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
|
||||||
|
exactly as it was written."""
|
||||||
|
if any(c.isupper() for c in head):
|
||||||
|
return head # the publisher wrote it with case; keep it
|
||||||
|
words = head.split()
|
||||||
|
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
|
||||||
|
for i, w in enumerate(words))
|
||||||
|
|
||||||
|
|
||||||
|
def season_episode(token):
|
||||||
|
"""(season, episode) an episode token states, or None when it states none."""
|
||||||
|
m = SEASON_RE.match(token)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
|
||||||
|
|
||||||
|
|
||||||
|
def sidecars(path):
|
||||||
|
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
|
||||||
|
d, stem = os.path.split(path)
|
||||||
|
stem = os.path.splitext(stem)[0]
|
||||||
|
out = []
|
||||||
|
try:
|
||||||
|
entries = sorted(os.listdir(d))
|
||||||
|
except OSError:
|
||||||
|
return out
|
||||||
|
for e in entries:
|
||||||
|
p = os.path.join(d, e)
|
||||||
|
if p == path or os.path.islink(p) or not os.path.isfile(p):
|
||||||
|
continue
|
||||||
|
s, _ext = os.path.splitext(e)
|
||||||
|
if s == stem and not SKIP_SIDECAR.match(e):
|
||||||
|
out.append(p)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def episodes_under(root):
|
||||||
|
"""(path, series, token) for every episode-shaped video file under root."""
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
dirnames[:] = sorted(d for d in dirnames
|
||||||
|
if not d.startswith(".") and d != "@eaDir")
|
||||||
|
for f in filenames:
|
||||||
|
p = os.path.join(dirpath, f)
|
||||||
|
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
|
||||||
|
continue
|
||||||
|
e = eng.ep_of(p)
|
||||||
|
if e:
|
||||||
|
yield p, e[0], e[1]
|
||||||
|
|
||||||
|
|
||||||
|
def walk(root, library):
|
||||||
|
"""Episode files under ROOT keyed by series, ignoring the destination."""
|
||||||
|
groups = collections.defaultdict(list)
|
||||||
|
lib = os.path.abspath(library)
|
||||||
|
for p, series, token in episodes_under(root):
|
||||||
|
if os.path.abspath(p).startswith(lib + os.sep):
|
||||||
|
continue
|
||||||
|
groups[series].append((p, token))
|
||||||
|
return groups
|
||||||
|
|
||||||
|
|
||||||
|
def holdings(library):
|
||||||
|
"""What the destination already carries: (series, season, episode) -> sizes.
|
||||||
|
|
||||||
|
The one expensive read here, and the one that matters. The movie library
|
||||||
|
holds a complete Monk box set that the TV library has held for years: 124
|
||||||
|
episodes, the same episode list, identical byte counts. Moving it would
|
||||||
|
have filed a second copy of a whole show into the library that already had
|
||||||
|
it. Byte count is the test on purpose — two encodes of one episode are two
|
||||||
|
files and both are wanted; the same bytes are the same file."""
|
||||||
|
out = collections.defaultdict(set)
|
||||||
|
for p, series, token in episodes_under(library):
|
||||||
|
se = season_episode(token)
|
||||||
|
if se is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out[(series, *se)].add(os.path.getsize(p))
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
|
||||||
|
groups = walk(root, library)
|
||||||
|
held_here = holdings(library)
|
||||||
|
mkdirs = {library}
|
||||||
|
rows, lines, skipped = set(), [], []
|
||||||
|
for series, items in sorted(groups.items()):
|
||||||
|
if len(items) < ARGS.min:
|
||||||
|
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
|
||||||
|
f"episode-shaped file{'s' if len(items) != 1 else ''},"
|
||||||
|
f" below {ARGS.min}: not a series")
|
||||||
|
continue
|
||||||
|
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
|
||||||
|
if y)
|
||||||
|
year = years.most_common(1)[0][0] if years else None
|
||||||
|
heads = collections.Counter(head_of(p) for p, _t in items)
|
||||||
|
head = heads.most_common(1)[0][0]
|
||||||
|
if year and YEAR_TAIL.search(head):
|
||||||
|
year = None
|
||||||
|
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
|
||||||
|
series_dir = os.path.join(library, name)
|
||||||
|
moved = held = 0
|
||||||
|
for path, token in items:
|
||||||
|
se = season_episode(token)
|
||||||
|
if se is None:
|
||||||
|
skipped.append(f"skipped: {path} — no season number in its name,"
|
||||||
|
" nothing here invents one")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
size = os.path.getsize(path)
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
if size in held_here.get((series, *se), ()):
|
||||||
|
held += 1
|
||||||
|
continue
|
||||||
|
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
|
||||||
|
mkdirs.update([series_dir, dst_dir])
|
||||||
|
base = episode_name(path)
|
||||||
|
rows.add((path, os.path.join(dst_dir, base)))
|
||||||
|
moved += 1
|
||||||
|
for side in sidecars(path):
|
||||||
|
rows.add((side, os.path.join(
|
||||||
|
dst_dir, clean(os.path.splitext(base)[0]
|
||||||
|
+ os.path.splitext(side)[1].lower()))))
|
||||||
|
if moved:
|
||||||
|
lines.append(f"{name}: {moved} episodes"
|
||||||
|
f"{'' if year else ' (no year in any folder name)'}"
|
||||||
|
f" -> {series_dir}")
|
||||||
|
if held:
|
||||||
|
lines.append(f"{name}: {held}"
|
||||||
|
f" {'episode' if held == 1 else 'episodes'} already in"
|
||||||
|
f" {library} at the same byte count — left there")
|
||||||
|
|
||||||
|
# An empty plan creates nothing, not even the library folder: a run that
|
||||||
|
# found no series must not be able to leave a stray directory behind.
|
||||||
|
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
|
||||||
|
with open(ARGS.out, "w") as out:
|
||||||
|
out.write("action\tsrc\tdst\tkind\n")
|
||||||
|
for d in dirs:
|
||||||
|
out.write(f"mkdir\t\t{d}\tdir\n")
|
||||||
|
for src, dst in sorted(rows):
|
||||||
|
out.write(f"move\t{src}\t{dst}\ttv\n")
|
||||||
|
|
||||||
|
for line in lines + skipped:
|
||||||
|
print(line)
|
||||||
|
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
|
||||||
|
if rows:
|
||||||
|
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
|
||||||
|
f" --root {os.path.dirname(root)}"
|
||||||
|
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
|
||||||
|
" (add --apply to run it)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Executable
+186
@@ -0,0 +1,186 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Prove a relayout landed, using the media server's own tables.
|
||||||
|
|
||||||
|
A scan answers `204` whether or not it found anything, and a rename that the
|
||||||
|
scanner has not followed looks exactly like a rename that never happened from
|
||||||
|
the outside. The only honest proof is the run's journal replayed against the
|
||||||
|
database the server actually serves from: every `move` row must have a row at
|
||||||
|
its destination, and NO row may still name its source — a source row is a scan
|
||||||
|
that has not caught up, and it is the one signature of a library pointing at
|
||||||
|
files that no longer exist.
|
||||||
|
|
||||||
|
Two doors, both read-only, both without a lock while the server runs
|
||||||
|
(`mode=ro&immutable=1`; there is no `sqlite3` binary on these hosts):
|
||||||
|
|
||||||
|
jellyfin `BaseItems.Path`. The container binds the share at `/data`, so the
|
||||||
|
database holds `/data/<library>/…` where the journal holds
|
||||||
|
`/mnt/entropy/<library>/…`. Translate, or every row reads as missing.
|
||||||
|
plex `media_parts.file`, joined up to `metadata_items`. Plex is bound at
|
||||||
|
the host path, so its rows match the journal verbatim. Filter
|
||||||
|
`deleted_at is null` on BOTH tables — a deleted scan leaves its parts
|
||||||
|
behind, and one of those rows is not proof anything landed.
|
||||||
|
|
||||||
|
Neither server holds a row for everything the journal renames: a `.log`, `.cue`
|
||||||
|
or cover image has no row to land in, so its absence is not a fault. The report
|
||||||
|
therefore breaks the missing rows down by extension, and `--ignore-ext` names
|
||||||
|
the extensions this server does not index, which are then counted separately and
|
||||||
|
kept out of the verdict.
|
||||||
|
|
||||||
|
Exit codes: 0 every move landed; 1 something is stale or missing (the details are
|
||||||
|
printed); 2 the journal or the database is not usable at all.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from collections import Counter
|
||||||
|
|
||||||
|
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
|
||||||
|
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
|
||||||
|
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
|
||||||
|
|
||||||
|
# Per kind: the table whose columns are required, those columns, and the query
|
||||||
|
# that returns every path the server believes it has.
|
||||||
|
KINDS = {
|
||||||
|
"jellyfin": {
|
||||||
|
"table": "BaseItems",
|
||||||
|
"columns": {"Path"},
|
||||||
|
"sql": "select Path from BaseItems",
|
||||||
|
"default_db": JF_DB,
|
||||||
|
},
|
||||||
|
"plex": {
|
||||||
|
"table": "media_parts",
|
||||||
|
"columns": {"file", "media_item_id", "deleted_at"},
|
||||||
|
"sql": ("select p.file from media_parts p"
|
||||||
|
" join media_items m on m.id = p.media_item_id"
|
||||||
|
" join metadata_items mi on mi.id = m.metadata_item_id"
|
||||||
|
" where p.deleted_at is null and mi.deleted_at is null"),
|
||||||
|
"default_db": PLEX_DB,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def die(msg):
|
||||||
|
print(f"media-verify: {msg}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def read_moves(path):
|
||||||
|
"""The (src, dst) pairs of a journal, in the order the run performed them."""
|
||||||
|
moves = []
|
||||||
|
try:
|
||||||
|
fh = open(path, encoding="utf-8")
|
||||||
|
except OSError as e:
|
||||||
|
die(f"cannot read the journal {path}: {e.strerror}")
|
||||||
|
with fh:
|
||||||
|
for n, line in enumerate(fh, 1):
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
rec = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
die(f"{path} line {n} is not a journal record")
|
||||||
|
if rec.get("op") == "move":
|
||||||
|
if not rec.get("src") or not rec.get("dst"):
|
||||||
|
die(f"{path} line {n} is a move without both src and dst")
|
||||||
|
moves.append((rec["src"], rec["dst"]))
|
||||||
|
return moves
|
||||||
|
|
||||||
|
|
||||||
|
def server_paths(db, kind):
|
||||||
|
"""Every path the server holds, or a refusal that names what is missing."""
|
||||||
|
spec = KINDS[kind]
|
||||||
|
try:
|
||||||
|
con = sqlite3.connect(f"file:{db}?mode=ro&immutable=1", uri=True)
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
die(f"cannot open {db}: {e}")
|
||||||
|
with con:
|
||||||
|
have = {r[1] for r in con.execute(f"pragma table_info({spec['table']})")}
|
||||||
|
if not spec["columns"] <= have:
|
||||||
|
die(f"{db} is not a {kind} library database ({spec['table']} has no "
|
||||||
|
f"{','.join(sorted(spec['columns'] - have))})")
|
||||||
|
try:
|
||||||
|
return {r[0] for r in con.execute(spec["sql"]) if r[0]}
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
die(f"{db} refused the {kind} query: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def to_host(path, container, share):
|
||||||
|
"""The database's path in the journal's terms; identity when it is not a
|
||||||
|
container-side path."""
|
||||||
|
if not container:
|
||||||
|
return path
|
||||||
|
prefix = container.rstrip("/") + "/"
|
||||||
|
if path.startswith(prefix):
|
||||||
|
return share.rstrip("/") + "/" + path[len(prefix):]
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
ap.add_argument("--journal", required=True,
|
||||||
|
help="the run's journal (scripts/media-apply.py writes it)")
|
||||||
|
ap.add_argument("--db-kind", choices=sorted(KINDS), required=True)
|
||||||
|
ap.add_argument("--db", default=None, help="the server's database on this host")
|
||||||
|
ap.add_argument("--share", default="/mnt/entropy",
|
||||||
|
help="the share every journal path lives under")
|
||||||
|
ap.add_argument("--container", default=None,
|
||||||
|
help="what the share is called inside the container; empty"
|
||||||
|
" compares verbatim (default /data for jellyfin, nothing"
|
||||||
|
" for plex)")
|
||||||
|
ap.add_argument("--show", type=int, default=20,
|
||||||
|
help="how many offenders to name before saying 'and N more'")
|
||||||
|
ap.add_argument("--ignore-ext", default="",
|
||||||
|
help="comma-separated extensions this server does not index,"
|
||||||
|
" kept out of the verdict (e.g. log,cue,nfo,srt)")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
db = args.db or KINDS[args.db_kind]["default_db"]
|
||||||
|
container = ("/data" if args.db_kind == "jellyfin" else "") \
|
||||||
|
if args.container is None else args.container
|
||||||
|
ignore = set()
|
||||||
|
for ext in args.ignore_ext.split(","):
|
||||||
|
ext = ext.strip().lower()
|
||||||
|
if ext:
|
||||||
|
ignore.add(ext if ext.startswith(".") else "." + ext)
|
||||||
|
|
||||||
|
moves = read_moves(args.journal)
|
||||||
|
stored = {to_host(p, container, args.share) for p in server_paths(db, args.db_kind)}
|
||||||
|
|
||||||
|
landed, stale, missing, ignored = 0, [], [], 0
|
||||||
|
for src, dst in moves:
|
||||||
|
if os.path.splitext(dst)[1].lower() in ignore:
|
||||||
|
ignored += 1
|
||||||
|
continue
|
||||||
|
at_dst, at_src = dst in stored, src in stored
|
||||||
|
if at_dst and not at_src:
|
||||||
|
landed += 1
|
||||||
|
continue
|
||||||
|
if at_src:
|
||||||
|
stale.append(src)
|
||||||
|
if not at_dst:
|
||||||
|
missing.append(dst)
|
||||||
|
|
||||||
|
print(f"{args.db_kind}: {len(moves)} moves checked")
|
||||||
|
print(f"landed: {landed}")
|
||||||
|
print(f"still at src: {len(stale)}")
|
||||||
|
print(f"missing from the DB: {len(missing)}")
|
||||||
|
if ignored:
|
||||||
|
print(f"not indexed (extension ignored): {ignored}")
|
||||||
|
if missing:
|
||||||
|
exts = Counter(os.path.splitext(d)[1].lower() or "(no extension)"
|
||||||
|
for d in missing)
|
||||||
|
print("missing by extension: "
|
||||||
|
+ ", ".join(f"{e} {n}" for e, n in sorted(exts.items())))
|
||||||
|
for label, items in (("stale", stale), ("missing", missing)):
|
||||||
|
for path in items[:args.show]:
|
||||||
|
print(f" {label}: {path}")
|
||||||
|
if len(items) > args.show:
|
||||||
|
print(f" {label}: … and {len(items) - args.show} more")
|
||||||
|
return 1 if (stale or missing) else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Executable
+241
@@ -0,0 +1,241 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
|
||||||
|
#
|
||||||
|
# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
|
||||||
|
# ONLY when it sits next to the media file. The media lives on the RAID6 array at
|
||||||
|
# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
|
||||||
|
# container, and slow to seek for a thousand tiny XML reads. So neither media
|
||||||
|
# server binds the array any more: they bind an OverlayFS merge of it with a
|
||||||
|
# writable layer on the NVMe.
|
||||||
|
#
|
||||||
|
# lowerdir /mnt/entropy RAID6 (ROTA=1) — source of truth, never written
|
||||||
|
# upperdir /var/lib/media-meta/upper NVMe — every sidecar and image byte
|
||||||
|
# workdir /var/lib/media-meta/work NVMe — kernel bookkeeping
|
||||||
|
# Where /var/lib/media-meta/view the merged tree jellyfin and plex bind
|
||||||
|
#
|
||||||
|
# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
|
||||||
|
# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
|
||||||
|
#
|
||||||
|
# Consequences worth knowing:
|
||||||
|
# * A sidecar write is a few KB onto the SSD; the media bytes never move and the
|
||||||
|
# array is still never written by anything automated.
|
||||||
|
# * Deleting through the view writes a whiteout into the upper layer instead of
|
||||||
|
# deleting from the array. Nothing should delete through the view: deletions
|
||||||
|
# go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
|
||||||
|
# * inotify events on the lower layer do not cross the merge, so a file that
|
||||||
|
# arrives on the raw array is invisible to a file-system watcher in the view:
|
||||||
|
# trigger a library refresh (see the Media Relayout page).
|
||||||
|
#
|
||||||
|
# Subcommands
|
||||||
|
# plan [--upper-base D] [--lower D] print the resolved shape, touch nothing
|
||||||
|
# verify prove the live mount IS that shape
|
||||||
|
# install [--upper-base D] create the layers, install+enable+start the unit
|
||||||
|
# reap [--apply] [--upper-base D] [--lower D]
|
||||||
|
# list upper-layer entries whose media is gone
|
||||||
|
# status one line per fact a human wants
|
||||||
|
#
|
||||||
|
# install must run as root (it writes /etc/systemd/system and mounts); the others
|
||||||
|
# are read-only and safe unprivileged.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
LOWER=/mnt/entropy
|
||||||
|
UPPER_BASE=/var/lib/media-meta
|
||||||
|
VIEW=$UPPER_BASE/view
|
||||||
|
# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
|
||||||
|
# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
|
||||||
|
# repo copy keeps a readable name; install writes it under the escaped one.
|
||||||
|
UNIT='var-lib-media\x2dmeta-view.mount'
|
||||||
|
UNIT_DIR=/etc/systemd/system
|
||||||
|
# The rootless containers write as this host uid (the deploy user). The upper
|
||||||
|
# layer has to accept those writes, so it is owned by them.
|
||||||
|
MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
|
||||||
|
MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
||||||
|
|
||||||
|
die() {
|
||||||
|
printf 'media-view: %s\n' "$*" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd="${1:-}"
|
||||||
|
[ -n "$cmd" ] || usage
|
||||||
|
shift || true
|
||||||
|
|
||||||
|
upper_base="$UPPER_BASE"
|
||||||
|
lower="$LOWER"
|
||||||
|
apply=0
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
|
||||||
|
--lower) [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
|
||||||
|
--apply) apply=1; shift ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
upper="$upper_base/upper"
|
||||||
|
work="$upper_base/work"
|
||||||
|
|
||||||
|
# The device backing a path, and whether the kernel calls it rotational.
|
||||||
|
rotational_of() {
|
||||||
|
local src rota
|
||||||
|
src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
|
||||||
|
[ -n "$src" ] || die "cannot find a block device behind $1"
|
||||||
|
rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
|
||||||
|
[ -n "$rota" ] || die "cannot read the rotational flag of $src"
|
||||||
|
printf '%s %s\n' "$src" "$rota"
|
||||||
|
}
|
||||||
|
|
||||||
|
require_upper_base() {
|
||||||
|
[ -d "$upper_base" ] \
|
||||||
|
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
|
||||||
|
}
|
||||||
|
|
||||||
|
do_plan() {
|
||||||
|
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
|
||||||
|
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
|
||||||
|
}
|
||||||
|
|
||||||
|
do_verify() {
|
||||||
|
local fail=0 fst opts src rota
|
||||||
|
if ! mountpoint -q "$VIEW"; then
|
||||||
|
printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
|
||||||
|
printf ' library that scans itself empty, not a metadata view.\n' >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
|
||||||
|
if [ "$fst" != overlay ]; then
|
||||||
|
printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
|
||||||
|
local want
|
||||||
|
for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
|
||||||
|
case ",$opts," in
|
||||||
|
*",${want},"*) ;;
|
||||||
|
*) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
|
||||||
|
fail=1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
|
||||||
|
if [ "$rota" != 0 ]; then
|
||||||
|
printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
|
||||||
|
printf ' belongs on the SSD, not on the array.\n' >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
if [ ! -w "$UPPER_BASE/upper" ]; then
|
||||||
|
printf 'refused: %s is not writable\n' "$upper" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
[ "$fail" -eq 0 ] || return 1
|
||||||
|
printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
|
||||||
|
}
|
||||||
|
|
||||||
|
do_install() {
|
||||||
|
[ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
|
||||||
|
require_upper_base
|
||||||
|
mountpoint -q "$LOWER" \
|
||||||
|
|| die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
|
||||||
|
local src rota
|
||||||
|
read -r src rota <<<"$(rotational_of "$upper_base")"
|
||||||
|
[ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
|
||||||
|
[ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
|
||||||
|
# systemd will not create a mount point; a missing directory fails the start.
|
||||||
|
install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
|
||||||
|
install -d -m 0700 "$upper_base/.AGENTTRASH"
|
||||||
|
install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable "$UNIT" >/dev/null 2>&1 || true
|
||||||
|
systemctl start "$UNIT"
|
||||||
|
do_verify
|
||||||
|
}
|
||||||
|
|
||||||
|
# An upper-layer DIRECTORY with no counterpart below is metadata for a media
|
||||||
|
# folder that has been moved or renamed on the RAW path — a ghost the media
|
||||||
|
# servers would still browse. Files are never candidates on their own: a sidecar
|
||||||
|
# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
|
||||||
|
# is for. A copy-up whose lower file was deleted by hand is the one case this
|
||||||
|
# leaves in place; the raw path is where deletions belong, so it stays visible.
|
||||||
|
do_reap() {
|
||||||
|
[ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
|
||||||
|
stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
|
||||||
|
count=0
|
||||||
|
ghosts=0
|
||||||
|
reap_walk ""
|
||||||
|
if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
|
||||||
|
printf 'reap: no ghosts under %s\n' "$upper"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if [ "$ghosts" -gt 0 ]; then
|
||||||
|
printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
|
||||||
|
"$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
|
||||||
|
printf ' Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
|
||||||
|
printf ' Deletions belong in .AGENTTRASH on the raw path.\n' >&2
|
||||||
|
fi
|
||||||
|
if [ "$count" -eq 0 ]; then return 0; fi
|
||||||
|
if [ "$apply" -eq 1 ]; then
|
||||||
|
printf 'reap: moved %d entr%s to %s\n' "$count" \
|
||||||
|
"$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
|
||||||
|
else
|
||||||
|
printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
|
||||||
|
"$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Descend only where the lower layer still has the directory: a directory that is
|
||||||
|
# missing below is itself the ghost, and its whole subtree goes with it.
|
||||||
|
reap_walk() {
|
||||||
|
local rel="$1" entries abs r name
|
||||||
|
entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
|
||||||
|
[ -n "$entries" ] || return 0
|
||||||
|
while IFS= read -r abs; do
|
||||||
|
r="${abs#"$upper"/}"
|
||||||
|
name="${abs##*/}"
|
||||||
|
if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
|
||||||
|
ghosts=$((ghosts + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if [ -e "$lower/$r" ]; then
|
||||||
|
[ -d "$abs" ] && reap_walk "$r"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
[ -d "$abs" ] || continue
|
||||||
|
count=$((count + 1))
|
||||||
|
printf '%s\n' "$r"
|
||||||
|
if [ "$apply" -eq 1 ]; then
|
||||||
|
mkdir -p "$stamp/$(dirname "$r")"
|
||||||
|
mv "$abs" "$stamp/$r"
|
||||||
|
fi
|
||||||
|
done <<<"$entries"
|
||||||
|
}
|
||||||
|
|
||||||
|
do_status() {
|
||||||
|
if mountpoint -q "$VIEW"; then
|
||||||
|
printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
|
||||||
|
else
|
||||||
|
printf 'view: %s NOT MOUNTED\n' "$VIEW"
|
||||||
|
fi
|
||||||
|
if [ -d "$upper" ]; then
|
||||||
|
printf 'upper: %s %s files %s\n' "$upper" \
|
||||||
|
"$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
|
||||||
|
else
|
||||||
|
printf 'upper: %s missing\n' "$upper"
|
||||||
|
fi
|
||||||
|
systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "$cmd" in
|
||||||
|
plan) do_plan ;;
|
||||||
|
verify) do_verify ;;
|
||||||
|
install) do_install ;;
|
||||||
|
reap) do_reap ;;
|
||||||
|
status) do_status ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
@@ -12,8 +12,13 @@
|
|||||||
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
|
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
|
||||||
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
|
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
|
||||||
#
|
#
|
||||||
# 1. the runner must still be alive and its container must answer /health, or this script
|
# 1. the runner must still be alive and its container must answer /health AND own the
|
||||||
# kills it, restores the service and exits non-zero — no push, ever;
|
# listening socket, or this script kills it, restores the service and exits non-zero —
|
||||||
|
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
|
||||||
|
# first, the dying process keeps its listener for a moment, and a rig container that
|
||||||
|
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
|
||||||
|
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
|
||||||
|
# then the pusher refused on the container name and the window died at second zero);
|
||||||
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
|
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
|
||||||
# marker names this container, and the container is running (scripts/calib-push.py
|
# marker names this container, and the container is running (scripts/calib-push.py
|
||||||
# --tap-marker), so a second lock survives a mistake in this file;
|
# --tap-marker), so a second lock survives a mistake in this file;
|
||||||
@@ -35,20 +40,26 @@ MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
|
|||||||
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||||
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
||||||
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
||||||
KLDREF="$CAMPAIGN/kld-ref/kld.json"
|
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
|
||||||
CALIB="$CAMPAIGN/calib"
|
|
||||||
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
|
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
|
||||||
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
|
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
|
||||||
RIG=qwen38-bf16
|
RIG=qwen38-bf16
|
||||||
SERVICE=frank-archon
|
SERVICE=frank-archon
|
||||||
PY="${PY:-python3}"
|
PY="${PY:-python3}"
|
||||||
|
|
||||||
# ---- preflight: everything this window needs, before a card is touched -------
|
# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
|
||||||
[ -f "$KLDREF" ] || { echo "REFUSED: no reference recording at $KLDREF — record it first (scripts/run-qwen38-bf16.sh kldref); this window only fills the calibration tap" >&2; exit 2; }
|
# The checks that used to live here were proxies, and each lied at least once: the space check
|
||||||
|
# asked a directory that merely LOOKED related, `/health` answered for a dying production
|
||||||
|
# process, `podman inspect` named a container that existed but had bound nothing. The shared
|
||||||
|
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
|
||||||
|
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
|
||||||
|
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
|
||||||
|
# keeps an existing tap, so the tap-empty fact is waived.
|
||||||
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
|
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
|
||||||
[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; }
|
FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
|
||||||
[ -d "$CALIB" ] || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; }
|
Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
|
||||||
[ -w "$CALIB" ] || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; }
|
Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
|
||||||
|
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
|
||||||
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
|
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
|
||||||
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
|
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
|
||||||
|
|
||||||
@@ -69,8 +80,18 @@ say "phase=calib serve begin"
|
|||||||
bash scripts/run-qwen38-bf16.sh calib & spid=$!
|
bash scripts/run-qwen38-bf16.sh calib & spid=$!
|
||||||
|
|
||||||
rig_up() {
|
rig_up() {
|
||||||
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true \
|
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
|
||||||
&& curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health
|
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
|
||||||
|
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
|
||||||
|
# dying production process keeps its listener for a moment while the freshly created
|
||||||
|
# rig container is still loading — so `container running` + `/health` 200 both hold
|
||||||
|
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
|
||||||
|
# pusher refused on the container name and the window died at second zero. Ask the kernel
|
||||||
|
# who owns the listening socket and require it to be a rig process.
|
||||||
|
local lpid
|
||||||
|
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
|
||||||
|
[ -n "${lpid:-}" ] || return 1
|
||||||
|
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
|
||||||
}
|
}
|
||||||
|
|
||||||
waited=0
|
waited=0
|
||||||
@@ -115,7 +136,7 @@ say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
|
|||||||
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
|
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
|
||||||
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
|
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
|
||||||
else
|
else
|
||||||
T="$HOME/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad"
|
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
|
||||||
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
|
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
|
||||||
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
|
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
|
||||||
fi
|
fi
|
||||||
|
|||||||
Executable
+136
@@ -0,0 +1,136 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
|
||||||
|
# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
|
||||||
|
#
|
||||||
|
# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
|
||||||
|
# 200 from the *production* process that was still draining its listener, while the freshly
|
||||||
|
# created rig container had not bound anything. Other runs died from a port that answered for
|
||||||
|
# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
|
||||||
|
# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
|
||||||
|
# them were a check that measured something adjacent to the thing it meant.
|
||||||
|
#
|
||||||
|
# Usage: bash scripts/qwen38-preflight.sh [--allow-existing-tap]
|
||||||
|
# Exit: 0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
|
||||||
|
|
||||||
|
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||||
|
TAP="$CAMPAIGN/calib"
|
||||||
|
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
||||||
|
NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
|
||||||
|
ALLOW_EXISTING=0
|
||||||
|
[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
|
||||||
|
|
||||||
|
# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
|
||||||
|
WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
|
||||||
|
IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
|
||||||
|
SERVICE="frank-archon"
|
||||||
|
PORT=7770
|
||||||
|
|
||||||
|
fails=0
|
||||||
|
ok() { printf 'PASS %-14s %s\n' "$1" "$2"; }
|
||||||
|
bad() { printf 'FAIL %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
|
||||||
|
skip() { printf 'WARN %-14s %s\n' "$1" "$2"; }
|
||||||
|
|
||||||
|
# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
|
||||||
|
if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
|
||||||
|
ok cwd "running from the repo root ($REPO_ROOT)"
|
||||||
|
else
|
||||||
|
bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
|
||||||
|
if command -v podman >/dev/null 2>&1; then
|
||||||
|
# An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
|
||||||
|
# asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
|
||||||
|
got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
|
||||||
|
| sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
|
||||||
|
if [ "${got:-0}" -ge 1 ]; then
|
||||||
|
ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
|
||||||
|
elif podman image exists "$IMAGE_REF" 2>/dev/null; then
|
||||||
|
bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
|
||||||
|
else
|
||||||
|
bad image "$IMAGE_REF is not present on this host"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
skip image "podman not on PATH — cannot resolve $IMAGE_REF"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
|
||||||
|
if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
|
||||||
|
avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
|
||||||
|
if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
|
||||||
|
ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
|
||||||
|
else
|
||||||
|
bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
|
||||||
|
fi
|
||||||
|
# 4. tap cleanliness — stale grams from another run must never be mixed in.
|
||||||
|
n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
|
||||||
|
if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
|
||||||
|
ok tap-empty "$n entries in $TAP"
|
||||||
|
else
|
||||||
|
bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
bad space "cannot create $TAP"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
|
||||||
|
# orphaned engine from a previous run makes the next boot die with `startup stopped: device
|
||||||
|
# error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
|
||||||
|
# CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
|
||||||
|
# carries the container ID, never the service name — both are proxies that lie.
|
||||||
|
PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
|
||||||
|
service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
|
||||||
|
if [ -e /dev/kfd ]; then
|
||||||
|
mine="$(service_pids | tr '\n' ' ')"
|
||||||
|
foreign=""
|
||||||
|
for p in $(fuser /dev/kfd 2>/dev/null); do
|
||||||
|
case " $mine " in
|
||||||
|
*" $p "*) : ;; # the service we will stop
|
||||||
|
*) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
if [ -z "$foreign" ]; then
|
||||||
|
ok device "no foreign /dev/kfd holders"
|
||||||
|
else
|
||||||
|
bad device "foreign holders:$foreign — kill by pid, never by pattern"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
skip device "/dev/kfd absent — not an AMD host?"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
|
||||||
|
lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
|
||||||
|
if [ -z "${lpid:-}" ]; then
|
||||||
|
ok port ":$PORT is free"
|
||||||
|
else
|
||||||
|
mine="$(service_pids | tr '\n' ' ')"
|
||||||
|
case " $mine " in
|
||||||
|
*" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
|
||||||
|
*) bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 7. corpus — the push source must exist and be the re-chunked file the window documents.
|
||||||
|
if [ -s "$CORPUS" ]; then
|
||||||
|
ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
|
||||||
|
else
|
||||||
|
bad corpus "$CORPUS is missing or empty"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 8. reference — the KL gate has nothing to compare against without it.
|
||||||
|
if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
|
||||||
|
ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
|
||||||
|
else
|
||||||
|
bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
if [ "$fails" -gt 0 ]; then
|
||||||
|
echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
echo "PREFLIGHT PASSED: every fact checked by identity."
|
||||||
@@ -28,10 +28,18 @@
|
|||||||
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
|
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
|
||||||
# separate driver — scripts/calib-push.py --port 7770
|
# separate driver — scripts/calib-push.py --port 7770
|
||||||
# --container qwen38-bf16 --corpus
|
# --container qwen38-bf16 --corpus
|
||||||
# /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
|
# /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
|
||||||
# 12M pooled positions the engine writes the Grams and the
|
# at the floor the engine writes the Grams and the tap switches
|
||||||
# tap switches itself off (calib-push prints TAP_DONE). Stop
|
# itself off (calib-push prints TAP_DONE). Stop this script
|
||||||
# this script after that; the trap brings frank-archon back.
|
# after that; the trap brings frank-archon back. THE FLOOR IS
|
||||||
|
# THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
|
||||||
|
# served iq4r this arm competes with was calibrated on a 10 M-
|
||||||
|
# token set, so the arm is calibrated at least as deeply. One
|
||||||
|
# round of the re-chunked corpus is ~9.31 M tokens, so sealing
|
||||||
|
# takes round 1 plus part of round 2 — a round that seals
|
||||||
|
# nothing is NORMAL (the floor is pooled), and the pusher's
|
||||||
|
# round_verdict treats a flat round as "continue", never as a
|
||||||
|
# failure. One seal writes ~40 GiB.
|
||||||
#
|
#
|
||||||
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
|
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
|
||||||
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
|
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
|
||||||
@@ -113,6 +121,18 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|||||||
# shellcheck disable=SC1091
|
# shellcheck disable=SC1091
|
||||||
source "$SCRIPT_DIR/frank-archon.env"
|
source "$SCRIPT_DIR/frank-archon.env"
|
||||||
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||||
|
# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
|
||||||
|
# moving target and a tap must name the producer that wrote it. `:latest` on
|
||||||
|
# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
|
||||||
|
# calib boot prints the arming line — "recorded passes: off, the program
|
||||||
|
# accumulates calibration Grams (gram_accum)" — where the production pin
|
||||||
|
# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
|
||||||
|
# into the tap dir while /health, /stats and the push all looked healthy
|
||||||
|
# (measured 2026-10-30). The tap code is identical in both trees' sources, so
|
||||||
|
# the shipped 1.8.1 build is what drops it; do not spend another window on it.
|
||||||
|
# Any other image must be named deliberately with --image.
|
||||||
|
CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
|
||||||
|
IMAGE_EXPLICIT=0
|
||||||
|
|
||||||
MODE=""
|
MODE=""
|
||||||
POS=()
|
POS=()
|
||||||
@@ -122,7 +142,7 @@ while [ $# -gt 0 ]; do
|
|||||||
kldref|calib|score) MODE="$1" ;;
|
kldref|calib|score) MODE="$1" ;;
|
||||||
--dry-run) DRY=1 ;;
|
--dry-run) DRY=1 ;;
|
||||||
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
|
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
|
||||||
IMAGE="$2"; shift ;;
|
IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
|
||||||
*)
|
*)
|
||||||
# score's two positional arguments, and nothing else may be bare.
|
# score's two positional arguments, and nothing else may be bare.
|
||||||
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
|
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
|
||||||
@@ -152,7 +172,7 @@ TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
|
|||||||
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
|
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
|
||||||
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
|
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
|
||||||
KLD_REF_DIR="$CAMPAIGN/kld-ref"
|
KLD_REF_DIR="$CAMPAIGN/kld-ref"
|
||||||
CALIB_DIR="$CAMPAIGN/calib"
|
CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
|
||||||
C=qwen38-bf16
|
C=qwen38-bf16
|
||||||
SERVICE=frank-archon
|
SERVICE=frank-archon
|
||||||
# The card split, taken from the file that owns it instead of a private pair:
|
# The card split, taken from the file that owns it instead of a private pair:
|
||||||
@@ -314,6 +334,21 @@ case "$MODE" in
|
|||||||
# buys throughput with the expert slab, so price it like any other
|
# buys throughput with the expert slab, so price it like any other
|
||||||
# VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
|
# VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
|
||||||
# through MAX_SEQS, exactly as kldref and score do.
|
# through MAX_SEQS, exactly as kldref and score do.
|
||||||
|
[ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
|
||||||
|
# ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
|
||||||
|
# 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
|
||||||
|
# the filesystem to zero, after which every write came back short ("calibration
|
||||||
|
# cannot write", "write was short") and the system itself began refusing
|
||||||
|
# writes. The room is demanded up front, on the tap's own filesystem.
|
||||||
|
if [ "$DRY" != 1 ]; then
|
||||||
|
avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
|
||||||
|
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
|
||||||
|
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
|
||||||
|
if [ "$avail_mib" -lt "$need_mib" ]; then
|
||||||
|
echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
ARGS+=(--model "$MODEL_IN" --port 7770
|
ARGS+=(--model "$MODEL_IN" --port 7770
|
||||||
--max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
|
--max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
|
||||||
# The tap is only real if this directory exists and the engine can write it:
|
# The tap is only real if this directory exists and the engine can write it:
|
||||||
@@ -417,6 +452,19 @@ cp -f "$TEMPLATE" "$TPL" || {
|
|||||||
# where the service reported `active` while its engine starved on the rig's VRAM.
|
# where the service reported `active` while its engine starved on the rig's VRAM.
|
||||||
restore() {
|
restore() {
|
||||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||||
|
# Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
|
||||||
|
# radiance kept /dev/kfd after `podman rm -f`, production's
|
||||||
|
# ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
|
||||||
|
# and Restart=always looped it 100 times with /health never answering. The
|
||||||
|
# service is stopped for the whole run, so any radiance still holding the
|
||||||
|
# device here is an orphan — kill it BY PID (pkill -f matches the caller's
|
||||||
|
# own command line, which is how a session gets killed instead of the orphan).
|
||||||
|
for p in $(fuser /dev/kfd 2>/dev/null); do
|
||||||
|
if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
|
||||||
|
echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
|
||||||
|
kill -9 "$p" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
|
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
|
||||||
}
|
}
|
||||||
trap restore EXIT
|
trap restore EXIT
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
|
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
|
||||||
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
||||||
# digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
|
# digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
|
||||||
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
|
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
||||||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||||||
# vendor build 10777, CUDA sm_86).
|
# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
|
||||||
#
|
#
|
||||||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||||||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||||||
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
# --- defaults (used when the env file does not set them) -------------------
|
# --- defaults (used when the env file does not set them) -------------------
|
||||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
|
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
|
||||||
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
||||||
: "${ALIAS:=frank-zealot}"
|
: "${ALIAS:=frank-zealot}"
|
||||||
: "${BIND_ADDR:=0.0.0.0}"
|
: "${BIND_ADDR:=0.0.0.0}"
|
||||||
@@ -119,7 +119,7 @@ fi
|
|||||||
# above are what pay for it. The draft cache dtype IS a further knob
|
# above are what pay for it. The draft cache dtype IS a further knob
|
||||||
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||||||
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||||||
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
|
# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
|
||||||
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||||||
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||||||
: "${SPEC_TYPE:=draft-mtp}"
|
: "${SPEC_TYPE:=draft-mtp}"
|
||||||
|
|||||||
@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
|
|||||||
dankserver/.env.example|dankclaw/.env.example)
|
dankserver/.env.example|dankclaw/.env.example)
|
||||||
add_test tests/env_consistency.bats
|
add_test tests/env_consistency.bats
|
||||||
;;
|
;;
|
||||||
|
# The SSD-backed metadata view: the installer, the host mount unit, and the
|
||||||
|
# two media units that bind the view rather than the raw array.
|
||||||
|
scripts/media-view.sh|dankserver/host/*)
|
||||||
|
add_test tests/media_overlay.bats
|
||||||
|
add_test tests/dankserver_quadlet.bats
|
||||||
|
add_test tests/deployment_scoping.bats
|
||||||
|
;;
|
||||||
|
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
|
||||||
|
add_test tests/media_overlay.bats
|
||||||
|
;;
|
||||||
shared/homepage/*)
|
shared/homepage/*)
|
||||||
add_test tests/homepage_caddy.bats
|
add_test tests/homepage_caddy.bats
|
||||||
add_test tests/homepage_config.bats
|
add_test tests/homepage_config.bats
|
||||||
|
|||||||
@@ -48,6 +48,14 @@ Use the `Authorization: token` header for everything.
|
|||||||
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
|
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
|
||||||
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
|
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
|
||||||
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
|
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
|
||||||
|
- **Comment on a PR**: the conversation thread is `POST $API/issues/<PR-number>/comments`
|
||||||
|
(201, and the answer's `html_url` points at `pulls/<n>`; Gitea's issue index is
|
||||||
|
unified, so the PR's number is the same number). `$API/pulls/<n>/comments` is the
|
||||||
|
**review**-comment store and answers `405` to a plain body.
|
||||||
|
- **A body built by `jq --rawfile b /dev/stdin` inside `$(…)` is empty**: the command
|
||||||
|
substitution does not inherit the heredoc, so Gitea receives `{}` and answers 422 —
|
||||||
|
and `jq -r '.id'` on the error prints nothing, which reads as success. Write the
|
||||||
|
body to a file and pass `--rawfile` that file.
|
||||||
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
|
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
|
||||||
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
|
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
|
||||||
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
|
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
|
||||||
|
|||||||
@@ -195,8 +195,9 @@ deploy.sh, never a live env poke.
|
|||||||
check every namespace (`sudo podman ps -a`) and native units
|
check every namespace (`sudo podman ps -a`) and native units
|
||||||
(`systemctl list-units`) before declaring a service absent. Compare
|
(`systemctl list-units`) before declaring a service absent. Compare
|
||||||
compose mount sources against the real filesystem — compose files name
|
compose mount sources against the real filesystem — compose files name
|
||||||
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
|
phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
|
||||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
|
||||||
|
compose through git first, deploy second.
|
||||||
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
||||||
or creation fails `statfs …: no such file or directory`.
|
or creation fails `statfs …: no such file or directory`.
|
||||||
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
---
|
---
|
||||||
type: reference
|
type: reference
|
||||||
title: Jellyfin
|
title: Jellyfin
|
||||||
description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
|
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
|
||||||
tags: [jellyfin, dankserver, media-server]
|
tags: [jellyfin, dankserver, media-server]
|
||||||
timestamp: 2026-09-03T00:00:00Z
|
timestamp: 2026-10-30T00:00:00Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# Jellyfin
|
# Jellyfin
|
||||||
@@ -18,7 +18,7 @@ clients.
|
|||||||
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
||||||
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
|
||||||
|
|
||||||
## Facts (verified from the live unit, 2026-09-03)
|
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
|
||||||
|
|
||||||
| Item | Value |
|
| Item | Value |
|
||||||
|------|-------|
|
|------|-------|
|
||||||
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
|
||||||
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
|
||||||
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
|
||||||
| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it |
|
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
|
||||||
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
|
||||||
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
|
||||||
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
| Health | `curl -f http://localhost:8096/health` every 30 s |
|
||||||
@@ -36,11 +36,27 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
|
|||||||
```
|
```
|
||||||
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
|
||||||
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
|
||||||
/mnt/entropy -> /data (read-only media volume)
|
/var/lib/media-meta/view -> /data (git: the overlay view)
|
||||||
|
/mnt/entropy -> /data (live today: the array, read-only)
|
||||||
```
|
```
|
||||||
|
|
||||||
Library folders point at `/data` paths; because the mount uses the same
|
Library folders point at `/data` paths: stable across container recreation and
|
||||||
absolute path as on the host, library paths survive container recreation.
|
identical to the raw array layout below. Which side of that path the container
|
||||||
|
is on is the open question on this host — the installed unit binds the read-only
|
||||||
|
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
|
||||||
|
|
||||||
|
`Local metadata` — save metadata to media folders — is enabled, and it is the
|
||||||
|
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
|
||||||
|
written once on the NVMe upper layer are files [[Plex]] can read too. While
|
||||||
|
`/data` is bound read-only every item update logs the error below and no sidecar
|
||||||
|
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
|
||||||
|
that raises the view.
|
||||||
|
|
||||||
|
**Failure signature:** `System.IO.IOException: Read-only file system :
|
||||||
|
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
|
||||||
|
instead of the view. On Dankserver that is today's expected state — the view has
|
||||||
|
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
|
||||||
|
same line is a fault: the quadlet drifted or the mount died.
|
||||||
|
|
||||||
## Ports & access
|
## Ports & access
|
||||||
|
|
||||||
@@ -105,12 +121,109 @@ where the LDAP plugin runs. Verified 2026-09-25: `alex` authenticates over
|
|||||||
`AuthenticateByName` (HTTP 200; full plugin flow logged — service bind,
|
`AuthenticateByName` (HTTP 200; full plugin flow logged — service bind,
|
||||||
search, `ldapUid`, user bind, token).
|
search, `ldapUid`, user bind, token).
|
||||||
|
|
||||||
|
## Metadata providers and their keys (2026-10-30)
|
||||||
|
|
||||||
|
TMDb, OMDb, StudioImages and MusicBrainz are **core plugins** in Jellyfin 12 —
|
||||||
|
no DLL, just a configuration file the running container owns under the data
|
||||||
|
root. TVDB is not core (its metadata provider was removed in 10.9), so it is an
|
||||||
|
official-catalog plugin.
|
||||||
|
|
||||||
|
| Provider | Shape on disk | Key lives in |
|
||||||
|
|---|---|---|
|
||||||
|
| TMDb | `config/plugins/configurations/Jellyfin.Plugin.Tmdb.xml` → `<TmdbApiKey>` | `/home/dank/Containers/jellyfin/provider-keys` → `TMDB_API_KEY=` (mode 600) |
|
||||||
|
| TVDB | `config/plugins/thetvdb_24.0.0.0/` + `…Jellyfin.Plugin.Tvdb.xml` → `<ApiKey>` | the same file → `TVDB_API_KEY=` |
|
||||||
|
| OMDb | `…Jellyfin.Plugin.Omdb.xml` | **not configured** — its key is issued by email, 1 000 req/day, and its ToS forbids storing the returned data; ratings stay Jellyfin's own lookup |
|
||||||
|
|
||||||
|
**None of it is tracked in git.** Those XML files belong to the container and
|
||||||
|
hold credentials, and `tests/jellyfin_providers.bats` fails the suite if one is
|
||||||
|
ever committed. The keys themselves sit in the per-service host env file that is
|
||||||
|
the fleet's convention for container credentials (grafana, caddy, pihole,
|
||||||
|
copyparty, romm, seerr): `provider-keys`, mode 600, beside the data root.
|
||||||
|
`scripts/jellyfin-providers.sh` is the only thing that writes the XML:
|
||||||
|
|
||||||
|
```
|
||||||
|
scripts/jellyfin-providers.sh plan # what would change; prints no key
|
||||||
|
scripts/jellyfin-providers.sh install # plugin + keys, previous copy to trash
|
||||||
|
scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
|
||||||
|
```
|
||||||
|
|
||||||
|
- **A key never enters git, argv, or the tool's output.** The installer reads
|
||||||
|
that 600-perm file itself (a value in argv is world-readable through
|
||||||
|
`/proc/<pid>/cmdline`) and refuses it when it is group- or world-readable —
|
||||||
|
an exposed key is an error, not a warning. A key that shows up in a transcript
|
||||||
|
or `ps` output has leaked: rotate it with the provider, rewrite
|
||||||
|
`provider-keys`, re-run `install`.
|
||||||
|
- **The TVDB artifact is pinned three ways**: sha256
|
||||||
|
`6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319`, the
|
||||||
|
official plugin guid `a677c0da-fac5-4cde-941a-7134223f14c8`, and version
|
||||||
|
`24.0.0.0` (`targetAbi 12.0.0.0`, which fits the deployed 12.4.0). A sha
|
||||||
|
match alone only proves the bytes are the bytes fetched — the guid check is
|
||||||
|
what says it is the official plugin. Rollback: move `thetvdb_24.0.0.0/` into
|
||||||
|
the trash and revert the pin; a second `thetvdb_*` directory is a refusal,
|
||||||
|
never a quiet second copy.
|
||||||
|
- **The homelab TVDB key is a licensed key** (its JWT claims say
|
||||||
|
`community_supported: false`), so it needs **no PIN** — the optional
|
||||||
|
`TVDB_PIN=` line exists only for a subscriber-shaped key.
|
||||||
|
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
|
||||||
|
identifies everything. Enabling TVDB as a provider would re-fetch show
|
||||||
|
metadata across the library, so it is a deliberate, separate change.
|
||||||
|
- **The deployer places the keys before it restarts jellyfin**
|
||||||
|
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
|
||||||
|
failing when `provider-keys` is absent, so a host without it still deploys.
|
||||||
|
- **Keys and plugins are read at start-up**: after a manual `install` the
|
||||||
|
change is inert until `./scripts/deploy.sh dankserver --service jellyfin`.
|
||||||
|
- TMDb's terms are the reason this stays narrow: attribution required, no
|
||||||
|
caching beyond 6 months, no derivative works. Fine for a personal homelab,
|
||||||
|
and the reason no bulk enrichment pass runs against it.
|
||||||
|
|
||||||
## Configuration workflow
|
## Configuration workflow
|
||||||
|
|
||||||
1. Create libraries pointing at `/data` folders (movies, shows, audio).
|
1. Create libraries pointing at `/data` folders (movies, shows, audio).
|
||||||
2. Set metadata/language preferences (`LANG=en_US.UTF-8` pinned in the unit).
|
2. Set metadata/language preferences (`LANG=en_US.UTF-8` pinned in the unit).
|
||||||
3. Configure clients to use `http://192.168.0.235:8096` or the guarded vhost.
|
3. Configure clients to use `http://192.168.0.235:8096` or the guarded vhost.
|
||||||
|
|
||||||
|
## Triggering a scan
|
||||||
|
|
||||||
|
The scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same value is
|
||||||
|
`JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a mode-600 curl
|
||||||
|
config, never a header argument, in the `MediaBrowser` form: 12.4 authenticates
|
||||||
|
only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token`
|
||||||
|
for every key, valid or not. Before believing a 401, prove the key against the
|
||||||
|
server's own table — `select count(*) from ApiKeys where AccessToken = ?` on
|
||||||
|
`jellyfin.db` opened `mode=ro&immutable=1`. The backslashes below matter:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umask 077
|
||||||
|
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
|
||||||
|
"$(cat ~/.config/secrets/jellyfin)"
|
||||||
|
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
|
||||||
|
echo 'request = "POST"'; } >/tmp/jf.hdr
|
||||||
|
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
|
||||||
|
shred -u /tmp/jf.hdr
|
||||||
|
```
|
||||||
|
|
||||||
|
`204` is acceptance; anything else queued no scan, so re-issue the identical
|
||||||
|
request instead of diagnosing. A `401` from a key that answers `200` on a
|
||||||
|
hand-typed request is this block's quoting, not the server: bash `printf`
|
||||||
|
collapses `\"` to `"`, curl ends the quoted value at that quote, and Jellyfin
|
||||||
|
receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answers
|
||||||
|
204, `\"` answers 401 — both run on this host).
|
||||||
|
|
||||||
|
One accepted refresh is the whole procedure. A second one while a scan runs moves
|
||||||
|
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
|
||||||
|
later `204` is accepted and then discarded — the status code stops meaning
|
||||||
|
anything. The fix is `systemctl --user restart jellyfin.service` (never
|
||||||
|
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
|
||||||
|
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
|
||||||
|
response code.
|
||||||
|
|
||||||
|
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
|
||||||
|
is the incremental pass, and inotify does not cross a rootless podman bind, so
|
||||||
|
folders created on the host stay invisible to it. The move is
|
||||||
|
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
|
||||||
|
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
|
||||||
|
record's `RefreshStatus` is the progress read.
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
- [[Plex]] — co-resident media server on the same volume
|
- [[Plex]] — co-resident media server on the same volume
|
||||||
|
|||||||
@@ -0,0 +1,130 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Media Metadata View
|
||||||
|
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
|
||||||
|
tags: [dankserver, media-server, jellyfin, plex, storage]
|
||||||
|
timestamp: 2026-10-30T00:00:00Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Media Metadata View
|
||||||
|
|
||||||
|
`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
|
||||||
|
array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
|
||||||
|
Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
|
||||||
|
[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
|
||||||
|
file, and the array is read-only and slow — so the media bytes stay there and the
|
||||||
|
*writes* land on SSD. One tree to both servers, one copy of the media.
|
||||||
|
|
||||||
|
The mount point is not a volume and holds nothing while unmounted. It sits inside
|
||||||
|
`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
|
||||||
|
merged door are one directory — and so the mount table grows no new `/mnt` entry
|
||||||
|
next to the array.
|
||||||
|
|
||||||
|
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
|
||||||
|
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
|
||||||
|
`/var/lib/media-meta` does not exist, and the installed quadlets under
|
||||||
|
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
|
||||||
|
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
|
||||||
|
is the shape git describes, not the shape the servers run. The cutover is an
|
||||||
|
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
|
||||||
|
`ensure_media_view` (`install` then `verify`) before it restarts the container,
|
||||||
|
so it must not be fired through the middle of a library scan. Until that deploy
|
||||||
|
lands there is no upper layer, so no relayout can leave ghost metadata and
|
||||||
|
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
|
||||||
|
view really is a mountpoint.
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
|
||||||
|
| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
|
||||||
|
| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
|
||||||
|
| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
|
||||||
|
| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
|
||||||
|
| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
|
||||||
|
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
|
||||||
|
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
|
||||||
|
|
||||||
|
Both consumers guard the view with
|
||||||
|
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
|
||||||
|
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
|
||||||
|
install` + `verify` before it restarts either container.
|
||||||
|
|
||||||
|
| Consumer | Bind | Mode | Why that path |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
|
||||||
|
| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
|
||||||
|
|
||||||
|
Putting the sidecars inside Plex's reach is not the same as making Plex use them:
|
||||||
|
a library parses `.nfo` only with its *Local media information* agent enabled, and
|
||||||
|
this change touches no library setting.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-view.sh plan # print the resolved shape, touch nothing
|
||||||
|
sudo scripts/media-view.sh install # create dirs, install the unit, mount, enable
|
||||||
|
scripts/media-view.sh verify # is the view really the overlay? (deploy gate)
|
||||||
|
scripts/media-view.sh status # mount options + upper-layer size/file count
|
||||||
|
scripts/media-view.sh reap # list ghost metadata (media moved, sidecars left)
|
||||||
|
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
|
||||||
|
```
|
||||||
|
|
||||||
|
`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
|
||||||
|
the defaults are the host values above.
|
||||||
|
|
||||||
|
## Laws
|
||||||
|
|
||||||
|
- **The upper layer must sit on non-rotational storage.** `install` and `verify`
|
||||||
|
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
|
||||||
|
whose upper is the spinners is the original problem with extra steps, and it
|
||||||
|
silently turns every metadata write back into a seek.
|
||||||
|
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
|
||||||
|
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
|
||||||
|
presents an *empty library* as truth, and a media scan would then clear the
|
||||||
|
database of everything. systemd does not create a mount point, so `install`
|
||||||
|
creates `view` itself; without it the mount simply fails.
|
||||||
|
- **Never delete through the view.** A delete writes an overlayfs whiteout
|
||||||
|
(`.wh.<name>`) in the upper layer, which hides the array's copy without
|
||||||
|
removing it — the file reappears the moment the view is unmounted, and the
|
||||||
|
upper layer now carries a landmine. Deletions belong on the raw path
|
||||||
|
(`/mnt/entropy`), which the relayout tool already treats as the only writer.
|
||||||
|
`reap` refuses to remove whiteouts and only warns about them.
|
||||||
|
- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
|
||||||
|
sidecars the view already wrote stay at the old coordinates: a ghost folder
|
||||||
|
the servers browse and find nothing in. `reap` lists upper-layer directories
|
||||||
|
with no counterpart below and moves them (with their subtree) into a dated
|
||||||
|
`.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
|
||||||
|
that exists only above is normal, so only *directories* are candidates.
|
||||||
|
- **File-watcher events do not cross the overlay.** Plex's NFO agent and
|
||||||
|
Jellyfin's real-time monitor see inotify events for writes that land *in the
|
||||||
|
upper layer*, not for changes on the array underneath. After any relayout or
|
||||||
|
externally-written metadata, trigger a library refresh — see
|
||||||
|
[[Media Relayout]] for the refresh procedure.
|
||||||
|
|
||||||
|
## Failure signatures
|
||||||
|
|
||||||
|
| Symptom | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
|
||||||
|
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
|
||||||
|
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
|
||||||
|
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
|
||||||
|
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Revert the commit that pointed the two quadlets at the view and re-deploy both
|
||||||
|
services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
|
||||||
|
The NVMe upper layer can stay: nothing reads it once the binds are back, and it
|
||||||
|
holds no state either server needs. Unmounting alone
|
||||||
|
(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
|
||||||
|
binds it, and the array is never modified by any of this.
|
||||||
|
|
||||||
|
## Links
|
||||||
|
|
||||||
|
- [[Jellyfin]] — the metadata writer; its data layout section.
|
||||||
|
- [[Plex]] — the read-only consumer; in-place bind-mount history.
|
||||||
|
- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
|
||||||
|
- [[Entropy Volume]] — the array underneath.
|
||||||
@@ -0,0 +1,497 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Media Relayout
|
||||||
|
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
|
||||||
|
tags: [dankserver, media-server, jellyfin, plex, procedure]
|
||||||
|
timestamp: 2026-10-30T00:00:00Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Media Relayout
|
||||||
|
|
||||||
|
The movie library on [[Entropy Volume]] arrived as release folders —
|
||||||
|
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
|
||||||
|
the release. [[Jellyfin]] matches a movie by folder and file name, so the
|
||||||
|
release noise cost posters, extras and versions. These scripts turn that
|
||||||
|
library into the shape its media server reads — the movies, the TV packs, the TV
|
||||||
|
that turns up in the movie share, the music library, and the audiobooks, whose
|
||||||
|
only reader is Plex — and nothing else in the repo touches a media file.
|
||||||
|
|
||||||
|
| Script | Touches media? | Output |
|
||||||
|
|---|---|---|
|
||||||
|
| `scripts/media-census.py` | reads headers and the servers' own DBs, never writes | `enum`, `probed`, and one authority TSV per library |
|
||||||
|
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
|
||||||
|
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
|
||||||
|
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
|
||||||
|
| `scripts/media-shows.py` | no — TSV out | a plan that re-files the TV library itself |
|
||||||
|
| `scripts/media-music.py` | no — TSV out | a plan that re-files the music library by its tags |
|
||||||
|
| `scripts/media-audiobooks.py` | no — TSV out | a plan that re-files the audiobooks by the book Plex matched |
|
||||||
|
| `scripts/media-verify.py` | no — reads the servers' DBs read-only | the verdict that every journaled move landed |
|
||||||
|
|
||||||
|
The seams are `tests/media_relayout.bats`, `tests/media_shows.bats`,
|
||||||
|
`tests/media_music.bats`, `tests/media_audiobooks.bats` and `tests/media_verify.bats`;
|
||||||
|
the movie and TV target contracts are quoted from
|
||||||
|
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
|
||||||
|
in `media-plan.py`'s docstring.
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
Run the census on the machine that mounts the library. `probe` is the slow step
|
||||||
|
(a header read per container) and is `nice`d — it is safe beside a live service
|
||||||
|
because it never competes for the GPU and never writes inside the library.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
C=scripts/media-census.py
|
||||||
|
$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
|
||||||
|
$C probe --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
|
||||||
|
$C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
|
||||||
|
```
|
||||||
|
|
||||||
|
`plex` reads the Plex metadata cache (`--db` defaults to the path the
|
||||||
|
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
|
||||||
|
`mode=ro&immutable=1`: a live server is never locked or written.
|
||||||
|
|
||||||
|
Then plan — this reads only the three TSVs, so it costs seconds and can run
|
||||||
|
anywhere, including a laptop:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-plan.py --root /mnt/entropy/Movies \
|
||||||
|
--enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
|
||||||
|
--plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
|
||||||
|
is a file the plan could not name a film for, a pile that needs a README, or a
|
||||||
|
collision it refused. The printed gate lines are the go/no-go:
|
||||||
|
`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
|
||||||
|
0 and `unplanned=0` must appear in the report. Any non-zero number means a file
|
||||||
|
that would be filed wrong, and the plan is not run until it is understood.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
|
||||||
|
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
|
||||||
|
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
|
||||||
|
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
|
||||||
|
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||||
|
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
An apply also leaves the release folders it emptied standing where they were —
|
||||||
|
hundreds of them at the library root — which is what `sweep` removes, dry-run
|
||||||
|
first, from the same journal.
|
||||||
|
|
||||||
|
After it lands, both servers need a scan or the change is invisible. [[Plex]] and
|
||||||
|
[[Jellyfin]] own the exact requests; two laws hold at this step: `204` is
|
||||||
|
acceptance and anything else queued nothing — re-issue the identical request
|
||||||
|
rather than diagnosing — and ONE accepted refresh is the whole procedure, because
|
||||||
|
a second while a scan runs moves `Scan Media Library` to `Cancelling` and every
|
||||||
|
later `204` is accepted and then discarded.
|
||||||
|
|
||||||
|
### Proving the move landed
|
||||||
|
|
||||||
|
A scan answers `204` whether or not it finds anything, so proof is the journal
|
||||||
|
replayed against both servers' own tables: every `move` row has a row at its
|
||||||
|
`dst`, and **zero** rows still name its `src`. `scripts/media-verify.py` is that
|
||||||
|
replay: it opens each database read-only (there is no `sqlite3` binary on these
|
||||||
|
hosts or inside either container) and exits non-zero while a row is stale.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
|
||||||
|
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
|
||||||
|
```
|
||||||
|
|
||||||
|
Ignore only what a server cannot index. A missing `.flac` or `.mkv` is the
|
||||||
|
fault; a row still naming its `src` means the scan has not followed the rename
|
||||||
|
yet, so re-run the check after the scan, never before.
|
||||||
|
|
||||||
|
Three doors that look right and are not: `GET /search?query=` matches metadata
|
||||||
|
titles, never file paths; `?path=` on `/library/sections/N/all` makes Plex's
|
||||||
|
HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
||||||
|
no `Accept` and client-identifier header (see the scan block above for the form
|
||||||
|
that works).
|
||||||
|
|
||||||
|
## TV in the movie library
|
||||||
|
|
||||||
|
A box set that arrives in the movie share stays there until someone moves it:
|
||||||
|
`media-plan.py` refuses to file TV and parks each series in
|
||||||
|
`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
|
||||||
|
walks the movie library itself (no census needed), groups episode-shaped names
|
||||||
|
by series, and writes a plan in the executor's own format:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
|
||||||
|
--out /mnt/entropy/media-tv-plan.tsv
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
|
||||||
|
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
|
||||||
|
--root /mnt/entropy --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
|
||||||
|
Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
|
||||||
|
because the publisher's spelling beats a re-derivation, with the year taken
|
||||||
|
from the box-set folder. What it refuses it prints and leaves alone:
|
||||||
|
|
||||||
|
- fewer than `--min` (5) files under one series name: one file named `Part 2`
|
||||||
|
is a film, not a series;
|
||||||
|
- an episode whose name states no season number — no invented `Season 01`;
|
||||||
|
- a destination two files both claim: the executor refuses the whole plan
|
||||||
|
before its first rename, and only Frank decides which copy is the keeper.
|
||||||
|
- an episode the destination library already holds at the same byte count. The
|
||||||
|
movie share carried a second copy of the whole Monk box set — 124 episodes
|
||||||
|
both sides, the same episode list, every byte count identical, 31.5 GiB each
|
||||||
|
way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
|
||||||
|
such a series in place and says so; a duplicate is deleted or kept by Frank,
|
||||||
|
never copied into one library by a tool.
|
||||||
|
|
||||||
|
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
|
||||||
|
|
||||||
|
Below the threshold the mover prints and leaves, so a lone downloaded episode of
|
||||||
|
a real show stays in the movie library on purpose — six did on 2026-10-30.
|
||||||
|
Filing one by hand is the same executor on a hand-written plan and the same
|
||||||
|
journal; the trap is the series year. Read it from the file's own name or the
|
||||||
|
folder it owns, never from the title whose bonus folder it happens to be parked
|
||||||
|
in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
|
||||||
|
show. A name whose marker is `Part 2` is a film and stays put.
|
||||||
|
|
||||||
|
## The TV library's own relayout
|
||||||
|
|
||||||
|
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
|
||||||
|
already matched and numbered most of it. `scripts/media-shows.py` re-files that
|
||||||
|
library into the shape Jellyfin reads, with **tvdb as the name authority**:
|
||||||
|
|
||||||
|
<Series> (<year>)/Season NN/<Series> (<year>) - SxxEyy - <Episode Title>.ext
|
||||||
|
|
||||||
|
It never guesses: every series name, episode number and title it writes traces to
|
||||||
|
an authority, and what has none is parked for a human to read.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
|
||||||
|
# and Tvdb ids and the title the row holds (the DB opens read-only).
|
||||||
|
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
|
||||||
|
--out /tmp/tv_titles.tsv
|
||||||
|
# 2. What is on disk (stat only), and the runtimes the film arm needs.
|
||||||
|
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
|
||||||
|
--out /tmp/tv_enum.tsv
|
||||||
|
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
|
||||||
|
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
|
||||||
|
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
|
||||||
|
# the last one — a duration belongs to the file, not to its name: re-key the
|
||||||
|
# previous census through that pass's own journal (`src` -> `dst`) instead of
|
||||||
|
# seeking the array again. It is also the only column the film arm reads, and
|
||||||
|
# only for a file with no marker and no provider row (4,794 of 58,551 today,
|
||||||
|
# all of them covered by a re-key that costs seconds, not an hour).
|
||||||
|
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
|
||||||
|
# which travels as the `api_key` query parameter. Only a v4 request token is
|
||||||
|
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
|
||||||
|
# which the summary line flags as NOT provider-complete. --tvdb reads its
|
||||||
|
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
|
||||||
|
# cache file, so the plan is reproducible offline afterwards: run it once to
|
||||||
|
# fill the cache, and quote the second run.
|
||||||
|
# Paths are only strings to a dry run, so this step can run on the workstation
|
||||||
|
# against scp'd TSVs — which is where the two provider secrets live.
|
||||||
|
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
|
||||||
|
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
|
||||||
|
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
|
||||||
|
--park /mnt/entropy/media-park/TVShows \
|
||||||
|
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||||
|
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
|
||||||
|
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
|
||||||
|
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
|
||||||
|
# applied pass is its unwind record, so a second pass that reuses the name
|
||||||
|
# destroys the only map back.
|
||||||
|
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
|
||||||
|
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Evidence, strongest first — the order the planner believes things:
|
||||||
|
|
||||||
|
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`);
|
||||||
|
- the **`Season NN` folder it sits in and the number its name leads with inside
|
||||||
|
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
|
||||||
|
- the **provider row**: which episode the file IS, and the numbering of a file
|
||||||
|
whose own name says nothing;
|
||||||
|
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
|
||||||
|
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
|
||||||
|
episode title, because a name the server read off a folder is not a show name;
|
||||||
|
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
|
||||||
|
series' canonical extra folder;
|
||||||
|
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
|
||||||
|
when the movie provider's title is a run of words the FILE itself states and its
|
||||||
|
year agrees with the file's: the query is only what `title_of` salvaged from a
|
||||||
|
release name, and the first hit for a short one is a near miss — a 1963 Bond film
|
||||||
|
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
|
||||||
|
parks.
|
||||||
|
|
||||||
|
Two refusals carry the design, and a dry run on the real library found both:
|
||||||
|
|
||||||
|
- **Folders resolve a series; they never invent one.** A folder's words may
|
||||||
|
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
|
||||||
|
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
|
||||||
|
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
|
||||||
|
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
|
||||||
|
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
|
||||||
|
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
|
||||||
|
season 16 episode 20, and one Gundam row says season 2004. The folder, then
|
||||||
|
the file's own leading number, win; a row that still insists on season 1989
|
||||||
|
parks its file instead of creating `Season 1989`.
|
||||||
|
|
||||||
|
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
|
||||||
|
after the folder it scanned, so a library filed by release folders ends up with
|
||||||
|
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
|
||||||
|
copying those names verbatim launders folder junk into the contract. The planner
|
||||||
|
resolves each series to its tvdb id and writes tvdb's name and year; the merged
|
||||||
|
identity is also what turns a box set split across five release folders into ONE
|
||||||
|
folder, and what makes two files claiming one episode MEET in the duplicate rule.
|
||||||
|
A series no id resolves keeps the provider's words and is listed under *no tvdb
|
||||||
|
identity* in the exceptions — that list, not a rename, is the review.
|
||||||
|
|
||||||
|
**The review pile lives outside the library.** Files parked inside
|
||||||
|
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
|
||||||
|
show's rows then looked like evidence. So a row whose series is the hidden
|
||||||
|
top-level folder the file itself sits in is dropped — and the folder is matched
|
||||||
|
with its leading underscore stripped, because the server *displays* `_unsorted` as
|
||||||
|
`unsorted`: compare the spelled names and the pile survives the relayout as a
|
||||||
|
visible series folder that real files get renamed into. `--park` outside the root
|
||||||
|
makes the next plan re-file the pile instead of skipping it. Inside a hidden
|
||||||
|
top-level folder a file's own words may MATCH a registered show, never invent one.
|
||||||
|
|
||||||
|
## The music library's own relayout
|
||||||
|
|
||||||
|
`/mnt/entropy/Music` is rips and discography packs, and Jellyfin has matched and
|
||||||
|
tagged almost all of it. `scripts/media-music.py` re-files it into the shape
|
||||||
|
Jellyfin's music scanner reads —
|
||||||
|
|
||||||
|
Music/<album artist>/<Album> (<year>)/NN - <Title>.ext
|
||||||
|
|
||||||
|
— and the authority is the tag row of the track, never the name the ripper gave
|
||||||
|
the file.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The authority: Jellyfin's audio rows, one per track, with the album and
|
||||||
|
# artist ids its own database carries (the DB opens read-only).
|
||||||
|
python3 scripts/media-census.py music --root /mnt/entropy/Music \
|
||||||
|
--out /tmp/music_rows.tsv
|
||||||
|
# 2. What is on disk (stat only).
|
||||||
|
python3 scripts/media-census.py enumerate --root /mnt/entropy/Music \
|
||||||
|
--out /tmp/music_enum.tsv
|
||||||
|
# 3. The plan.
|
||||||
|
python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.tsv \
|
||||||
|
--rows /tmp/music_rows.tsv --out /tmp/music_plan.tsv \
|
||||||
|
--exceptions /tmp/music_exceptions.txt
|
||||||
|
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
|
||||||
|
# executor, journal and rollback as above, on its own journal.
|
||||||
|
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/media-relayout-music.jsonl --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
The group is what the tags assert, not where the files sit: **(album-artist tag,
|
||||||
|
album, year)**. One album arrives as three disc folders and as two copies of one
|
||||||
|
rip, and one MusicBrainz release group is one album. A compilation is one album
|
||||||
|
whose tracks name many performers, so it files once under `Various Artists`:
|
||||||
|
keying on the performer would file one album once per artist. Two different
|
||||||
|
album-artist tags on one album name are two albums, which is what Jellyfin's two
|
||||||
|
album items say.
|
||||||
|
|
||||||
|
What it refuses:
|
||||||
|
|
||||||
|
+- **A sidecar is placed by the audio around it, never by its own name.** A
|
||||||
|
`.lrc`, `.txt` or picture that shares a track's stem follows that track and
|
||||||
|
takes the name the track now has, or Jellyfin cannot find it; anything else
|
||||||
|
goes to the album its own folder was placed in, or to the folder whose discs
|
||||||
|
sit one level below it. A folder whose audio belongs to two albums parks its
|
||||||
|
sidecar rather than guessing.
|
||||||
|
+- **An image album keeps its name.** A folder holding one audio file and a
|
||||||
|
`.cue` is a whole album in one file; numbering it `01 - …` tells Jellyfin to
|
||||||
|
index it as one track.
|
||||||
|
+- **A video container parks**, because Jellyfin will not index it as music and a
|
||||||
|
rename to `.m4a` would claim it is; **audio no provider row names parks** too,
|
||||||
|
because unmatched audio is a tagging job, not a renaming job.
|
||||||
|
|
||||||
|
ONE writer at a time on this array: two rename passes race, and neither journal
|
||||||
|
tells — which is why each arm above carries its own journal name.
|
||||||
|
|
||||||
|
## The audiobook library's own relayout
|
||||||
|
|
||||||
|
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
|
||||||
|
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
|
||||||
|
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
|
||||||
|
is the NARRATOR, both matched out of the files' own tags. Today one book is split
|
||||||
|
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
|
||||||
|
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
|
||||||
|
re-files the tree so the property both servers' book scanners assume is true —
|
||||||
|
|
||||||
|
Audiobooks/<Narrator>/<Book> (<year>)/<the file keeps its own name>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The authority: Plex's chapter rows for this root (the DB opens read-only).
|
||||||
|
python3 scripts/media-census.py audiobooks --root /mnt/entropy/Audiobooks \
|
||||||
|
--out /tmp/ab_rows.tsv
|
||||||
|
# 2. What is on disk (stat only).
|
||||||
|
python3 scripts/media-census.py enumerate --root /mnt/entropy/Audiobooks \
|
||||||
|
--out /tmp/ab_enum.tsv
|
||||||
|
# 3. The plan.
|
||||||
|
python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
|
||||||
|
--enum /tmp/ab_enum.tsv --rows /tmp/ab_rows.tsv --out /tmp/ab_plan.tsv \
|
||||||
|
--exceptions /tmp/ab_exceptions.txt
|
||||||
|
# 4. Read the exceptions file FIRST. Then the same executor, journal and rollback
|
||||||
|
# as above, on its own journal, and refresh the Plex section as in step 5 of
|
||||||
|
# the movie Procedure — from inside the container, never over the host port.
|
||||||
|
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
|
||||||
|
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
The group is the identity Plex asserts in its own database — **(narrator, book,
|
||||||
|
year)** — so one book arriving in four part folders becomes one folder, and a
|
||||||
|
1965 and a 2020 recording that share a title stay two books.
|
||||||
|
|
||||||
|
Two laws are forced by what the rows actually hold:
|
||||||
|
|
||||||
|
+- **A chapter is never renumbered.** Most books contain chapters whose index Plex
|
||||||
|
does not know, so a number written into a name would be invented — and it would
|
||||||
|
buy nothing, because both servers order playback by the embedded track tag, not
|
||||||
|
by the file name. A wrong number in a name is the number a human later trusts.
|
||||||
|
Fix order in the tags: a Plex scan re-reads tags and never renames a file.
|
||||||
|
+- **The shelf is the narrator, not the author.** Every book has exactly one artist
|
||||||
|
row and it is the narrator; the author survives only inside some album titles
|
||||||
|
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving it would
|
||||||
|
mean parsing titles where a ripper wrote one and inventing it everywhere else.
|
||||||
|
The author stays visible in the folder name Plex itself shows.
|
||||||
|
|
||||||
|
What it refuses, the music arm's list with one addition: a video container, audio
|
||||||
|
no chapter row names, and a loose file in a folder holding two books all park under
|
||||||
|
`_unsorted` with the path they have today — and so does a chapter whose row names
|
||||||
|
**no book or no narrator**: a placeholder (`[Unknown Album]`, `[Unknown Artist]`)
|
||||||
|
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
|
||||||
|
`Various Artists` is not a placeholder — that is a multi-cast read.
|
||||||
|
|
||||||
|
This arm applies behind the music writer — one writer at a time — on its own
|
||||||
|
journal, and then refreshes the Plex section.
|
||||||
|
|
||||||
|
## Laws
|
||||||
|
|
||||||
|
- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
|
||||||
|
database keeps the release-folder path, so playback fails `Couldn't find the
|
||||||
|
file to stream` while the scanner works through the section. Never read such a
|
||||||
|
404 as a lost file: the file is where the journal says it is.
|
||||||
|
- **Plex's matched title outranks the filename**, and a Plex year of `0` means
|
||||||
|
*indexed but never matched* — trusting it named folders `Title (0)`, which is
|
||||||
|
a Jellyfin item with no year, no poster and no identity. A year that is not a
|
||||||
|
plausible year is no information.
|
||||||
|
- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
|
||||||
|
`V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
|
||||||
|
levels deep that neither matcher reads as a movie.
|
||||||
|
- **A folder inside a movie's folder is bonus territory even when its name is
|
||||||
|
misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
|
||||||
|
`Extras`; the vocabulary rule never fired, the featurette became a movie
|
||||||
|
folder, and its words then out-scored the real film and drew that box set's
|
||||||
|
other extras into it. The rule that replaced it needs no metadata: the folder
|
||||||
|
holding the longest film owns the shorter video filed under it, unless that
|
||||||
|
file's own name carries a different year — which is another film speaking.
|
||||||
|
- **Matching words are split on everything that is not a letter or a digit.**
|
||||||
|
Splitting on separators but not on `/` left tokens like `(1960)/extras` that
|
||||||
|
no other name could ever agree with, so a film could not be found by the
|
||||||
|
folder holding it, and extras drifted into a neighbour. A bare number counts
|
||||||
|
only when it could be a release year, so `1080` and `2160` never pose as
|
||||||
|
agreement between two names.
|
||||||
|
- **Compare two media libraries with the series engine, never a path
|
||||||
|
substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
|
||||||
|
letting whichever file a walk happens to reach first stand for an episode
|
||||||
|
paired two libraries into a fake duplicate — the same sloppiness pointed the
|
||||||
|
other way deletes a file that is unique. The comparison is `media-titles.ep_of`
|
||||||
|
on both sides: series, season, episode, bytes, and a hash before anything is
|
||||||
|
trashed.
|
||||||
|
- **A deletion is staged on the item's own filesystem.** The trash rule names
|
||||||
|
`~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
|
||||||
|
stage there is a copy rather than a rename, it has to fit in the 187 GiB of
|
||||||
|
free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
|
||||||
|
`/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
|
||||||
|
rolls back exactly like the move that made it.
|
||||||
|
- **A file whose name carries no words is bonus material, not a folder.**
|
||||||
|
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
|
||||||
|
is not a movie. It becomes a review line telling a human to name the file.
|
||||||
|
- **The executor renames and nothing else.** `rename(2)` cannot cross a
|
||||||
|
filesystem, so a plan that would is refused rather than silently copying tens
|
||||||
|
of terabytes.
|
||||||
|
- **The journal is a write-ahead log: the intent is fsynced before the rename.**
|
||||||
|
An interrupted run therefore leaves only two states — an intent whose move
|
||||||
|
happened, and an intent whose move never ran — and the next run finishes the
|
||||||
|
job without moving anything twice. A file that moved with no journal line did
|
||||||
|
not move through this tool, and the tool refuses to guess on its behalf.
|
||||||
|
- **The journal lives BESIDE the library root, never inside it** (default:
|
||||||
|
`media-relayout.jsonl` next to the root): a log inside the library is a file
|
||||||
|
Jellyfin indexes.
|
||||||
|
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
|
||||||
|
duplicate destination stops the run *before the first rename*, so the operator
|
||||||
|
fixes one thing and re-runs instead of untangling a half-applied plan.
|
||||||
|
`--force` relaxes only the opinion guards; it never permits overwriting.
|
||||||
|
- **A sweep deletes only what the journal proves the run emptied.** A folder is
|
||||||
|
eligible when a journalled file moved out of it or out of something under it,
|
||||||
|
it now holds nothing but empty directories, no journalled destination lives
|
||||||
|
beneath it, and it is not a folder the run created. A folder nobody moved —
|
||||||
|
someone's placeholder — cannot be selected at all, and a symlink anywhere
|
||||||
|
inside disqualifies the subtree, because the thing behind a link is not this
|
||||||
|
tool's to delete. Every removal is journalled as `rmdir` and `rollback`
|
||||||
|
recreates the folder, so the sweep is undone by the same command as the moves.
|
||||||
|
Proof carries downward: inside a folder the journal proves emptied, every
|
||||||
|
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
|
||||||
|
move source itself, so it can hold an emptied parent until the next sweep.
|
||||||
|
A failed `rmdir` is not a removal: only a folder that is really gone leaves
|
||||||
|
scope, so the next sweep finishes what a permission problem stopped.
|
||||||
|
- **An episode-shaped name marks a SERIES, not a film — and the series is the
|
||||||
|
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
|
||||||
|
`Monk`. Keyed on the whole title instead, every episode became a group of one
|
||||||
|
and never reached the five-episode size that calls something a series, so a
|
||||||
|
box set was filed as 124 movies in 124 folders. The threshold still earns its
|
||||||
|
keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
|
||||||
|
in `_TV needs a TV library - <series>` for a human to move to a TV source —
|
||||||
|
the movie tool never files TV.
|
||||||
|
- **One journal is one undo.** The TV move runs on its own journal, and with
|
||||||
|
`--root` set to the share that holds BOTH libraries: the executor refuses any
|
||||||
|
path outside `--root`, and a journal shared with the movie relayout would
|
||||||
|
undo both runs when only one of them is wanted.
|
||||||
|
|
||||||
|
- **Sidecar metadata does not follow a rename, because it lives on another
|
||||||
|
filesystem.** The media servers write `.nfo` and artwork through the
|
||||||
|
[[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
|
||||||
|
array path, so where that view is mounted a sweep or a move must be followed by
|
||||||
|
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
|
||||||
|
that both servers browse and report as content. Whether this host mounts the
|
||||||
|
view at all is stated on that page; `media-apply.py` stays silent when it does not.
|
||||||
|
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
|
||||||
|
whiteout in the upper layer, which hides the array's copy without removing it;
|
||||||
|
the file returns when the view is unmounted. Deletions in this pipeline happen
|
||||||
|
on the raw path only, which is what the journal can undo.
|
||||||
|
- **The music album key is case-folded; the folder name is not.** `Master Of
|
||||||
|
Puppets` and `Master of Puppets` are one album to the matcher, so the spelling
|
||||||
|
that reaches the folder is the one most tracks carry. Taking it from whichever
|
||||||
|
row sorted first made the folder name depend on row order, and re-planning an
|
||||||
|
already-filed library renamed albums that were already in place.
|
||||||
|
- **An occupied path is never a destination**, not even when its occupant is
|
||||||
|
scheduled to leave: the executor refuses an occupied target, so a plan that
|
||||||
|
aims at one fails after the renames that were supposed to clear it. A file
|
||||||
|
standing where the contract puts it is a keep, and a file already carrying a
|
||||||
|
collision suffix keeps that suffix — otherwise the plan is a shuffle, not a
|
||||||
|
fixed point, and a re-run moves albums that are already right.
|
||||||
|
- **`media-census.py music` reads the `BaseItemProviders` table, not the item's
|
||||||
|
JSON.** This Jellyfin build carries an empty `ProviderIds` inside
|
||||||
|
`BaseItems.Data` for albums, so a census that parses the JSON finds no
|
||||||
|
MusicBrainz or Spotify id on any album and reads as "the library is untagged".
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
One command, from the journal of the run:
|
||||||
|
`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
|
||||||
|
any folder a sweep removed, restores every source path, removes only the folders
|
||||||
|
that run created *and that are now empty*, and refuses everything if any moved
|
||||||
|
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
|
||||||
|
|
||||||
|
## Citations
|
||||||
|
|
||||||
|
- Jellyfin Movie documentation — folder/file naming, extras subfolders,
|
||||||
|
`Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
|
||||||
|
<https://jellyfin.org/docs/general/server/media/movies>
|
||||||
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
|
|||||||
tags:
|
tags:
|
||||||
- dankserver
|
- dankserver
|
||||||
- plex
|
- plex
|
||||||
timestamp: 2026-10-21
|
timestamp: 2026-10-30
|
||||||
---
|
---
|
||||||
|
|
||||||
# Plex
|
# Plex
|
||||||
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
|
|||||||
quadlet (`plex.service`) under the systemd **user** generator, generated from
|
quadlet (`plex.service`) under the systemd **user** generator, generated from
|
||||||
`dankserver/plex/quadlet/plex.container`. It shares the host with a second live
|
`dankserver/plex/quadlet/plex.container`. It shares the host with a second live
|
||||||
media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
|
media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
|
||||||
from the same read-only Entropy Volume mount.
|
from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
|
||||||
|
sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
|
||||||
|
|
||||||
**Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
|
**Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
|
||||||
only port PMS needs for client streaming; LAN clients connect directly,
|
only port PMS needs for client streaming; LAN clients connect directly,
|
||||||
@@ -26,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
|
||||||
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
|
||||||
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
|
||||||
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
|
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
|
||||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||||
@@ -38,7 +39,9 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
|
|
||||||
The migration moved zero bytes: the container bind-mounts the native data
|
The migration moved zero bytes: the container bind-mounts the native data
|
||||||
root where it already lived, and the media archive re-mounts at
|
root where it already lived, and the media archive re-mounts at
|
||||||
`/mnt/entropy` inside the container, so every absolute library path in the
|
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
|
||||||
|
(the array today; the overlay view in git has never been deployed), so every
|
||||||
|
absolute library path in the
|
||||||
PMS database keeps resolving. The one metadata change: rootless podman
|
PMS database keeps resolving. The one metadata change: rootless podman
|
||||||
cannot map the real host uid 997, so the fleet model applies — container
|
cannot map the real host uid 997, so the fleet model applies — container
|
||||||
root runs as the `dank` user, and the deployer adopts the data root with
|
root runs as the `dank` user, and the deployer adopts the data root with
|
||||||
@@ -84,3 +87,45 @@ root runs as the `dank` user, and the deployer adopts the data root with
|
|||||||
while the HTTP layer finishes loading; the first boot of a years-old
|
while the HTTP layer finishes loading; the first boot of a years-old
|
||||||
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
|
||||||
600 s before failing; later boots are much faster (warm page cache).
|
600 s before failing; later boots are much faster (warm page cache).
|
||||||
|
|
||||||
|
## Triggering a scan
|
||||||
|
|
||||||
|
A rename on disk is invisible until the section is rescanned, and a moved file is
|
||||||
|
a `Couldn't find the file to stream` 404 until that scan passes it. Run this from
|
||||||
|
the host shell through `podman exec`, never from a neighbour container: PMS's
|
||||||
|
HTTPD closes without a single byte on any request that arrives from another
|
||||||
|
container's netns (10.89.x.x reads as "empty reply", not as a 401). The token
|
||||||
|
rides stdin, never argv, where any process on the host could read it out of
|
||||||
|
`/proc/<pid>/cmdline`; it lives in `Preferences.xml` inside the RW data bind.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umask 077
|
||||||
|
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
|
||||||
|
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
|
||||||
|
cat >/tmp/pxscan.sh <<'EOS'
|
||||||
|
read -r T
|
||||||
|
cfg=$(mktemp)
|
||||||
|
mk() { { printf 'header = "X-Plex-Token: %s"\nheader = "Accept: application/json"\n' "$T"
|
||||||
|
printf 'url = "%s"\n' "$1"; } > "$cfg"; }
|
||||||
|
mk "http://127.0.0.1:32400/library/sections"
|
||||||
|
for k in $(curl -s -K "$cfg" | grep -o '"key":"[0-9]*"' | cut -d\" -f4); do
|
||||||
|
mk "http://127.0.0.1:32400/library/sections/$k/refresh"
|
||||||
|
printf 'request = "POST"\n' >> "$cfg"
|
||||||
|
echo "plex section $k: $(curl -s -o /dev/null -w '%{http_code}' -K "$cfg")"
|
||||||
|
done
|
||||||
|
shred -u "$cfg"
|
||||||
|
EOS
|
||||||
|
podman cp /tmp/pxscan.sh plex:/tmp/pxscan.sh
|
||||||
|
podman exec -i plex sh /tmp/pxscan.sh </tmp/px.tok
|
||||||
|
shred -u /tmp/px.tok
|
||||||
|
```
|
||||||
|
|
||||||
|
Every section answers `204` — or `200` with a Directory body when
|
||||||
|
`Accept: application/json` is sent. Inside the quoted `printf` format a
|
||||||
|
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
|
||||||
|
one glued line and one stray backslash is a config file curl rejects, and the
|
||||||
|
failure prints as `option -K: is unknown`, never as a bad config line.
|
||||||
|
|
||||||
|
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
|
||||||
|
replays the run's journal against the database — the only read that
|
||||||
|
distinguishes "not scanned yet" from "gone"; see [[Media Relayout]].
|
||||||
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
|
|||||||
|
|
||||||
## Server Overview
|
## Server Overview
|
||||||
|
|
||||||
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
|
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
|
||||||
## Hardware Specifications
|
## Hardware Specifications
|
||||||
|
|
||||||
### CPU
|
### CPU
|
||||||
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
|
|||||||
|
|
||||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||||
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
||||||
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
|
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
|
||||||
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
|
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
|
||||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||||
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
|
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
|
||||||
|
|||||||
@@ -571,6 +571,29 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
|
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
|
||||||
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
|
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
|
||||||
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
|
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
|
||||||
|
- **A calibration step has an OPTIMUM, and it is not the widest step that boots.** The tap's cost is
|
||||||
|
expert re-reads per document (a 7,168-token document cut into four 2,048-token steps re-streams
|
||||||
|
each layer's routed expert union four times), which argues for the widest step — but the measured
|
||||||
|
curve turns over: at the 32,768 calibration window one 2-way prefill of the same document ran
|
||||||
|
**106 tok/s at an 8,192-token step and 76 tok/s at 16,384**, with the computed activation arena
|
||||||
|
growing 3.39 → 4.50 GiB, so the wider step pays more compute than it saves in reads. The
|
||||||
|
200,000-window `QSA score plane` refusal that made the wide step look impossible is an artifact of
|
||||||
|
serving the production window: the same 16,384-token step needs a 512 MiB plane at 32,768.
|
||||||
|
`Q38_CALIB_MNBT` is the only knob — the KL modes keep production's step, because their numbers are
|
||||||
|
only comparable at one shape.
|
||||||
|
- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
|
||||||
|
bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
|
||||||
|
while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
|
||||||
|
at second zero), and the identity check written to catch that compared two different number spaces:
|
||||||
|
`podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
|
||||||
|
cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
|
||||||
|
The binding is `podman top <ctr> hpid` against the listener from
|
||||||
|
`ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
|
||||||
|
`scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
|
||||||
|
free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
|
||||||
|
pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
|
||||||
|
Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
|
||||||
|
`No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
|
||||||
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
|
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
|
||||||
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
|
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
|
||||||
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
|
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
type: reference
|
type: reference
|
||||||
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
|
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
|
||||||
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
|
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
|
||||||
tags:
|
tags:
|
||||||
- llm
|
- llm
|
||||||
@@ -10,10 +10,10 @@ tags:
|
|||||||
- gfx1151
|
- gfx1151
|
||||||
- halogen
|
- halogen
|
||||||
- endpoint
|
- endpoint
|
||||||
timestamp: 2026-10-21
|
timestamp: 2026-10-30
|
||||||
---
|
---
|
||||||
|
|
||||||
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
|
# frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
|
||||||
|
|
||||||
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
|
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
|
||||||
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
|
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
|
||||||
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
|
|||||||
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
|
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
|
||||||
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
|
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
|
||||||
|
|
||||||
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
|
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
|
||||||
(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
|
(0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
|
||||||
unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
unauthenticated token port 8730 stays on container loopback. Engine stdout →
|
||||||
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
|
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
|
||||||
|
|
||||||
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
|
|||||||
|
|
||||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
|
| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
|
||||||
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
|
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
|
||||||
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
|
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
|
||||||
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
|
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
|
||||||
@@ -190,9 +191,11 @@ created it — earlier topologies in .
|
|||||||
|
|
||||||
## Contract (wire)
|
## Contract (wire)
|
||||||
|
|
||||||
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`
|
- Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
|
||||||
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
|
POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
|
||||||
wall-clock plus `usage.prompt_tokens`.
|
`/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
|
||||||
|
`Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
|
||||||
|
`usage.prompt_tokens`.
|
||||||
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
|
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
|
||||||
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
|
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
|
||||||
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
|
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
|
||||||
@@ -232,11 +235,15 @@ created it — earlier topologies in .
|
|||||||
XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
|
XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
|
||||||
([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
|
([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
|
||||||
same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
|
same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
|
||||||
is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog
|
is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
|
||||||
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`,
|
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
|
||||||
`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts
|
`qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
|
||||||
away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2`
|
`embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
|
||||||
(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3.
|
`d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
|
||||||
|
env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
|
||||||
|
re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
|
||||||
|
had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
|
||||||
|
bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
|
||||||
|
|
||||||
Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
|
Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
|
||||||
(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
|
(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
|
||||||
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
|
|||||||
unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
|
unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
|
||||||
size and sha256 against the catalog and the payload mount carries the tree in.
|
size and sha256 against the catalog and the payload mount carries the tree in.
|
||||||
|
|
||||||
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
|
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
|
||||||
|
|
||||||
Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
|
Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
|
||||||
set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,
|
set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
type: reference
|
type: reference
|
||||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
|
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 12118), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
|
||||||
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
|
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
|
||||||
tags:
|
tags:
|
||||||
- llm
|
- llm
|
||||||
@@ -31,7 +31,7 @@ is the bug.
|
|||||||
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
|
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
|
||||||
| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
|
| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
|
||||||
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
|
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
|
||||||
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
|
| Runtime | `/home/dank/GitPackages/llama-prism-0244744/bin/llama-server` — PrismML **fork**, `0.7.0-dev (build 12118, commit 024474448)`; the only loader for this payload |
|
||||||
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
|
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
|
||||||
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
|
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
|
||||||
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
|
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
|
||||||
@@ -136,6 +136,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
|
|||||||
|
|
||||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
|
| 2026-10-30 → | same 9B MTP file, Q8_0 mmproj and refusal adapter (payload untouched) | PrismML fork **12118** (`0.7.0-dev`, `024474448`) | unchanged (3 × 200,704; f16 K + q8_0 V; MTP depth 2) | fork rebuilt from `origin/prism`; the payload gate held — 202 `pq2_0` tensors and 202 Hadamard-folded weights loaded, MTP drafting engaged, chat/tools/vision answered | `LLAMA_BIN` back to `llama-prism-b10777/llama-server` (kept on disk) + redeploy |
|
||||||
| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
|
| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
|
||||||
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
|
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
|
||||||
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
|
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
|
||||||
@@ -201,7 +202,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
|
|||||||
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
|
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
|
||||||
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
|
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
|
||||||
- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
|
- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
|
||||||
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter.
|
- **A fork rebuild needs an install RPATH and a bin/lib-aware `LLAMA_BIN`.** `cmake --install` strips the
|
||||||
|
build-tree rpath, so the installed `llama-server` dies on `libllama-server-impl.so` unless the configure
|
||||||
|
carries `-DCMAKE_INSTALL_RPATH='$ORIGIN;$ORIGIN/../lib'`; the installed layout is `bin/` + `lib/`, not the
|
||||||
|
flat directory b10777 shipped, so the env's `LLAMA_BIN` has to name `…/bin/llama-server`. Build 12118 was
|
||||||
|
made from `origin/prism` with the 10777 configure (Release, `GGML_CUDA=ON`, `sm_86`,
|
||||||
|
`CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13`, Ninja) — gcc 15 is the box default and nvcc 12.4 rejects it.
|
||||||
|
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. Build 12118 clears all
|
||||||
|
three.
|
||||||
|
|
||||||
## Contract
|
## Contract
|
||||||
|
|
||||||
|
|||||||
@@ -49,8 +49,10 @@ is a locator: one row, one clause, no numbers.
|
|||||||
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
|
||||||
| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. |
|
| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. |
|
||||||
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
|
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
|
||||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. |
|
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||||
|
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||||
|
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||||
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
|
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
|
||||||
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
|
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
|
||||||
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
|
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
|
||||||
|
|||||||
@@ -133,6 +133,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
|||||||
[ -z "$output" ]
|
[ -z "$output" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "a compose edit never restarts a Quadlet-owned media server" {
|
||||||
|
# dankserver/compose.yaml is an inventory; the media servers' Quadlets own
|
||||||
|
# their containers. A compose edit that restarted them would cut real viewers
|
||||||
|
# for a change that cannot affect them.
|
||||||
|
run python3 "$PLANNER" --target dankserver --paths dankserver/compose.yaml
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
! printf '%s\n' "$output" | grep -qx jellyfin
|
||||||
|
! printf '%s\n' "$output" | grep -qx plex
|
||||||
|
}
|
||||||
|
|
||||||
@test "compose planner narrows a compose edit to the changed service" {
|
@test "compose planner narrows a compose edit to the changed service" {
|
||||||
local base head
|
local base head
|
||||||
base=$(git -C "$REPO_ROOT" rev-parse HEAD)
|
base=$(git -C "$REPO_ROOT" rev-parse HEAD)
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env bats
|
#!/usr/bin/env bats
|
||||||
# Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11)
|
# Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11)
|
||||||
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
|
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
|
||||||
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
|
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.18.20,
|
||||||
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
|
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
|
||||||
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
|
||||||
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
|
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
|
||||||
|
|||||||
@@ -126,11 +126,22 @@ run_stage() {
|
|||||||
@test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" {
|
@test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" {
|
||||||
# The payload path IS the endpoint's identity, and the runtime is part of
|
# The payload path IS the endpoint's identity, and the runtime is part of
|
||||||
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata
|
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata
|
||||||
# make the PrismML build the only loader, and this build must be >= b10776
|
# make the PrismML build the only loader, and that build must be >= b10776
|
||||||
# because MTP beside q8_0 V is illegal below it. A retarget that moves the
|
# because MTP beside q8_0 V is illegal below it. A retarget that moves the
|
||||||
# model but leaves the old build behind ships an endpoint that cannot load
|
# model but leaves a stock llama.cpp behind ships an endpoint that cannot
|
||||||
# (type 142) or cannot run the documented MTP arm (the build gate).
|
# load (type 142) or cannot run the documented MTP arm (the build gate).
|
||||||
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE"
|
#
|
||||||
|
# Assert the fork FAMILY, not the directory one rebuild happens to install:
|
||||||
|
# a literal path turns every runtime move into a test edit, and the suite
|
||||||
|
# runs on a box that has no /home/dank at all, so existence cannot be
|
||||||
|
# checked here either.
|
||||||
|
local bin
|
||||||
|
bin="$(sed -n 's/^LLAMA_BIN=//p' "$ENV_FILE")"
|
||||||
|
if [[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/llama-server$ ]] &&
|
||||||
|
[[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/bin/llama-server$ ]]; then
|
||||||
|
echo "LLAMA_BIN does not name a PrismML fork build: '$bin'" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
|
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
|
||||||
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
|
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
|
||||||
grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE"
|
grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE"
|
||||||
|
|||||||
Executable
+297
@@ -0,0 +1,297 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/jellyfin-providers.sh — the only mechanism that puts a
|
||||||
|
# metadata-provider key into Jellyfin's plugin configuration on Dankserver.
|
||||||
|
#
|
||||||
|
# Output vocabulary the tests depend on:
|
||||||
|
# "+ <action>" something this run will change
|
||||||
|
# "= <fact>" something already true, so nothing changes
|
||||||
|
#
|
||||||
|
# The contracts, in the order the seam tests them:
|
||||||
|
# 1. plan reports every action and never prints a key;
|
||||||
|
# 2. a secret file that is not mode 600 is refused by name and nothing is
|
||||||
|
# written — a key readable by other users is an error, not a warning;
|
||||||
|
# 3. a downloaded plugin artifact is accepted only on its pinned sha256, its
|
||||||
|
# official guid and its pinned version (a swapped or downgraded artifact
|
||||||
|
# is refused before anything lands in plugins/);
|
||||||
|
# 4. the TMDb key is written into the EXISTING config with every other
|
||||||
|
# setting preserved — Jellyfin owns that file, and clobbering it would
|
||||||
|
# silently reset country, language and adult settings;
|
||||||
|
# 5. a TVDB config the operator already saved through the UI is kept;
|
||||||
|
# 6. install is idempotent: a second run changes no byte;
|
||||||
|
# 7. whatever is replaced is copied to the trash first (fleet law: never
|
||||||
|
# overwrite without a recoverable copy);
|
||||||
|
# 8. verify answers whether the endpoint is configured, key LENGTH only;
|
||||||
|
# 9. no plugin configuration XML is tracked in git, the deployer places the
|
||||||
|
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
|
||||||
|
# installer to jellyfin.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
PROV="$REPO_ROOT/scripts/jellyfin-providers.sh"
|
||||||
|
TMDB_KEY="tmdbk3a9Zq7Rtestkey"
|
||||||
|
TVDB_KEY="tvdbk8b2Wt4Ptestkey"
|
||||||
|
GUID_OFFICIAL="a677c0da-fac5-4cde-941a-7134223f14c8"
|
||||||
|
|
||||||
|
# make_plugin <file> <guid> <version> — a zip shaped like the official TheTVDB
|
||||||
|
# artifact (meta.json + its two DLLs), so extraction is exercised for real.
|
||||||
|
make_plugin() {
|
||||||
|
python3 - "$1" "$2" "$3" <<'PYEOF'
|
||||||
|
import json, sys, zipfile
|
||||||
|
path, guid, version = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||||
|
meta = {
|
||||||
|
"category": "MoviesAndShows",
|
||||||
|
"guid": guid,
|
||||||
|
"name": "TheTVDB",
|
||||||
|
"owner": "jellyfin",
|
||||||
|
"targetAbi": "12.0.0.0",
|
||||||
|
"version": version,
|
||||||
|
}
|
||||||
|
with zipfile.ZipFile(path, "w") as z:
|
||||||
|
z.writestr("meta.json", json.dumps(meta))
|
||||||
|
z.writestr("Jellyfin.Plugin.Tvdb.dll", "dll-bytes")
|
||||||
|
z.writestr("Tvdb.Sdk.dll", "dll-bytes")
|
||||||
|
PYEOF
|
||||||
|
sha256sum "$1" | cut -d' ' -f1
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
JF="$BATS_TEST_TMPDIR/jf"
|
||||||
|
TRASH="$BATS_TEST_TMPDIR/trash"
|
||||||
|
SEC="$BATS_TEST_TMPDIR/secrets"
|
||||||
|
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
|
||||||
|
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<MetadataCountryCode>US</MetadataCountryCode>
|
||||||
|
<PreferOriginalLanguage />
|
||||||
|
<Language>en</Language>
|
||||||
|
<EnableCinematicForTrailers>false</EnableCinematicForTrailers>
|
||||||
|
<IncludeAdult>false</IncludeAdult>
|
||||||
|
<MaxCastMembers>25</MaxCastMembers>
|
||||||
|
<AddGenresFromKeywords>true</AddGenresFromKeywords>
|
||||||
|
<TmdbApiKey />
|
||||||
|
</Configuration>
|
||||||
|
XML
|
||||||
|
KEYS="$SEC/provider-keys"
|
||||||
|
printf 'TMDB_API_KEY=%s\nTVDB_API_KEY=%s\n' "$TMDB_KEY" "$TVDB_KEY" >"$KEYS"
|
||||||
|
chmod 600 "$KEYS"
|
||||||
|
ZIP="$BATS_TEST_TMPDIR/thetvdb.zip"
|
||||||
|
PIN_SHA="$(make_plugin "$ZIP" "$GUID_OFFICIAL" 24.0.0.0)"
|
||||||
|
PROV_ENV=(
|
||||||
|
JF_ROOT="$JF"
|
||||||
|
PROV_KEYS="$KEYS"
|
||||||
|
TRASH_DIR="$TRASH"
|
||||||
|
TVDB_PLUGIN_URL="file://$ZIP"
|
||||||
|
TVDB_PLUGIN_SHA256="$PIN_SHA"
|
||||||
|
TVDB_PLUGIN_GUID="$GUID_OFFICIAL"
|
||||||
|
TVDB_PLUGIN_VERSION=24.0.0.0
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
prov() { env "${PROV_ENV[@]}" bash "$PROV" "$@"; }
|
||||||
|
|
||||||
|
tree_fingerprint() { find "$JF" -type f | sort | xargs -r sha256sum | sha256sum; }
|
||||||
|
|
||||||
|
@test "plan names the plugin and both config files, and prints no key" {
|
||||||
|
run prov plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '^+ install thetvdb_24\.0\.0\.0'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '^+ set TmdbApiKey'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '^+ create Jellyfin\.Plugin\.Tvdb\.xml'
|
||||||
|
# The whole point of running this over a shoulder: a key never appears.
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "plan writes nothing" {
|
||||||
|
before="$(tree_fingerprint)"
|
||||||
|
run prov plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(tree_fingerprint)" = "$before" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an already-configured endpoint reports no pending action" {
|
||||||
|
prov install >/dev/null
|
||||||
|
run prov plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -q '^+ '
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '= thetvdb_24\.0\.0\.0 already installed'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '= TmdbApiKey already set'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -q '= Jellyfin\.Plugin\.Tvdb\.xml already has an ApiKey'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a keys file that is not mode 600 is refused and nothing is written" {
|
||||||
|
chmod 644 "$KEYS"
|
||||||
|
before="$(tree_fingerprint)"
|
||||||
|
run prov install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qF '600'
|
||||||
|
[ "$(tree_fingerprint)" = "$before" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a missing TVDB key is refused by name before anything is fetched" {
|
||||||
|
printf 'TMDB_API_KEY=%s\n' "$TMDB_KEY" >"$KEYS"
|
||||||
|
chmod 600 "$KEYS"
|
||||||
|
run prov install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qF 'TVDB_API_KEY'
|
||||||
|
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a root with no plugin directory is refused instead of fabricated" {
|
||||||
|
# Creating plugins/ inside a wrong path would install a plugin Jellyfin
|
||||||
|
# never loads, and would hide the operator's typo.
|
||||||
|
run env "${PROV_ENV[@]}" JF_ROOT="$BATS_TEST_TMPDIR/nope" bash "$PROV" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[ ! -d "$BATS_TEST_TMPDIR/nope/plugins" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an artifact whose sha256 is not the pinned one is refused" {
|
||||||
|
run env "${PROV_ENV[@]}" TVDB_PLUGIN_SHA256="$(printf 'f%.0s' {1..64})" \
|
||||||
|
bash "$PROV" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'sha256'
|
||||||
|
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an artifact that is not the official TVDB plugin is refused" {
|
||||||
|
# A matching sha proves only that the bytes are the bytes we fetched.
|
||||||
|
EVIL="$BATS_TEST_TMPDIR/evil.zip"
|
||||||
|
EVIL_SHA="$(make_plugin "$EVIL" "deadbeef-0000-0000-0000-000000000000" 24.0.0.0)"
|
||||||
|
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$EVIL" TVDB_PLUGIN_SHA256="$EVIL_SHA" \
|
||||||
|
bash "$PROV" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'guid'
|
||||||
|
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an artifact that is not the pinned version is refused" {
|
||||||
|
OLD="$BATS_TEST_TMPDIR/old.zip"
|
||||||
|
OLD_SHA="$(make_plugin "$OLD" "$GUID_OFFICIAL" 22.0.0.0)"
|
||||||
|
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$OLD" TVDB_PLUGIN_SHA256="$OLD_SHA" \
|
||||||
|
bash "$PROV" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'version'
|
||||||
|
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a different plugin version already in plugins/ is refused, not doubled" {
|
||||||
|
# Two thetvdb_* directories means Jellyfin loads one plugin twice; the
|
||||||
|
# operator removes the old one (into the trash) deliberately, not by an
|
||||||
|
# installer quietly adding a second.
|
||||||
|
prov install >/dev/null
|
||||||
|
OLD="$BATS_TEST_TMPDIR/old.zip"
|
||||||
|
OLD_SHA="$(make_plugin "$OLD" "$GUID_OFFICIAL" 22.0.0.0)"
|
||||||
|
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$OLD" TVDB_PLUGIN_SHA256="$OLD_SHA" \
|
||||||
|
TVDB_PLUGIN_VERSION=22.0.0.0 bash "$PROV" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qF 'thetvdb_24.0.0.0'
|
||||||
|
[ ! -d "$JF/plugins/thetvdb_22.0.0.0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "install places the plugin, keys TMDb, and preserves every other setting" {
|
||||||
|
run prov install
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
|
||||||
|
[ -f "$JF/plugins/thetvdb_24.0.0.0/meta.json" ]
|
||||||
|
[ -f "$JF/plugins/thetvdb_24.0.0.0/Jellyfin.Plugin.Tvdb.dll" ]
|
||||||
|
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml"
|
||||||
|
# Jellyfin owns this file: the settings it wrote must survive the edit.
|
||||||
|
for setting in MetadataCountryCode Language IncludeAdult MaxCastMembers AddGenresFromKeywords; do
|
||||||
|
grep -qF "<$setting>" "$cfg" || {
|
||||||
|
echo "lost $setting"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
done
|
||||||
|
grep -qF '<IncludeAdult>false</IncludeAdult>' "$cfg"
|
||||||
|
grep -qF "<TmdbApiKey>$TMDB_KEY</TmdbApiKey>" "$cfg"
|
||||||
|
grep -qF "<ApiKey>$TVDB_KEY</ApiKey>" "$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a TVDB config the operator saved through the UI is kept" {
|
||||||
|
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml" <<XML
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
|
||||||
|
<ApiKey>ui-saved-key</ApiKey>
|
||||||
|
<EnableSeasonPathMatching>true</EnableSeasonPathMatching>
|
||||||
|
</Configuration>
|
||||||
|
XML
|
||||||
|
run prov install
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'kept'
|
||||||
|
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
grep -qF '<ApiKey>ui-saved-key</ApiKey>' "$cfg"
|
||||||
|
grep -qF '<EnableSeasonPathMatching>true</EnableSeasonPathMatching>' "$cfg"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a pin in the TVDB secret lands in the plugin config" {
|
||||||
|
# A subscriber-shaped key needs the account's PIN beside it; a licensed key
|
||||||
|
# does not, so the element appears only when the secret carries one.
|
||||||
|
printf 'TMDB_API_KEY=%s\nTVDB_API_KEY=%s\nTVDB_PIN=123456\n' "$TMDB_KEY" "$TVDB_KEY" >"$KEYS"
|
||||||
|
chmod 600 "$KEYS"
|
||||||
|
prov install >/dev/null
|
||||||
|
grep -qF '<Pin>123456</Pin>' "$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "install is idempotent: a second run changes no byte" {
|
||||||
|
prov install >/dev/null
|
||||||
|
before="$(tree_fingerprint)"
|
||||||
|
run prov install
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(tree_fingerprint)" = "$before" ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'no change'
|
||||||
|
# One plugin directory only: two versions in plugins/ means two loads.
|
||||||
|
[ "$(find "$JF/plugins" -maxdepth 1 -type d -name 'thetvdb_*' | wc -l)" -eq 1 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "what install replaces is copied to the trash first" {
|
||||||
|
prov install >/dev/null
|
||||||
|
# The TMDb config existed before this tool touched it, so the pre-edit copy
|
||||||
|
# must stay recoverable.
|
||||||
|
[ -n "$(find "$TRASH" -name 'Jellyfin.Plugin.Tmdb.xml*' -print -quit)" ]
|
||||||
|
grep -qF '<TmdbApiKey />' "$(find "$TRASH" -name 'Jellyfin.Plugin.Tmdb.xml*' | head -1)"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "verify refuses an unconfigured endpoint and names what is missing" {
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'thetvdb'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qiF 'TmdbApiKey'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "verify passes once the plugin and both keys are in place" {
|
||||||
|
prov install >/dev/null
|
||||||
|
run prov verify
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
|
||||||
|
# Length, not bytes: the operator sees the key loaded without reading it.
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "no plugin configuration XML is tracked in this repo" {
|
||||||
|
# These files hold provider keys and are owned by the running container;
|
||||||
|
# tracking them would put credentials in git and guarantee drift.
|
||||||
|
run git -C "$REPO_ROOT" ls-files
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qF 'plugins/configurations'
|
||||||
|
! printf '%s\n' "${lines[@]}" | grep -qiF 'Plugin.Tvdb.xml'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the planner routes this installer to jellyfin" {
|
||||||
|
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
|
||||||
|
--paths scripts/jellyfin-providers.sh
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$output" = "jellyfin" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the deployer places the keys before it restarts jellyfin" {
|
||||||
|
# The plugin and both keys are read at start-up, so a restart that runs
|
||||||
|
# ahead of the placement boots the old, keyless state.
|
||||||
|
body="$(sed -n '/^deploy_jellyfin()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
[ -n "$body" ]
|
||||||
|
up="$(printf '%s\n' "$body" | grep -n 'ensure_provider_keys' | head -1 | cut -d: -f1)"
|
||||||
|
restart="$(printf '%s\n' "$body" | grep -n 'systemctl --user restart jellyfin.service' | head -1 | cut -d: -f1)"
|
||||||
|
[ "${up:-999}" -lt "${restart:-0}" ]
|
||||||
|
}
|
||||||
Executable
+438
@@ -0,0 +1,438 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: the audiobook-library relayout — scripts/media-census.py audiobooks (the
|
||||||
|
# export of the rows Plex matched) and scripts/media-audiobooks.py (the plan).
|
||||||
|
#
|
||||||
|
# The audiobook tree is the one library on this box that Jellyfin does not read
|
||||||
|
# at all and Plex reads completely: every chapter here is a Plex Track row under
|
||||||
|
# an Album row (the BOOK) under one Artist row (the NARRATOR). That is the
|
||||||
|
# authority this seam files by — never the collection folder a release pack made.
|
||||||
|
#
|
||||||
|
# Two laws here are the opposite of the music tool's, and both come from what the
|
||||||
|
# rows actually contain. Files are NEVER renumbered: 7,556 of 10,485 books carry
|
||||||
|
# at least one chapter whose index Plex does not know, so a `00 -` invented from
|
||||||
|
# a missing index would order a book wrong, and playback order comes from the
|
||||||
|
# embedded track tag either way — the name buys nothing and can only lie. And the
|
||||||
|
# folder artist is the NARRATOR even though a reader thinks in authors: every
|
||||||
|
# book has exactly one artist row, while the author survives only in 10% of the
|
||||||
|
# album titles (`Kryptos by Graham McNeill`) and in no column Plex stores, so one
|
||||||
|
# rule beats a shelf that is half authors and half narrators.
|
||||||
|
#
|
||||||
|
# Every test states which authority a file has and asserts where the PLAN files
|
||||||
|
# it, never how the planner decided.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
PLAN="$REPO_ROOT/scripts/media-audiobooks.py"
|
||||||
|
CENSUS="$REPO_ROOT/scripts/media-census.py"
|
||||||
|
APPLY="$REPO_ROOT/scripts/media-apply.py"
|
||||||
|
|
||||||
|
# at PATH SIZE — a file with deterministic content.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
# row FILE NARRATOR BOOK YEAR TRACK CHAPTER
|
||||||
|
# One row of the authority export scripts/media-census.py audiobooks writes.
|
||||||
|
row() {
|
||||||
|
printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||||
|
"$1" "$2" "$3" "${4:-}" "${5:-}" "$6" >> "$ROWS"
|
||||||
|
}
|
||||||
|
|
||||||
|
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
|
||||||
|
dst() {
|
||||||
|
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
# kind SRC — the kind the plan gave SRC's move row.
|
||||||
|
kind() {
|
||||||
|
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $4}' "$OUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
AB="$BATS_TEST_TMPDIR/Audiobooks"
|
||||||
|
PARK="$AB/_unsorted"
|
||||||
|
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
|
||||||
|
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
|
||||||
|
OUT="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
|
||||||
|
# The header is part of the authority export, so the fixture carries it.
|
||||||
|
printf 'file\tnarrator\tbook\tyear\ttrack\tchapter\n' > "$ROWS"
|
||||||
|
|
||||||
|
# One book of one author, split across the two part folders a pack shipped it
|
||||||
|
# in, with a cover above part 1 and a playlist beside the chapters.
|
||||||
|
A="$AB/Stephen King/It (1986)"
|
||||||
|
at "$A [Part 1]/01 - Chapter 1.mp3" 9000
|
||||||
|
at "$A [Part 1]/02 - Chapter 2.mp3" 9000
|
||||||
|
at "$A [Part 1]/cover.jpg" 20
|
||||||
|
at "$A [Part 1]/It.m3u" 30
|
||||||
|
at "$A [Part 2]/03 - Chapter 3.mp3" 9000
|
||||||
|
row "$A [Part 1]/01 - Chapter 1.mp3" "George Guidall" "It" 1986 1 "Chapter 1"
|
||||||
|
row "$A [Part 1]/02 - Chapter 2.mp3" "George Guidall" "It" 1986 2 "Chapter 2"
|
||||||
|
row "$A [Part 2]/03 - Chapter 3.mp3" "George Guidall" "It" 1986 3 "Chapter 3"
|
||||||
|
|
||||||
|
# A book whose album row carries no year at all.
|
||||||
|
B="$AB/Collected/Project Gold"
|
||||||
|
at "$B/01.ogg" 9000
|
||||||
|
row "$B/01.ogg" "Operator X" "Project Gold" "" 1 "Opening"
|
||||||
|
|
||||||
|
# One book title, two narrators: two different books.
|
||||||
|
C="$AB/Shared"
|
||||||
|
at "$C/Dune side a.m4b" 9000
|
||||||
|
at "$C/Dune side b.m4b" 9000
|
||||||
|
at "$C/Dune 2020.m4b" 9000
|
||||||
|
row "$C/Dune side a.m4b" "Scott Brick" "Dune" 1965 1 "Prologue"
|
||||||
|
row "$C/Dune side b.m4b" "Scott Brick" "Dune" 1965 2 "Dune"
|
||||||
|
row "$C/Dune 2020.m4b" "Malcolm Hillgartner" "Dune" 2020 1 "Book One"
|
||||||
|
|
||||||
|
# Two part folders of one book whose chapters share a file name.
|
||||||
|
D="$AB/Pulp"
|
||||||
|
at "$D/Book A/vol1.mp3" 9000
|
||||||
|
at "$D/Book A vol2/vol1.mp3" 9000
|
||||||
|
row "$D/Book A/vol1.mp3" "Nora Pace" "Hard Food" 2001 1 "Vol 1"
|
||||||
|
row "$D/Book A vol2/vol1.mp3" "Nora Pace" "Hard Food" 2001 2 "Vol 1"
|
||||||
|
|
||||||
|
# The same narrator spelled two ways across one book's chapters.
|
||||||
|
E="$AB/Spell"
|
||||||
|
at "$E/kharn 1.mp3" 9000
|
||||||
|
at "$E/kharn 2.mp3" 9000
|
||||||
|
at "$E/kharn 3.mp3" 9000
|
||||||
|
row "$E/kharn 1.mp3" "David Timson" "Kharn" 2013 1 "One"
|
||||||
|
row "$E/kharn 2.mp3" "David timson" "Kharn" 2013 2 "Two"
|
||||||
|
row "$E/kharn 3.mp3" "David Timson" "Kharn" 2013 3 "Three"
|
||||||
|
|
||||||
|
# A folder holding two different books: its loose note has no owner.
|
||||||
|
F="$AB/Mixed"
|
||||||
|
at "$F/book one chapter.mp3" 9000
|
||||||
|
at "$F/book two chapter.mp3" 9000
|
||||||
|
at "$F/notes.txt" 10
|
||||||
|
row "$F/book one chapter.mp3" "Ann Reader" "Book One" 2001 1 "One"
|
||||||
|
row "$F/book two chapter.mp3" "Bob Reader" "Book Two" 2002 1 "Two"
|
||||||
|
|
||||||
|
# A file already parked: the pile is for a human, so a row that would file it
|
||||||
|
# must not pull it back into the tree.
|
||||||
|
at "$PARK/Old/nobody.mp3" 9000
|
||||||
|
row "$PARK/Old/nobody.mp3" "Ann Reader" "Book One" 2001 1 "One"
|
||||||
|
|
||||||
|
# The odd cases: no row, a video container, a name carrying a character Plex
|
||||||
|
# says breaks playback, rows with a blank book and a blank narrator, and a
|
||||||
|
# matched chapter whose file has no extension.
|
||||||
|
G="$AB/Odd"
|
||||||
|
at "$G/loose.mp3" 9000
|
||||||
|
at "$G/concert.mp4" 9000
|
||||||
|
at "$G/Bad: Name/01 chapter.mp3" 9000
|
||||||
|
row "$G/Bad: Name/01 chapter.mp3" "Some Reader" "Trail: Marked" 1999 1 "Marked"
|
||||||
|
at "$G/blank book.mp3" 9000
|
||||||
|
row "$G/blank book.mp3" "Some Reader" "" 2000 1 "No Book"
|
||||||
|
at "$G/blank narrator.mp3" 9000
|
||||||
|
row "$G/blank narrator.mp3" "" "No Artist" 2000 1 "No Artist"
|
||||||
|
at "$G/NoExt/part01" 9000
|
||||||
|
row "$G/NoExt/part01" "Archivist" "Tapes" 1971 1 "Tape One"
|
||||||
|
|
||||||
|
# Tag-scan placeholders. Two unrelated books whose album rows both read
|
||||||
|
# `[Unknown Album]`: the name is shared, so filing on it would merge books
|
||||||
|
# that have nothing to do with each other. A rip timestamp can be glued to
|
||||||
|
# the placeholder, and a multi-cast read says `Various Artists` — which IS a
|
||||||
|
# narrator.
|
||||||
|
H="$AB/Packs"
|
||||||
|
at "$H/Stross/Ship/c1.mp3" 9000
|
||||||
|
at "$H/Leckie/State/c1.mp3" 9000
|
||||||
|
at "$H/Rip/tape.mp3" 9000
|
||||||
|
at "$H/NoWho/Real Book.mp3" 9000
|
||||||
|
at "$H/Multicast/Genesis.mp3" 9000
|
||||||
|
row "$H/Stross/Ship/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
|
||||||
|
row "$H/Leckie/State/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
|
||||||
|
row "$H/Rip/tape.mp3" "Ann Reader" "Unknown album (03/03/2015 21:39:55)" "" 1 "One"
|
||||||
|
row "$H/NoWho/Real Book.mp3" "[Unknown Artist]" "A Named Book" 2001 1 "One"
|
||||||
|
row "$H/Multicast/Genesis.mp3" "Various Artists" "The Bible" "" 1 "Genesis"
|
||||||
|
at "$H/Numbered/01 - Grimoire.mp3" 9000
|
||||||
|
at "$H/Digits/1984.mp3" 9000
|
||||||
|
row "$H/Numbered/01 - Grimoire.mp3" "0.4" "Grimoire of the Lamb" "" 1 "One"
|
||||||
|
row "$H/Digits/1984.mp3" "Ann Reader" "1984" 1949 1 "One"
|
||||||
|
|
||||||
|
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
|
||||||
|
# path, no header.
|
||||||
|
python3 - "$AB" "$ENUM" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
root, out = sys.argv[1], sys.argv[2]
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
for dirpath, _dirs, files in os.walk(root):
|
||||||
|
for fn in files:
|
||||||
|
p = os.path.join(dirpath, fn)
|
||||||
|
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
plan() {
|
||||||
|
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS" \
|
||||||
|
--out "$OUT" --exceptions "$EXC"
|
||||||
|
if [ "$status" -ne 0 ]; then
|
||||||
|
echo "planner said: $output"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# db_with_part DB FILE NARRATOR BOOK YEAR TRACK CHAPTER — one Plex Track row
|
||||||
|
# under an Album under an Artist, in the three tables the export joins.
|
||||||
|
db_with_part() {
|
||||||
|
local db="$1" file="$2" narrator="$3" book="$4" year="$5" track="$6" chapter="$7"
|
||||||
|
local del="${8:-0}"
|
||||||
|
python3 - "$db" "$file" "$narrator" "$book" "$year" "$track" "$chapter" "$del" <<'PY'
|
||||||
|
import sqlite3, sys
|
||||||
|
db, file, narrator, book, year, track, chapter, deleted = sys.argv[1:9]
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
c.execute("create table if not exists media_parts (id integer primary key,"
|
||||||
|
" media_item_id, file, deleted_at)")
|
||||||
|
c.execute("create table if not exists media_items (id integer primary key,"
|
||||||
|
" metadata_item_id, deleted_at)")
|
||||||
|
c.execute("""create table if not exists metadata_items (id integer primary key,
|
||||||
|
parent_id, metadata_type, title, year, "index", deleted_at)""")
|
||||||
|
now = None if deleted == "0" else "2026-01-01"
|
||||||
|
ar = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
|
||||||
|
" \"index\", deleted_at) VALUES (NULL, 8, ?, NULL, NULL, NULL)",
|
||||||
|
(narrator,)).lastrowid
|
||||||
|
al = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
|
||||||
|
" \"index\", deleted_at) VALUES (?, 9, ?, ?, NULL, NULL)",
|
||||||
|
(ar, book, int(year) if year else None)).lastrowid
|
||||||
|
mi = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
|
||||||
|
" \"index\", deleted_at) VALUES (?, 10, ?, NULL, ?, NULL)",
|
||||||
|
(al, chapter, int(track) if track else -1)).lastrowid
|
||||||
|
it = c.execute("insert into media_items (metadata_item_id, deleted_at) VALUES (?, NULL)",
|
||||||
|
(mi,)).lastrowid
|
||||||
|
c.execute("insert into media_parts (media_item_id, file, deleted_at) VALUES (?,?,?)",
|
||||||
|
(it, file, now))
|
||||||
|
c.commit()
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ the export
|
||||||
|
|
||||||
|
@test "the census exports the narrator, book and chapter Plex matched for a part" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/plex.db"
|
||||||
|
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01.mp3" \
|
||||||
|
"George Guidall" "It" 1986 1 "Chapter 1"
|
||||||
|
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "${lines[0]}" = "file narrator book year track chapter" ]
|
||||||
|
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
|
||||||
|
| grep -q "^$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01\.mp3 George Guidall It 1986 1 Chapter 1$"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the census exports nothing outside the root, and nothing Plex deleted" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/plex.db"
|
||||||
|
db_with_part "$DB" "/mnt/other/Library/01.mp3" "Someone" "Elsewhere" 1990 1 "One"
|
||||||
|
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Gone/01.mp3" "Someone" \
|
||||||
|
"Deleted" 1990 1 "One" 1
|
||||||
|
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(wc -l < "$BATS_TEST_TMPDIR/exp.tsv")" -eq 1 ] # header only
|
||||||
|
}
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------- the plan
|
||||||
|
|
||||||
|
@test "a matched chapter files under its narrator and its book, name untouched" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
|
||||||
|
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "one book arrives in one folder however many part folders it shipped in" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A [Part 2]/03 - Chapter 3.mp3")" = \
|
||||||
|
"$AB/George Guidall/It (1986)/03 - Chapter 3.mp3" ]
|
||||||
|
[ "$(dst "$A [Part 1]/02 - Chapter 2.mp3")" = \
|
||||||
|
"$AB/George Guidall/It (1986)/02 - Chapter 2.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a chapter is never renumbered, whether or not Plex knows its index" {
|
||||||
|
# The law this guards: playback order comes from the embedded track tag, and
|
||||||
|
# three quarters of the books have chapters with no index at all, so a
|
||||||
|
# number written into a name would be invented and would lie.
|
||||||
|
plan
|
||||||
|
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
|
||||||
|
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
|
||||||
|
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a book whose album row has no year gets no parentheses" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two chapters claiming one name inside one book are named apart" {
|
||||||
|
plan
|
||||||
|
first="$(dst "$D/Book A/vol1.mp3")"
|
||||||
|
second="$(dst "$D/Book A vol2/vol1.mp3")"
|
||||||
|
[ "$first" != "$second" ]
|
||||||
|
[ -n "$first" ] && [ -n "$second" ]
|
||||||
|
# and the plan still passes the executor's preflight, which refuses a target
|
||||||
|
# two sources claim.
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" != *"double-claim"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a cover follows the book whose folder it lies in" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A [Part 1]/cover.jpg")" = "$AB/George Guidall/It (1986)/cover.jpg" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a playlist moves with its book and its bytes are never rewritten" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A [Part 1]/It.m3u")" = "$AB/George Guidall/It (1986)/It.m3u" ]
|
||||||
|
before=$(md5sum "$A [Part 1]/It.m3u" | cut -d" " -f1)
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" --journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(md5sum "$AB/George Guidall/It (1986)/It.m3u" | cut -d" " -f1)" = "$before" ]
|
||||||
|
# the stored paths inside it still name the old file names, which the tool
|
||||||
|
# leaves alone rather than editing a file the operator owns.
|
||||||
|
[ "$(kind "$A [Part 1]/It.m3u")" != "chapter" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a note in a folder holding two books has no owner and parks" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$F/notes.txt")" = "park" ]
|
||||||
|
[ "$(dst "$F/notes.txt")" = "$PARK/Mixed/notes.txt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an audio file no row names parks with the path it has today" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$G/loose.mp3")" = "park" ]
|
||||||
|
[ "$(dst "$G/loose.mp3")" = "$PARK/Odd/loose.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a video container parks instead of being renamed into audio" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$G/concert.mp4")" = "park" ]
|
||||||
|
run grep -c "concert\.mp4" "$EXC"
|
||||||
|
[ "$output" != "0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a character Plex says breaks playback never reaches a destination" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$G/Bad: Name/01 chapter.mp3")" = \
|
||||||
|
"$AB/Some Reader/Trail Marked (1999)/01 chapter.mp3" ]
|
||||||
|
run cut -f3 "$OUT"
|
||||||
|
[[ "$output" != *":"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "one book title with two narrators stays two books" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$C/Dune side a.m4b")" = "$AB/Scott Brick/Dune (1965)/Dune side a.m4b" ]
|
||||||
|
[ "$(dst "$C/Dune 2020.m4b")" = \
|
||||||
|
"$AB/Malcolm Hillgartner/Dune (2020)/Dune 2020.m4b" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the spelling most chapters carry names the folder, and they share one folder" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$E/kharn 1.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 1.mp3" ]
|
||||||
|
[ "$(dst "$E/kharn 2.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 2.mp3" ]
|
||||||
|
[ "$(dst "$E/kharn 3.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 3.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a row with no book title parks rather than inventing a folder" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$G/blank book.mp3")" = "park" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a row with no narrator parks rather than inventing a shelf" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$G/blank narrator.mp3")" = "park" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "books that share a placeholder album name never share a shelf" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$H/Stross/Ship/c1.mp3")" = "$PARK/Packs/Stross/Ship/c1.mp3" ]
|
||||||
|
[ "$(dst "$H/Leckie/State/c1.mp3")" = "$PARK/Packs/Leckie/State/c1.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a placeholder with a rip timestamp glued to it is still a placeholder" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$H/Rip/tape.mp3")" = "park" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a narrator placeholder parks even when the book is named" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$H/NoWho/Real Book.mp3")" = "park" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "Various Artists is a multi-cast narrator and files its book" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$H/Multicast/Genesis.mp3")" = \
|
||||||
|
"$AB/Various Artists/The Bible/Genesis.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a chapter number mistagged as the narrator parks" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$H/Numbered/01 - Grimoire.mp3")" = "park" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a book whose title is digits still files, because only a name must have a letter" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$H/Digits/1984.mp3")" = "$AB/Ann Reader/1984 (1949)/1984.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the row, not the extension, decides what a chapter is" {
|
||||||
|
plan
|
||||||
|
[ "$(kind "$G/NoExt/part01")" = "chapter" ]
|
||||||
|
[ "$(dst "$G/NoExt/part01")" = "$AB/Archivist/Tapes (1971)/part01" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the plan replays through the executor with nothing refused" {
|
||||||
|
plan
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
|
||||||
|
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" != *"refusing"* ]]
|
||||||
|
# The refusal this guards is two sources claiming one target: the executor
|
||||||
|
# stops the whole run on it, so the plan must never hand a path out twice.
|
||||||
|
[ -z "$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an occupied destination is refused rather than overwritten" {
|
||||||
|
plan
|
||||||
|
mkdir -p "$AB/George Guidall/It (1986)"
|
||||||
|
at "$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" 1
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"occupied"* ]]
|
||||||
|
[ -e "$A [Part 1]/01 - Chapter 1.mp3" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "applying the plan and re-planning moves nothing" {
|
||||||
|
plan
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
|
||||||
|
--journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
python3 - "$AB" "$ENUM" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
root, out = sys.argv[1], sys.argv[2]
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
for dirpath, _dirs, files in os.walk(root):
|
||||||
|
for fn in files:
|
||||||
|
p = os.path.join(dirpath, fn)
|
||||||
|
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
|
||||||
|
PY
|
||||||
|
# The rows still name the paths the chapters were placed at: that is what a
|
||||||
|
# server re-scan of the moved tree reports.
|
||||||
|
python3 - "$OUT" "$ROWS" <<'PY'
|
||||||
|
import csv, sys
|
||||||
|
out, rows = sys.argv[1], sys.argv[2]
|
||||||
|
moved = {r[1]: r[2] for r in csv.reader(open(out), delimiter="\t") if r[0] == "move"}
|
||||||
|
with open(rows) as fh, open(rows + ".2", "w") as fh2:
|
||||||
|
fh2.write(fh.readline().rstrip("\n") + "\n")
|
||||||
|
for line in fh:
|
||||||
|
f = line.split("\t")[0]
|
||||||
|
fh2.write("\t".join([moved.get(f, f)] + line.rstrip("\n").split("\t")[1:]) + "\n")
|
||||||
|
PY
|
||||||
|
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS.2" \
|
||||||
|
--out "$OUT.2" --exceptions "$EXC.2"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(awk -F '\t' '$1 == "move"' "$OUT.2" | wc -l)" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "nothing under the park root is planned again" {
|
||||||
|
plan
|
||||||
|
run awk -F '\t' '$1 == "move" && $2 ~ /_unsorted/' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
Executable
+373
@@ -0,0 +1,373 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: the music-library relayout — scripts/media-census.py music (the tag and
|
||||||
|
# provider authority export) and scripts/media-music.py (the plan).
|
||||||
|
#
|
||||||
|
# Jellyfin's music rule is the opposite of the movie and TV ones: the embedded
|
||||||
|
# tags are the metadata and the file name is only a fallback title, so the
|
||||||
|
# authority here is the Audio row Jellyfin already matched (its album-artist,
|
||||||
|
# album, year, disc and track columns) and NOT the path a ripper chose. Every
|
||||||
|
# test states which authority a file has — row only, row with no album item,
|
||||||
|
# sidecar of a file that has a row, nothing at all — and asserts where the PLAN
|
||||||
|
# files it, never how the planner decided.
|
||||||
|
#
|
||||||
|
# The three outcomes that cost a re-run of a 90k-file plan: a track filed under
|
||||||
|
# the wrong artist (it splits one album across two artists), a destination that
|
||||||
|
# two tracks claim (the executor refuses, and the operator loses the run), and a
|
||||||
|
# lyric file that stops matching the basename of the track it belongs to, which
|
||||||
|
# silently loses the lyrics. Everything below is written to catch those, plus the
|
||||||
|
# gate that matters most: the plan must replay through scripts/media-apply.py,
|
||||||
|
# whose preflight is the fleet's only overwrite guard.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
PLAN="$REPO_ROOT/scripts/media-music.py"
|
||||||
|
CENSUS="$REPO_ROOT/scripts/media-census.py"
|
||||||
|
APPLY="$REPO_ROOT/scripts/media-apply.py"
|
||||||
|
|
||||||
|
# at PATH SIZE — a file with deterministic content.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
# row FILE ARTIST ALBUM YEAR DISC TRACK TITLE [MB] [PERFORMER]
|
||||||
|
# One row of the authority export scripts/media-census.py music writes.
|
||||||
|
row() {
|
||||||
|
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||||
|
"$1" "$2" "$3" "${4:-}" "${5:-}" "${6:-}" "$7" "${8:-}" "${9:-}" >> "$ROWS"
|
||||||
|
}
|
||||||
|
|
||||||
|
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
|
||||||
|
dst() {
|
||||||
|
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
MUSIC="$BATS_TEST_TMPDIR/Music"
|
||||||
|
PARK="$MUSIC/_unsorted"
|
||||||
|
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
|
||||||
|
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
|
||||||
|
OUT="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
|
||||||
|
|
||||||
|
# A discography pack: the artist's own albums buried under "01. Albums",
|
||||||
|
# with a lyric, a cover, an EAC log and a playlist beside the tracks.
|
||||||
|
A="$MUSIC/Artist One - Discography [FLAC]/01. Albums/Album X"
|
||||||
|
at "$A/01. First.flac" 9000
|
||||||
|
at "$A/02. Second.flac" 9000
|
||||||
|
at "$A/01. First.lrc" 10
|
||||||
|
at "$A/cover.jpg" 20
|
||||||
|
at "$A/rip.log" 10
|
||||||
|
at "$A/Album X.m3u" 10
|
||||||
|
|
||||||
|
# A multi-disc compilation: one album (the album artist is the compilation
|
||||||
|
# tag), two discs, and a different performer on each track.
|
||||||
|
B="$MUSIC/Box (2019)"
|
||||||
|
at "$B/Disc 1/01 Track A.flac" 9000
|
||||||
|
at "$B/Disc 2/01 Track B.flac" 9000
|
||||||
|
|
||||||
|
# The box cover lies above the two disc folders: the discs below it own it.
|
||||||
|
at "$B/box.jpg" 10
|
||||||
|
|
||||||
|
# One image plus its cuesheet: a whole album in one file.
|
||||||
|
C="$MUSIC/Image Cue"
|
||||||
|
at "$C/Image.flac" 9000
|
||||||
|
at "$C/Image.cue" 10
|
||||||
|
|
||||||
|
# The hard cases: no row at all, a video container, two tracks claiming one
|
||||||
|
# number and one title, a track with no number, and a track whose album
|
||||||
|
# artist tag is empty so only the performer names it.
|
||||||
|
D="$MUSIC/Odd"
|
||||||
|
at "$D/loose.flac" 9000
|
||||||
|
at "$D/concert.mkv" 9000
|
||||||
|
at "$D/dup1.flac" 9000
|
||||||
|
at "$D/dup2.flac" 9000
|
||||||
|
at "$D/unnumbered.flac" 9000
|
||||||
|
at "$D/lone.flac" 9000
|
||||||
|
|
||||||
|
# A sidecar in a folder that holds two different albums.
|
||||||
|
E="$MUSIC/Mixed"
|
||||||
|
at "$E/one.flac" 9000
|
||||||
|
at "$E/two.flac" 9000
|
||||||
|
at "$E/notes.pdf" 10
|
||||||
|
|
||||||
|
# An album title carrying a character Jellyfin says breaks playback.
|
||||||
|
F="$MUSIC/Question Mark"
|
||||||
|
at "$F/01. Bang.flac" 9000
|
||||||
|
|
||||||
|
# An album whose tags name no album artist at all, and two different
|
||||||
|
# performers; and one album title carried by two different album artists.
|
||||||
|
G="$MUSIC/Tagless"
|
||||||
|
at "$G/one.flac" 9000
|
||||||
|
at "$G/two.flac" 9000
|
||||||
|
H="$MUSIC/Same Title"
|
||||||
|
at "$H/a.flac" 9000
|
||||||
|
at "$H/b.flac" 9000
|
||||||
|
|
||||||
|
# One album whose tracks disagree on the spelling of its name; the minority
|
||||||
|
# spelling sorts first, so it must not win the folder.
|
||||||
|
I="$MUSIC/Puppets"
|
||||||
|
at "$I/one.flac" 9000
|
||||||
|
at "$I/two.flac" 9000
|
||||||
|
at "$I/three.flac" 9000
|
||||||
|
|
||||||
|
# The authority export carries a header; the planner reads it by column name.
|
||||||
|
printf 'file\tartist\talbum\tyear\tdisc\ttrack\ttitle\tmb\tperformer\n' > "$ROWS"
|
||||||
|
row "$A/01. First.flac" "Artist One" "Album X" 2015 1 1 "First Song"
|
||||||
|
row "$A/02. Second.flac" "Artist One" "Album X" 2015 1 2 "Second Song: Reprise"
|
||||||
|
row "$B/Disc 1/01 Track A.flac" "Various Artists" "Best Of" 2019 1 1 "Track A" \
|
||||||
|
"" "Aaa Singer"
|
||||||
|
row "$B/Disc 2/01 Track B.flac" "Various Artists" "Best Of" 2019 2 1 "Track B" \
|
||||||
|
"" "Bbb Singer"
|
||||||
|
row "$C/Image.flac" "Solo" "Single Album" 2001 1 1 "Single Album"
|
||||||
|
row "$D/dup1.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
|
||||||
|
row "$D/dup2.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
|
||||||
|
row "$D/unnumbered.flac" "No Number" "No Number Album" "" "" "" "Untitled"
|
||||||
|
row "$D/lone.flac" "" "Session Works" "" "" 1 "Lone" "" "Session Singer"
|
||||||
|
row "$E/one.flac" "Mix Artist" "First Set" 2000 "" 1 "One"
|
||||||
|
row "$E/two.flac" "Mix Artist" "Second Set" 2001 "" 1 "Two"
|
||||||
|
row "$F/01. Bang.flac" "Billie" "WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?" \
|
||||||
|
2019 "" 1 "!!!!!!"
|
||||||
|
row "$G/one.flac" "" "Two Seasons" 2005 "" 1 "One" "" "Alpha Band"
|
||||||
|
row "$G/two.flac" "" "Two Seasons" 2005 "" 2 "Two" "" "Beta Band"
|
||||||
|
row "$H/a.flac" "Aaa Singer" "Same Title" 2011 "" 1 "A Song"
|
||||||
|
row "$H/b.flac" "Bbb Singer" "Same Title" 2011 "" 1 "B Song"
|
||||||
|
row "$I/one.flac" "Metallica" "Master Of Puppets" 2015 "" 1 "One"
|
||||||
|
row "$I/two.flac" "Metallica" "Master of Puppets" 2015 "" 2 "Two"
|
||||||
|
row "$I/three.flac" "Metallica" "Master of Puppets" 2015 "" 3 "Three"
|
||||||
|
|
||||||
|
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
|
||||||
|
# path, no header.
|
||||||
|
python3 - "$MUSIC" "$ENUM" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
root, out = sys.argv[1], sys.argv[2]
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
for dirpath, _dirs, files in os.walk(root):
|
||||||
|
for fn in files:
|
||||||
|
p = os.path.join(dirpath, fn)
|
||||||
|
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
plan() {
|
||||||
|
run python3 "$PLAN" --root "$MUSIC" --enum "$ENUM" --rows "$ROWS" \
|
||||||
|
--out "$OUT" --exceptions "$EXC"
|
||||||
|
[ "$status" -eq 0 ] || echo "planner said: $output"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the census exports the tag and provider row Jellyfin matched for a track" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/jellyfin.db"
|
||||||
|
python3 - "$DB" <<'PY'
|
||||||
|
import sqlite3, sys, uuid
|
||||||
|
c = sqlite3.connect(sys.argv[1])
|
||||||
|
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
|
||||||
|
ParentIndexNumber, ProductionYear, Album,
|
||||||
|
AlbumArtists, Artists, Data)""")
|
||||||
|
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
|
||||||
|
AU = "MediaBrowser.Controller.Entities.Audio.Audio"
|
||||||
|
AL = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
|
||||||
|
aid, trk = str(uuid.uuid4()), "/data/Music/Pack/Album X/01. First.flac"
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(aid, None, AL, "Album X", "/data/Music/Pack/Album X", None, None, 2015,
|
||||||
|
None, "Artist One", None, '{}'))
|
||||||
|
c.execute("insert into BaseItemProviders values (?,?,?)",
|
||||||
|
(aid, "MusicBrainzReleaseGroup", "rg-1234"))
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(str(uuid.uuid4()), aid, AU, "First Song", trk, 1, 1, 2015,
|
||||||
|
"album x (the tag)", "Artist One", "Artist One", '{}'))
|
||||||
|
c.commit()
|
||||||
|
PY
|
||||||
|
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "${lines[0]}" = "file artist album year disc track title mb performer" ]
|
||||||
|
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
|
||||||
|
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/Album X/01\. First\.flac Artist One Album X 2015 1 1 First Song rg-1234 Artist One$"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a track whose folder is no album item still exports from its own tags" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/jellyfin.db"
|
||||||
|
python3 - "$DB" <<'PY'
|
||||||
|
import sqlite3, sys, uuid
|
||||||
|
c = sqlite3.connect(sys.argv[1])
|
||||||
|
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
|
||||||
|
ParentIndexNumber, ProductionYear, Album,
|
||||||
|
AlbumArtists, Artists, Data)""")
|
||||||
|
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(str(uuid.uuid4()), None, "MediaBrowser.Controller.Entities.Audio.Audio",
|
||||||
|
"Lone", "/data/Music/Pack/lone.flac", 1, None, None,
|
||||||
|
"Session Works", None, "Session Singer", '{}'))
|
||||||
|
c.commit()
|
||||||
|
PY
|
||||||
|
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
|
||||||
|
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/lone\.flac Session Singer Session Works 1 Lone Session Singer$"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a tagged track files under artist, album and year with a numbered title" {
|
||||||
|
plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(dst "$A/01. First.flac")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the tag title wins over the name the ripper gave the file" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A/02. Second.flac")" = "$MUSIC/Artist One/Album X (2015)/02 - Second Song Reprise.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a character Jellyfin says breaks playback never reaches a destination" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$F/01. Bang.flac")" = "$MUSIC/Billie/WHEN WE ALL FALL ASLEEP, WHERE DO WE GO (2019)/01 - !!!!!!.flac" ]
|
||||||
|
run awk -F '\t' '$1 == "move" && $3 ~ /[<>:"|?*]/ {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "every disc of a multi-disc album lands in one folder, numbered apart" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$B/Disc 1/01 Track A.flac")" = "$MUSIC/Various Artists/Best Of (2019)/1.01 - Track A.flac" ]
|
||||||
|
[ "$(dst "$B/Disc 2/01 Track B.flac")" = "$MUSIC/Various Artists/Best Of (2019)/2.01 - Track B.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an album whose tracks name several artists files under Various Artists" {
|
||||||
|
plan
|
||||||
|
run awk -F '\t' '$1 == "move" && $3 ~ /\/Best Of \(2019\)\/.*\.flac$/ {print $3}' "$OUT"
|
||||||
|
[ "${#lines[@]}" -eq 2 ]
|
||||||
|
[[ "${lines[0]}" == "$MUSIC/Various Artists/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an album with no album artist and two performers is a compilation" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$G/one.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/01 - One.flac" ]
|
||||||
|
[ "$(dst "$G/two.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/02 - Two.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the same album title under two album artists is two albums" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$H/a.flac")" = "$MUSIC/Aaa Singer/Same Title (2011)/01 - A Song.flac" ]
|
||||||
|
[ "$(dst "$H/b.flac")" = "$MUSIC/Bbb Singer/Same Title (2011)/01 - B Song.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the folder carries the album spelling most tracks carry" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$I/one.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/01 - One.flac" ]
|
||||||
|
[ "$(dst "$I/two.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/02 - Two.flac" ]
|
||||||
|
[ "$(dst "$I/three.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/03 - Three.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a lyric file is renamed to match the track it belongs to" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A/01. First.lrc")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.lrc" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the album image, the rip log and the playlist follow the album folder" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$A/cover.jpg")" = "$MUSIC/Artist One/Album X (2015)/cover.jpg" ]
|
||||||
|
[ "$(dst "$A/rip.log")" = "$MUSIC/Artist One/Album X (2015)/rip.log" ]
|
||||||
|
[ "$(dst "$A/Album X.m3u")" = "$MUSIC/Artist One/Album X (2015)/Album X.m3u" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a cover above the disc folders of one album follows that album" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$B/box.jpg")" = "$MUSIC/Various Artists/Best Of (2019)/box.jpg" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an image carrying a whole album keeps its own name beside its cuesheet" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$C/Image.flac")" = "$MUSIC/Solo/Single Album (2001)/Image.flac" ]
|
||||||
|
[ "$(dst "$C/Image.cue")" = "$MUSIC/Solo/Single Album (2001)/Image.cue" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an audio file no provider row names is parked, not filed under an artist" {
|
||||||
|
plan
|
||||||
|
[[ "$(dst "$D/loose.flac")" == "$PARK/Odd/loose.flac" ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a video container in the music library is parked, never renamed to an audio one" {
|
||||||
|
plan
|
||||||
|
[[ "$(dst "$D/concert.mkv")" == "$PARK/Odd/concert.mkv" ]]
|
||||||
|
run awk -F '\t' '$1 == "move" && $3 ~ /\.mka$|\.m4a$/ {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two tracks claiming one number and one title get two destinations" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$D/dup1.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same.flac" ]
|
||||||
|
[ "$(dst "$D/dup2.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same (2).flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an album with no year is named without one, a track with no number likewise" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$D/unnumbered.flac")" = "$MUSIC/No Number/No Number Album/Untitled.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a track naming only a performer files under that performer" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$D/lone.flac")" = "$MUSIC/Session Singer/Session Works/01 - Lone.flac" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a sidecar in a folder holding two albums parks instead of guessing" {
|
||||||
|
plan
|
||||||
|
[[ "$(dst "$E/notes.pdf")" == "$PARK/Mixed/notes.pdf" ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the plan creates every folder before the move that needs it" {
|
||||||
|
plan
|
||||||
|
python3 - "$OUT" "$MUSIC" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
made = set()
|
||||||
|
for line in open(sys.argv[1]).read().splitlines()[1:]:
|
||||||
|
action, _src, dest, _kind = line.split("\t")
|
||||||
|
if action == "mkdir":
|
||||||
|
made.add(dest)
|
||||||
|
elif action == "move":
|
||||||
|
parent = os.path.dirname(dest)
|
||||||
|
while parent and parent != sys.argv[2]:
|
||||||
|
assert parent in made, f"{dest} needs {parent}, which is never made"
|
||||||
|
parent = os.path.dirname(parent)
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "no move lands outside the music library" {
|
||||||
|
plan
|
||||||
|
run awk -F '\t' -v l="$MUSIC" \
|
||||||
|
'$1 == "move" && $3 !~ "^" l "/" {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the executor preflights the plan and no target is claimed twice" {
|
||||||
|
plan
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$BATS_TEST_TMPDIR" \
|
||||||
|
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
dup=$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)
|
||||||
|
[ -z "$dup" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "re-planning a library the plan already satisfies moves nothing" {
|
||||||
|
plan
|
||||||
|
# Applying the plan and re-censusing is what the operator does next: both
|
||||||
|
# authority files are keyed by path, so both are replayed through the plan
|
||||||
|
# the way a fresh census of the moved tree would see them. (The enum has no
|
||||||
|
# header; the rows export does, and its header row is not a path.)
|
||||||
|
python3 - "$OUT" "$ENUM" "$ROWS" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
plan = sys.argv[1]
|
||||||
|
moves = {}
|
||||||
|
for line in open(plan).read().splitlines()[1:]:
|
||||||
|
action, src, dest, _kind = line.split("\t")
|
||||||
|
if action == "move":
|
||||||
|
moves[src] = dest
|
||||||
|
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||||||
|
os.rename(src, dest)
|
||||||
|
for path, col in ((sys.argv[2], 2), (sys.argv[3], 0)):
|
||||||
|
rows = [line.rstrip("\n").split("\t") for line in open(path)]
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
for f in rows:
|
||||||
|
f[col] = moves.get(f[col], f[col])
|
||||||
|
fh.write("\t".join(f) + "\n")
|
||||||
|
PY
|
||||||
|
plan
|
||||||
|
run awk -F '\t' '$1 == "move" {print $2 " -> " $3}' "$OUT"
|
||||||
|
[ -z "$output" ] || { echo "moved again: $output"; false; }
|
||||||
|
}
|
||||||
Executable
+161
@@ -0,0 +1,161 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: the SSD-backed metadata view (OverlayFS) that lets Jellyfin write
|
||||||
|
# Kodi-format sidecars for Plex to read, without ever writing to the spindle
|
||||||
|
# array and without letting either media server start against a missing view.
|
||||||
|
#
|
||||||
|
# The contracts, in the order the seam tests them:
|
||||||
|
# 1. the plan puts the writable layer on the SSD and the media layer below it;
|
||||||
|
# 2. bad input is refused by name instead of mounting something surprising;
|
||||||
|
# 3. verify refuses a view that is not mounted (so nothing trusts a directory);
|
||||||
|
# 4. both media units bind the VIEW and refuse to start without it;
|
||||||
|
# 5. only those two units touch the view — the LLM and library tenants keep the
|
||||||
|
# raw volume, so the blast radius is two containers;
|
||||||
|
# 6. the deployer brings the view up BEFORE it restarts either server;
|
||||||
|
# 7. the planner routes a change to these files to both media services.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
VIEW="$REPO_ROOT/scripts/media-view.sh"
|
||||||
|
JELLY="$REPO_ROOT/dankserver/jellyfin/quadlet/jellyfin.container"
|
||||||
|
PLEX="$REPO_ROOT/dankserver/plex/quadlet/plex.container"
|
||||||
|
MOUNT=/var/lib/media-meta/view
|
||||||
|
UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
|
||||||
|
|
||||||
|
@test "the plan names an overlay with the spindle below and the SSD above" {
|
||||||
|
run bash "$VIEW" plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# Lower = the array; upper/work = the NVMe. A user asked for metadata to be
|
||||||
|
# SSD-resident, so a spindle upper layer would defeat the whole mechanism.
|
||||||
|
[ "${lines[0]}" = "lowerdir=/mnt/entropy" ]
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qx 'upperdir=/var/lib/media-meta/upper'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -qx 'workdir=/var/lib/media-meta/work'
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -Fqx "mountpoint=$MOUNT"
|
||||||
|
# systemd names a mount unit after its escaped path, backslash and all.
|
||||||
|
printf '%s\n' "${lines[@]}" | grep -Fqx \
|
||||||
|
'unit=/etc/systemd/system/var-lib-media\x2dmeta-view.mount'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "install refuses a caller who cannot write the unit directory or mount" {
|
||||||
|
# Half an install is worse than none: the unit would exist without the
|
||||||
|
# directories behind it. Refuse by name before anything is created.
|
||||||
|
if [ "$(id -u)" -eq 0 ]; then
|
||||||
|
skip "this seam is running as root"
|
||||||
|
fi
|
||||||
|
run bash "$VIEW" install
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"root"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "verify refuses a view that is not mounted" {
|
||||||
|
# Every consumer trusts the mount point to BE the library. On a host where it
|
||||||
|
# is a plain empty directory, verify must fail loudly: that is the difference
|
||||||
|
# between a scan that finds nothing and a scan that empties a library DB.
|
||||||
|
if mountpoint -q "$MOUNT"; then
|
||||||
|
skip "$MOUNT is a real mountpoint on this host"
|
||||||
|
fi
|
||||||
|
run bash "$VIEW" verify
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"$MOUNT"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "both media servers bind the view and refuse to start without it" {
|
||||||
|
for unit in "$JELLY" "$PLEX"; do
|
||||||
|
grep -q "^Volume=$MOUNT:" "$unit" || return 1
|
||||||
|
grep -q "^UnitConfigItem=ConditionPathIsMountPoint=$MOUNT$" "$unit" || return 1
|
||||||
|
# The raw array must no longer be a container source: that bind is what
|
||||||
|
# made sidecar sharing impossible (read-only) and metadata slow (seeks).
|
||||||
|
! grep -q '^Volume=/mnt/entropy' "$unit"
|
||||||
|
done
|
||||||
|
# Jellyfin is the writer, so it needs the view writable; Plex only reads the
|
||||||
|
# sidecars, and keeping its bind read-only costs nothing.
|
||||||
|
grep -q "^Volume=$MOUNT:/data$" "$JELLY"
|
||||||
|
grep -q "^Volume=$MOUNT:/mnt/entropy:ro$" "$PLEX"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "only the two media units reference the view" {
|
||||||
|
run grep -rl --include='*.container' "$MOUNT" "$REPO_ROOT"/dankserver \
|
||||||
|
"$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro 2>/dev/null
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(printf '%s\n' "$output" | wc -l)" -eq 2 ]
|
||||||
|
printf '%s\n' "$output" | grep -q 'jellyfin/quadlet/jellyfin.container'
|
||||||
|
printf '%s\n' "$output" | grep -q 'plex/quadlet/plex.container'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the mount unit is an overlay whose upper layer lives on the SSD" {
|
||||||
|
[ -f "$UNIT" ]
|
||||||
|
run grep -E '^Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work' "$UNIT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# The unit must mount exactly where the two containers bind, or the guard and
|
||||||
|
# the bind disagree and a media server browses an empty directory.
|
||||||
|
run grep -Fqx "Where=$MOUNT" "$UNIT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# It must wait for the array to be unlocked: mounting an overlay whose lower
|
||||||
|
# dir is an empty /mnt/entropy would present an empty library, not an error.
|
||||||
|
run grep -E '^RequiresMountsFor=/mnt/entropy$' "$UNIT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the deployer raises the view before it restarts either media server" {
|
||||||
|
for svc in jellyfin plex; do
|
||||||
|
body="$(sed -n "/^deploy_${svc}()/,/^\}/p" "$REPO_ROOT/scripts/deploy-remote.sh")"
|
||||||
|
[ -n "$body" ]
|
||||||
|
up="$(printf '%s\n' "$body" | grep -n 'ensure_media_view' | head -1 | cut -d: -f1)"
|
||||||
|
up_line="${up:-999}"
|
||||||
|
restart="$(printf '%s\n' "$body" | grep -n "systemctl --user restart $svc.service" | head -1 | cut -d: -f1)"
|
||||||
|
restart_line="${restart:-0}"
|
||||||
|
[ "$up_line" -lt "$restart_line" ]
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the planner routes the view files to both media services" {
|
||||||
|
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
||||||
|
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$output" = "jellyfin
|
||||||
|
plex" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a docs-only change still plans nothing for the view" {
|
||||||
|
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
|
||||||
|
"serverAppConfigs-docs/Dankserver/Media Metadata View.md"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "reap names the metadata whose media has moved and touches nothing" {
|
||||||
|
# A relayout moves media on the RAW path; the sidecars the view already wrote
|
||||||
|
# stay behind at the old coordinates. Left alone they become folders a media
|
||||||
|
# server browses and finds nothing in.
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
|
||||||
|
"$tmp/meta/upper/TVShows/Big Little Lies/Season 01" "$tmp/meta/work" \
|
||||||
|
"$tmp/array/Movies/Keeper (2020)" "$tmp/array/TVShows"
|
||||||
|
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
|
||||||
|
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
|
||||||
|
: > "$tmp/meta/upper/TVShows/Big Little Lies/Season 01/show.nfo"
|
||||||
|
: > "$tmp/array/Movies/Keeper (2020)/Keeper (2020).mkv"
|
||||||
|
|
||||||
|
run bash "$VIEW" reap --upper-base "$tmp/meta" --lower "$tmp/array"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "$output" | grep -qxF 'Movies/Ghost (2020)'
|
||||||
|
# The series folder is still below, so the ghost is the season inside it — and
|
||||||
|
# it is reported at the level that is actually missing, not deeper.
|
||||||
|
printf '%s\n' "$output" | grep -qxF 'TVShows/Big Little Lies'
|
||||||
|
! printf '%s\n' "$output" | grep -q 'Keeper'
|
||||||
|
[ -f "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "reap --apply moves ghosts to the SSD trash and keeps live metadata" {
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
|
||||||
|
"$tmp/array/Movies/Keeper (2020)"
|
||||||
|
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
|
||||||
|
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
|
||||||
|
|
||||||
|
run bash "$VIEW" reap --apply --upper-base "$tmp/meta" --lower "$tmp/array"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ ! -e "$tmp/meta/upper/Movies/Ghost (2020)" ]
|
||||||
|
[ -f "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo" ]
|
||||||
|
# Recoverable, not deleted: the item lands in a dated trash under the SSD base.
|
||||||
|
[ "$(find "$tmp/meta/.AGENTTRASH" -name movie.nfo | wc -l)" -eq 1 ]
|
||||||
|
}
|
||||||
Executable
+652
@@ -0,0 +1,652 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: the movie-library relayout — scripts/media-census.py (the read-only
|
||||||
|
# census), scripts/media-plan.py (the plan) and scripts/media-apply.py (the only
|
||||||
|
# thing that moves files).
|
||||||
|
#
|
||||||
|
# The library is 8k files on a 100 %-full array, so the two things this seam
|
||||||
|
# exists to catch are the ones that cost a re-run of a 20-minute census: a file
|
||||||
|
# that ends up in the WRONG movie folder, and a move that destroys something.
|
||||||
|
# Everything below runs on a synthetic library in the test's own tmpdir: no
|
||||||
|
# ffprobe, no network, no server. The probe step is the one part a fixture
|
||||||
|
# cannot run (it opens real containers), so the tests feed the planner the
|
||||||
|
# probed TSV in the column order scripts/media-census.py exports — which pins
|
||||||
|
# that contract as a side effect.
|
||||||
|
#
|
||||||
|
# The plan's own gate lines (unplanned, left-unmoved, double-claim, occupied)
|
||||||
|
# are asserted as output, because they are how the operator learns the plan is
|
||||||
|
# trustworthy before pointing the executor at 24 TB.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
PLAN="$REPO_ROOT/scripts/media-plan.py"
|
||||||
|
CENSUS="$REPO_ROOT/scripts/media-census.py"
|
||||||
|
APPLY="$REPO_ROOT/scripts/media-apply.py"
|
||||||
|
TV="$REPO_ROOT/scripts/media-tv.py"
|
||||||
|
|
||||||
|
# probe_row FILE DUR [RES] [AUDIO] [SUBS] — one probed-TSV row: the twelve
|
||||||
|
# columns media-census.py probe writes (bytes, path, dur, vcodec, res, fps,
|
||||||
|
# vbit, pix, audio, subs, spare, fmt).
|
||||||
|
probe_row() {
|
||||||
|
local p="$1" dur="$2" res="${3:-1920x1080}" aud="${4:-aac+aac}" subs="${5:-0}"
|
||||||
|
printf '%s\t%s\t%s\thevc\t%s\t23.976\t5000000\t8bit\t%s\t%s\t0\tmatroska,webm\n' \
|
||||||
|
"$(stat -c %s "$p")" "$p" "$dur" "$res" "$aud" "$subs" >> "$PROBED"
|
||||||
|
}
|
||||||
|
|
||||||
|
# plex_row FILE TITLE YEAR — one Plex metadata row; the planner reads these
|
||||||
|
# three columns and treats a row without a plausible year as no match at all.
|
||||||
|
plex_row() {
|
||||||
|
printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$PLEX"
|
||||||
|
}
|
||||||
|
|
||||||
|
# at PATH SIZE — a file with deterministic content, big enough to matter.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
LIB="$BATS_TEST_TMPDIR/Movies"
|
||||||
|
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
|
||||||
|
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
|
||||||
|
PLEX="$BATS_TEST_TMPDIR/plex.tsv"
|
||||||
|
OUT="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
|
||||||
|
TVLIB="$BATS_TEST_TMPDIR/TVShows"
|
||||||
|
TVPLAN="$BATS_TEST_TMPDIR/tv.tsv"
|
||||||
|
TVJ="$BATS_TEST_TMPDIR/tv.jsonl"
|
||||||
|
: > "$PROBED"
|
||||||
|
printf 'file\tleaf_title\tleaf_year\n' > "$PLEX"
|
||||||
|
|
||||||
|
# Two films that share a box set and a word, plus their bonus material:
|
||||||
|
# the shape that made one film's featurettes land in another's folder.
|
||||||
|
at "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4000
|
||||||
|
at "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 3300000
|
||||||
|
at "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 900000
|
||||||
|
at "$LIB/Alien (1979)/subs/3_Eng.srt" 40000
|
||||||
|
at "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 5000000
|
||||||
|
at "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 2400000
|
||||||
|
# A long extra: over 45 minutes, so only the folder may call it bonus.
|
||||||
|
at "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 2600000
|
||||||
|
# A misspelled bonus folder: no bonus vocabulary, so only the folder that
|
||||||
|
# holds the feature can decide that this is not a film.
|
||||||
|
at "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2500000
|
||||||
|
# A Plex title that carries "/" and a Plex row that was never matched.
|
||||||
|
at "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 4200000
|
||||||
|
at "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 3100000
|
||||||
|
# Junk the layout has to park somewhere other than a movie folder.
|
||||||
|
at "$LIB/.DS_Store" 6148
|
||||||
|
at "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv" 220
|
||||||
|
# A bonus file whose name is a number: never a movie folder named "1".
|
||||||
|
at "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 2700000
|
||||||
|
# A TV box set sitting in the movie library. This one names each file
|
||||||
|
# "Show - SxxEyy - Episode Title", so the episode title comes AFTER the
|
||||||
|
# marker; that convention is what made 124 episodes of one series look like
|
||||||
|
# 124 different films, each under its own threshold.
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
|
||||||
|
done
|
||||||
|
# The same trouble named the other way round, with no episode title at all.
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3000000
|
||||||
|
done
|
||||||
|
# A film whose own name carries a bare "Part 2": one episode-shaped token is
|
||||||
|
# not a series, and it must not be dragged into the TV pile.
|
||||||
|
at "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 9000000
|
||||||
|
|
||||||
|
probe_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4700
|
||||||
|
probe_row "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 2700
|
||||||
|
probe_row "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 30
|
||||||
|
probe_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 4800
|
||||||
|
probe_row "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 1200
|
||||||
|
probe_row "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 3000
|
||||||
|
probe_row "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2600
|
||||||
|
probe_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 5700
|
||||||
|
probe_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 6300
|
||||||
|
probe_row "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 5220
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
probe_row "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 2600
|
||||||
|
probe_row "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3300
|
||||||
|
done
|
||||||
|
probe_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 6200
|
||||||
|
plex_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" "Alien" "1979"
|
||||||
|
plex_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" "Marnie" "1964"
|
||||||
|
plex_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" "V/H/S/2" "2013"
|
||||||
|
# An unmatched Plex row: the leaf title echoes the file and the year is 0.
|
||||||
|
plex_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" "AMERICAN POP" "0"
|
||||||
|
plex_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" \
|
||||||
|
"The Godfather Part 2" "1974"
|
||||||
|
|
||||||
|
export MEDIA_ROOT="$LIB"
|
||||||
|
run python3 "$CENSUS" enumerate --root "$LIB" --out "$ENUM"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# plan — run the planner over the fixture, keeping its plan TSV, its exception
|
||||||
|
# report and the gate lines it prints (that stdout is the operator's evidence
|
||||||
|
# that the plan accounts for every file, so the tests read it too).
|
||||||
|
plan() {
|
||||||
|
PLANOUT="$BATS_TEST_TMPDIR/plan.out"
|
||||||
|
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --probed "$PROBED" \
|
||||||
|
--plex "$PLEX" --out "$OUT" --exceptions "$EXC"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "$output" > "$PLANOUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
# dst_of SRC — the destination the plan gives one file.
|
||||||
|
dst_of() {
|
||||||
|
python3 -c '
|
||||||
|
import sys
|
||||||
|
for line in open(sys.argv[1]):
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if len(f) > 2 and f[1] == sys.argv[2]:
|
||||||
|
print(f[2]); break
|
||||||
|
' "$OUT" "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# tv_plan — the TV mover over the fixture library; its stdout is the operator's
|
||||||
|
# census of what it called a series and what it refused to touch.
|
||||||
|
tv_plan() {
|
||||||
|
run python3 "$TV" --root "$LIB" --library "$TVLIB" --out "$TVPLAN"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
printf '%s\n' "$output" > "$BATS_TEST_TMPDIR/tv.out"
|
||||||
|
}
|
||||||
|
|
||||||
|
# tv_apply — that plan through the same executor. The destination is a sibling
|
||||||
|
# share, so the plan is applied at the root that holds both libraries.
|
||||||
|
tv_apply() {
|
||||||
|
run python3 "$APPLY" apply --plan "$TVPLAN" --root "$BATS_TEST_TMPDIR" \
|
||||||
|
--journal "$TVJ" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# tv_dst_of SRC — the destination the TV plan gives one file, empty if none.
|
||||||
|
tv_dst_of() {
|
||||||
|
python3 -c '
|
||||||
|
import sys
|
||||||
|
for line in open(sys.argv[1]):
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if len(f) > 2 and f[1] == sys.argv[2]:
|
||||||
|
print(f[2]); break
|
||||||
|
' "$TVPLAN" "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a film lands in Title (year)/Title (year).mkv, named by Plex" {
|
||||||
|
plan
|
||||||
|
[ "$(dst_of "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv")" \
|
||||||
|
= "$LIB/Alien (1979)/Alien (1979).mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a Plex title that carries a slash becomes one folder, not a nested tree" {
|
||||||
|
plan
|
||||||
|
local d; d="$(dst_of "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv")"
|
||||||
|
[ "$(dirname "$d")" = "$LIB/V-H-S-2 (2013)" ]
|
||||||
|
[ ! -e "$LIB/V" ]
|
||||||
|
# the folder must be ONE mkdir: an un-sanitized name would make the
|
||||||
|
# executor build Movies/V/H/S/2 (2013) instead
|
||||||
|
run grep -F "mkdir" "$OUT"
|
||||||
|
[[ "$output" == *"$(printf '%s' "$LIB/V-H-S-2 (2013)")"* ]]
|
||||||
|
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
|
||||||
|
[[ "$output" != *"$LIB/V/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an unmatched Plex row never names a folder with a zero year" {
|
||||||
|
plan
|
||||||
|
run awk -F'\t' '{print $3}' "$OUT"
|
||||||
|
[[ "$output" != *"(0)"* ]]
|
||||||
|
# the filename's own year carries the folder, the folder and the file share
|
||||||
|
# the name (Jellyfin's rule), and the film sits at the library root rather
|
||||||
|
# than in an unsorted pile
|
||||||
|
local d; d="$(dst_of "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB" ]
|
||||||
|
[[ "$(basename "$(dirname "$d")")" == *" (2023)" ]]
|
||||||
|
[ "$(basename "$d")" = "$(basename "$(dirname "$d")").mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a bonus folder keeps its featurettes inside its own film, not a neighbour's" {
|
||||||
|
plan
|
||||||
|
local d
|
||||||
|
d="$(dst_of "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
|
||||||
|
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
|
||||||
|
d="$(dst_of "$LIB/Alien (1979)/Extras/Extra - Logo.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
|
||||||
|
# the feature-length extra is bonus too: it sits in a bonus folder and no
|
||||||
|
# metadata matched it, so it may not become a movie of its own
|
||||||
|
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a folder inside a movie folder is bonus even when its name is misspelled" {
|
||||||
|
plan
|
||||||
|
local d; d="$(dst_of "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a subtitle follows the film of the folder it sits in" {
|
||||||
|
plan
|
||||||
|
local d; d="$(dst_of "$LIB/Alien (1979)/subs/3_Eng.srt")"
|
||||||
|
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "macOS sidecars are parked in one pile, never inside a movie folder" {
|
||||||
|
plan
|
||||||
|
local d; d="$(dst_of "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv")"
|
||||||
|
[[ "$(dirname "$d")" == "$LIB"/_* ]]
|
||||||
|
d="$(dst_of "$LIB/.DS_Store")"
|
||||||
|
[[ "$(dirname "$d")" == "$LIB"/_* ]]
|
||||||
|
grep -q "AppleDouble" "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file whose name is a number becomes a review line, not a folder named 1" {
|
||||||
|
plan
|
||||||
|
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
|
||||||
|
[[ "$output" != "$LIB/1"$'\n'* ]] && [[ "$output" != "$LIB/1" ]]
|
||||||
|
[[ "$output" != *"$LIB/1/"* ]]
|
||||||
|
grep -q "UNNAMED" "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a box set in the movie library is quarantined, not filed as movies" {
|
||||||
|
plan
|
||||||
|
local d
|
||||||
|
d="$(dst_of "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv")"
|
||||||
|
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ monk* ]]
|
||||||
|
d="$(dst_of "$LIB/Big Little Lies Season 1/Big Little Lies S01E01.mp4")"
|
||||||
|
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ big\ little\ lies* ]]
|
||||||
|
# one series is one pile, whatever its episodes are called
|
||||||
|
run python3 -c '
|
||||||
|
import re, sys
|
||||||
|
pile = {}
|
||||||
|
for line in open(sys.argv[1]):
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if len(f) > 2 and f[0] == "move":
|
||||||
|
m = re.search(r"/_TV needs a TV library - ([^/]*)/", f[2] + "/")
|
||||||
|
if m:
|
||||||
|
pile[m.group(1)] = pile.get(m.group(1), 0) + 1
|
||||||
|
for k in sorted(pile):
|
||||||
|
print(f"{k}={pile[k]}")
|
||||||
|
' "$OUT"
|
||||||
|
[ "${#lines[@]}" -eq 2 ]
|
||||||
|
[[ "${lines[0]}" == big\ little\ lies*=5 ]]
|
||||||
|
[[ "${lines[1]}" == monk*=5 ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a film named Part 2 is not called a series" {
|
||||||
|
plan
|
||||||
|
[ "$(dst_of "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv")" \
|
||||||
|
= "$LIB/The Godfather Part 2 (1974)/The Godfather Part 2 (1974).mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the TV mover files a box set into the TV library in Jellyfin shape" {
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
|
||||||
|
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
|
||||||
|
[ ! -e "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
|
||||||
|
# films are not the TV mover's business
|
||||||
|
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the TV mover leaves a film whose name only looks like an episode" {
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
local gf="$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv"
|
||||||
|
[ "$(tv_dst_of "$gf")" = "" ]
|
||||||
|
[ -e "$gf" ]
|
||||||
|
grep -q "Godfather.*not a series" "$BATS_TEST_TMPDIR/tv.out"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the TV mover keeps an episode's own subtitle with the episode" {
|
||||||
|
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.srt" 40000
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.srt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a second TV move finds nothing left to move" {
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
tv_plan
|
||||||
|
grep -q "0 moves" "$BATS_TEST_TMPDIR/tv.out"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a show whose own name carries the year is not given it twice" {
|
||||||
|
local d="$LIB/Leah Remini Scientology and the Aftermath (2016)/Season 1"
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$d/Leah Remini Scientology and the Aftermath (2016) - S01E$e - Episode.mkv" 3000000
|
||||||
|
done
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
[ -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016)/Season 01/Leah Remini Scientology and the Aftermath (2016) - S01E01 - Episode.mkv" ]
|
||||||
|
[ ! -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016) (2016)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an all-lowercase series name is written the way the library writes names" {
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$LIB/downton abbey 2010/Season 1/downton abbey - s01e$e - episode title.mkv" 3000000
|
||||||
|
done
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
# the FOLDER is titled; the file keeps the name the library already holds
|
||||||
|
[ -e "$TVLIB/Downton Abbey (2010)/Season 01/downton abbey - s01e01 - episode title.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the mover leaves an episode the TV library already holds" {
|
||||||
|
local src="$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv"
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$TVLIB/Monk (2002)/Season 01/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
|
||||||
|
done
|
||||||
|
tv_plan
|
||||||
|
[ -z "$(tv_dst_of "$src")" ]
|
||||||
|
[ -e "$src" ]
|
||||||
|
grep -q "Monk.*already in" "$BATS_TEST_TMPDIR/tv.out"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an episode the TV library holds in another encode is still moved" {
|
||||||
|
for e in 01 02 03 04 05; do
|
||||||
|
at "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E$e 1080p.mp4" 4000000
|
||||||
|
done
|
||||||
|
tv_plan
|
||||||
|
tv_apply
|
||||||
|
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
|
||||||
|
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01 1080p.mp4" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "every file in the census is placed or kept — nothing is dropped" {
|
||||||
|
plan
|
||||||
|
python3 - "$ENUM" "$OUT" <<'PY'
|
||||||
|
import sys
|
||||||
|
enum = {l.rstrip("\n").split("\t")[2] for l in open(sys.argv[1])}
|
||||||
|
placed = set()
|
||||||
|
for line in open(sys.argv[2]):
|
||||||
|
f = line.rstrip("\n").split("\t")
|
||||||
|
if f[0] in ("move", "keep"):
|
||||||
|
placed.add(f[1])
|
||||||
|
missing = enum - placed
|
||||||
|
assert not missing, f"{len(missing)} census files with no plan row: {sorted(missing)[:3]}"
|
||||||
|
PY
|
||||||
|
grep -q "left-unmoved=0" "$PLANOUT"
|
||||||
|
grep -q "double-claim=0" "$PLANOUT"
|
||||||
|
grep -q "occupied=0" "$PLANOUT"
|
||||||
|
grep -q "bad-names=0" "$PLANOUT"
|
||||||
|
grep -q "unplanned=0" "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- executor
|
||||||
|
|
||||||
|
# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
|
||||||
|
apply_plan() {
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
# `/` is always a mountpoint, so the overlay note behaves as it does on a
|
||||||
|
# host that runs the metadata view.
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" \
|
||||||
|
--journal "$JOURNAL" --metadata-view / --apply
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a run without --apply says what it would do and touches nothing" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"--apply"* ]]
|
||||||
|
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
|
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
|
||||||
|
# a dry run may not even build the folders: an operator must be able to
|
||||||
|
# run it against the live library with no --apply and change nothing
|
||||||
|
[ ! -d "$LIB/V-H-S-2 (2013)" ]
|
||||||
|
[ ! -e "$JOURNAL" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "apply moves every planned file, keeps its bytes, and journals each move" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
|
||||||
|
[ ! -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
|
local planned moved
|
||||||
|
planned=$(awk -F'\t' '$1=="move"' "$OUT" | wc -l)
|
||||||
|
moved=$(python3 -c 'import json,sys
|
||||||
|
print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "$JOURNAL")
|
||||||
|
[ "$moved" -eq "$planned" ]
|
||||||
|
# The media servers' sidecars live in the metadata overlay and did not follow
|
||||||
|
# these moves, so the run names that cleanup at the moment it is needed.
|
||||||
|
[[ "$output" == *"media-view.sh reap"* ]]
|
||||||
|
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# The overlay note is only true where the metadata view is really mounted. On a
|
||||||
|
# host whose media servers bind the array directly (dankserver today: both
|
||||||
|
# quadlets mount /mnt/entropy, /var/lib/media-meta does not exist) the sidecars
|
||||||
|
# moved with the media, and "next: run reap" is an order that cannot be carried
|
||||||
|
# out. --metadata-view is how the caller names that path; the default is the
|
||||||
|
# host's view path, and the note rides only when it is a mountpoint.
|
||||||
|
@test "an apply with no mounted metadata view does not order a reap" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"moved"* ]]
|
||||||
|
[[ "$output" != *"media-view.sh reap"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a rollback with no mounted metadata view does not order a reap" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run python3 "$APPLY" rollback --journal "$JOURNAL" --root "$LIB" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"rolled back"* ]]
|
||||||
|
[[ "$output" != *"media-view.sh reap"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a target already holding a different file stops the whole run" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
at "$LIB/Alien (1979)/Alien (1979).mkv" 7
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"occupied"* ]]
|
||||||
|
# nothing moved at all: the other films are still where the census saw them
|
||||||
|
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
|
||||||
|
[ ! -e "$JOURNAL" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a plan row whose source has gone stops the run instead of inventing a move" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
rm "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv"
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"missing"* ]]
|
||||||
|
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a run interrupted between the journal write and the move finishes the job" {
|
||||||
|
plan
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
# The journal is written before each move, so an interruption leaves two
|
||||||
|
# states and only two: an intent whose move happened, and an intent whose
|
||||||
|
# move never ran. Both are simulated here by hand.
|
||||||
|
python3 - "$OUT" "$JOURNAL" <<'PY'
|
||||||
|
import json, os, sys
|
||||||
|
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
|
||||||
|
if r[0] == "move"][:3]
|
||||||
|
with open(sys.argv[2], "w") as fh:
|
||||||
|
for i, r in enumerate(moves):
|
||||||
|
fh.write(json.dumps({"op": "move", "src": r[1], "dst": r[2],
|
||||||
|
"bytes": os.path.getsize(r[1])}) + "\n")
|
||||||
|
if i < 2: # the first two moves actually ran
|
||||||
|
os.makedirs(os.path.dirname(r[2]), exist_ok=True)
|
||||||
|
os.rename(r[1], r[2])
|
||||||
|
PY
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
python3 - "$OUT" "$JOURNAL" <<'PY'
|
||||||
|
import json, os, sys
|
||||||
|
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
|
||||||
|
if r[0] == "move"]
|
||||||
|
journal = [json.loads(l) for l in open(sys.argv[2])]
|
||||||
|
seen = [j["dst"] for j in journal]
|
||||||
|
assert len(seen) == len(set(seen)), "a destination was journalled twice"
|
||||||
|
for r in moves:
|
||||||
|
assert os.path.exists(r[2]) and not os.path.exists(r[1]), r[2]
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "rollback puts every file back where the census found it" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
|
||||||
|
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
|
||||||
|
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
|
||||||
|
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
|
||||||
|
# the folders the run created are empty afterwards and must not litter the
|
||||||
|
# library: this one never existed before the plan ran
|
||||||
|
[ -d "$LIB/VHS 2 (2013)" ]
|
||||||
|
[ ! -e "$LIB/V-H-S-2 (2013)" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "rollback refuses when a moved file has gone missing" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
rm "$LIB/Alien (1979)/Alien (1979).mkv"
|
||||||
|
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"missing"* ]]
|
||||||
|
# the refusal is all-or-nothing: another film did not move back
|
||||||
|
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- sweep
|
||||||
|
#
|
||||||
|
# A relayout leaves the release folders it emptied standing at the library root:
|
||||||
|
# 1675 of them after the live run, each holding nothing but empty subfolders. The
|
||||||
|
# sweep pass removes only folders THIS RUN emptied — proven from the journal,
|
||||||
|
# never from "it looks empty" — so a folder someone made by hand is untouchable,
|
||||||
|
# and every removal rides the same rollback as the moves.
|
||||||
|
|
||||||
|
@test "sweep removes only the folders this run emptied" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
mkdir -p "$LIB/A Empty Placeholder" # someone's folder, never a move source
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"swept"* ]]
|
||||||
|
[ ! -e "$LIB/AMERICAN POP" ]
|
||||||
|
[ ! -e "$LIB/VHS 2 (2013)" ]
|
||||||
|
[ ! -e "$LIB/Alien (1979)/Featurettes" ]
|
||||||
|
[ ! -e "$LIB/Alien (1979)/subs" ]
|
||||||
|
[ ! -e "$LIB/Marnie (1964)/Exrtas" ]
|
||||||
|
# a folder the run never moved a file out of survives, empty or not
|
||||||
|
[ -d "$LIB/A Empty Placeholder" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a swept folder carries the empty junk folders inside it" {
|
||||||
|
mkdir -p "$LIB/AMERICAN POP/Screens" # screenshots, never a move source
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"0 left in place"* ]]
|
||||||
|
[ ! -e "$LIB/AMERICAN POP/Screens" ]
|
||||||
|
[ ! -e "$LIB/AMERICAN POP" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a sweep whose rmdir failed tries those folders again" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
mkdir -p "$LIB/AMERICAN POP/Screens"
|
||||||
|
chmod 500 "$LIB/AMERICAN POP" # readable, but nothing can be removed
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"left in place"* ]]
|
||||||
|
[ -e "$LIB/AMERICAN POP/Screens" ]
|
||||||
|
chmod 700 "$LIB/AMERICAN POP" # the operator cleared the blocker
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" != *"left in place: "* ]]
|
||||||
|
[ ! -e "$LIB/AMERICAN POP" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "sweep without --apply reports the husks and changes nothing" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
local before jlines
|
||||||
|
before=$(find "$LIB" -type d | wc -l)
|
||||||
|
jlines=$(wc -l < "$JOURNAL")
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"--apply"* ]]
|
||||||
|
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
|
||||||
|
[ "$(wc -l < "$JOURNAL")" -eq "$jlines" ]
|
||||||
|
[ -e "$LIB/AMERICAN POP" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a folder that still holds content is never swept" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# the film folder, the folder the run created, and the piles all stay
|
||||||
|
[ -d "$LIB/Marnie (1964)" ]
|
||||||
|
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
|
||||||
|
[ -d "$LIB/V-H-S-2 (2013)" ]
|
||||||
|
[ -d "$LIB/_unsorted bonus/extras" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "rollback after a sweep brings the folders and the files back" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ ! -e "$LIB/AMERICAN POP" ]
|
||||||
|
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" ]
|
||||||
|
[ -e "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" ]
|
||||||
|
[ -e "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" ]
|
||||||
|
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a second sweep has nothing left to do and says so" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
local before
|
||||||
|
before=$(find "$LIB" -type d | wc -l)
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"swept: 0"* ]]
|
||||||
|
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "sweep never follows a symlink and never removes the folder it guards" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
mkdir -p "$LIB/Elsewhere"
|
||||||
|
at "$LIB/Elsewhere/keep.txt" 10
|
||||||
|
ln -s "$LIB/Elsewhere" "$LIB/AMERICAN POP/pointer"
|
||||||
|
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# a link is content, not nothing: the husk it sits in survives untouched
|
||||||
|
[ -d "$LIB/AMERICAN POP" ]
|
||||||
|
[ -L "$LIB/AMERICAN POP/pointer" ]
|
||||||
|
[ -e "$LIB/Elsewhere/keep.txt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "sweep refuses a journal that belongs to another library" {
|
||||||
|
plan
|
||||||
|
apply_plan
|
||||||
|
mkdir -p "$BATS_TEST_TMPDIR/Other"
|
||||||
|
run python3 "$APPLY" sweep --root "$BATS_TEST_TMPDIR/Other" \
|
||||||
|
--journal "$JOURNAL" --apply
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"outside"* ]]
|
||||||
|
[ -d "$LIB/AMERICAN POP" ]
|
||||||
|
}
|
||||||
Executable
+459
@@ -0,0 +1,459 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: the TV-library relayout — scripts/media-census.py jellyfin (the metadata
|
||||||
|
# authority export) and scripts/media-shows.py (the plan).
|
||||||
|
#
|
||||||
|
# The TV library is 52k files written in five different release layouts, and the
|
||||||
|
# one thing that makes a rename of that size survivable is that the plan is
|
||||||
|
# derived from what a metadata provider already matched, not from a guess at the
|
||||||
|
# file name. So every test here states which authority a file had — name only,
|
||||||
|
# provider only, both, neither — and asserts where the PLAN files it, never how
|
||||||
|
# the planner decided. A fixture Jellyfin database stands in for the live one and
|
||||||
|
# the name caches are input files, so no test touches the network.
|
||||||
|
#
|
||||||
|
# The target contract is tvdb-grade naming:
|
||||||
|
# <Series> (<year>)/Season NN/<Series> (<year>) - SxxEyy - <Episode Title>.<ext>
|
||||||
|
# tvdb is the IDENTITY authority: a provider name a server derived from a folder
|
||||||
|
# is not a series name, so a polluted name is undone through the series' tvdb id.
|
||||||
|
# The review pile parks OUTSIDE the library — a parked folder inside it indexes
|
||||||
|
# as one junk show, and that show's rows are then mistaken for evidence.
|
||||||
|
#
|
||||||
|
# The two outcomes that cost a re-run of a 50k-file plan are a file filed as an
|
||||||
|
# episode that is not one (it steals an episode number from its real owner) and a
|
||||||
|
# move that lands on an existing file. Everything below is written to catch
|
||||||
|
# exactly those, plus the gate that matters most: the plan must be replayable by
|
||||||
|
# scripts/media-apply.py, whose preflight is the fleet's only overwrite guard.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
PLAN="$REPO_ROOT/scripts/media-shows.py"
|
||||||
|
CENSUS="$REPO_ROOT/scripts/media-census.py"
|
||||||
|
APPLY="$REPO_ROOT/scripts/media-apply.py"
|
||||||
|
|
||||||
|
# at PATH SIZE — a file with deterministic content.
|
||||||
|
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||||
|
|
||||||
|
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] [TMDB] [TITLE] [TVDB]
|
||||||
|
# — one row of the authority export scripts/media-census.py jellyfin writes.
|
||||||
|
title_row() {
|
||||||
|
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" \
|
||||||
|
"${5:-}" "${6:-}" "${7:-}" "${8:-}" "${9:-}" >> "$TITLES"
|
||||||
|
}
|
||||||
|
|
||||||
|
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
|
||||||
|
dst() {
|
||||||
|
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
LIB="$BATS_TEST_TMPDIR/TVShows"
|
||||||
|
PARK="$LIB/_unsorted"
|
||||||
|
OUTSIDE="$BATS_TEST_TMPDIR/park"
|
||||||
|
MOVIES="$BATS_TEST_TMPDIR/Movies"
|
||||||
|
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
|
||||||
|
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
|
||||||
|
TITLES="$BATS_TEST_TMPDIR/titles.tsv"
|
||||||
|
CACHE="$BATS_TEST_TMPDIR/tmdb.json"
|
||||||
|
OUT="$BATS_TEST_TMPDIR/plan.tsv"
|
||||||
|
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
|
||||||
|
: > "$PROBED"
|
||||||
|
|
||||||
|
SEASON="$LIB/The Examples (2010) Season 1 S01 (1080p BluRay x265 Silence)"
|
||||||
|
# An episode its own name states, with a subtitle, a season image and a
|
||||||
|
# trailer: four things that have to end up in four different places.
|
||||||
|
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" 9000
|
||||||
|
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt" 200
|
||||||
|
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt" 250
|
||||||
|
# A scene pack buries its subtitles two levels down, in Subs/<release-name>/.
|
||||||
|
at "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt" 220
|
||||||
|
at "$SEASON/folder.jpg" 300
|
||||||
|
at "$SEASON/Trailers/teaser.mkv" 1000
|
||||||
|
at "$SEASON/tvshow.nfo" 400
|
||||||
|
# An anime pack: no episode marker anywhere in its name, numbered by the
|
||||||
|
# provider only. Its year is the one thing no authority states.
|
||||||
|
ANIME="$LIB/Anime Pack [grp] (1080p HEVC)"
|
||||||
|
at "$ANIME/[grp] Anime Pack - 05.mkv" 5000
|
||||||
|
# Two releases claiming one episode, and a file holding two episodes.
|
||||||
|
at "$LIB/Dupes/Show Two S01E01.mkv" 4000
|
||||||
|
at "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" 7000
|
||||||
|
at "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" 7000
|
||||||
|
# A file no authority numbers, and a long file the movie provider knows.
|
||||||
|
at "$LIB/Orphan Tapes/clip001.mp4" 3000
|
||||||
|
at "$LIB/Longform/Some Documentary Special (2015).mkv" 8000
|
||||||
|
# A bonus folder inside a release pack whose own name is not the show's. The
|
||||||
|
# pack's rows name the show; the bonus folder may never invent a series.
|
||||||
|
at "$ANIME/01. Extras/NCOP 01.mkv" 1200
|
||||||
|
# A season folder whose rows are stale: the database numbers every file of this
|
||||||
|
# pack season 16 episode 20, four seasons' worth of them.
|
||||||
|
at "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" 6000
|
||||||
|
at "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" 4000
|
||||||
|
# A box-set folder whose Series item the server named AFTER THE FOLDER: the
|
||||||
|
# row's own name is junk, and only a provider id can undo it.
|
||||||
|
BOX="$LIB/Show Box (2016) Season 1 S01 (1080p WEB-DL)"
|
||||||
|
at "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" 6000
|
||||||
|
# The park inside the library: the server indexed the whole pile as ONE show
|
||||||
|
# named after the folder, so its rows state a series that is not a show.
|
||||||
|
at "$PARK/Park Show S01E01.mkv" 5000
|
||||||
|
at "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" 5200
|
||||||
|
|
||||||
|
# The enum census lists every file; the planner reads that instead of walking
|
||||||
|
# the tree, so the fixture writes the same three columns.
|
||||||
|
python3 - "$LIB" "$ENUM" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
root, out = sys.argv[1], sys.argv[2]
|
||||||
|
with open(out, "w") as fh:
|
||||||
|
for d, _, fs in os.walk(root):
|
||||||
|
for f in fs:
|
||||||
|
p = os.path.join(d, f)
|
||||||
|
fh.write(f"{os.stat(p).st_size}\t{os.path.splitext(f)[1]}\t{p}\n")
|
||||||
|
PY
|
||||||
|
|
||||||
|
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\ttitle\ttvdb\n' > "$TITLES"
|
||||||
|
printf '{}\n' > "$CACHE"
|
||||||
|
title_row "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" \
|
||||||
|
"The Examples" 2010 1 1 "" "" "Pilot"
|
||||||
|
title_row "$ANIME/[grp] Anime Pack - 05.mkv" "Anime Pack" "" 1 5
|
||||||
|
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
|
||||||
|
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
|
||||||
|
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2 "" "" "Double Trouble"
|
||||||
|
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3 "" "" "Loose Ends"
|
||||||
|
title_row "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" \
|
||||||
|
"SNL Pack" "" 16 20
|
||||||
|
title_row "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \
|
||||||
|
"Show Box (2016) Season 1" 2016 1 1 "" "2334" "Pilot" "78850"
|
||||||
|
# The row the server wrote for a parked file: the series is the PARK FOLDER.
|
||||||
|
title_row "$PARK/Park Show S01E01.mkv" "_unsorted" "" 1 1
|
||||||
|
}
|
||||||
|
|
||||||
|
plan() {
|
||||||
|
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
|
||||||
|
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
|
||||||
|
--out "$OUT" --exceptions "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
plan_park() { # the review pile outside the library
|
||||||
|
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
|
||||||
|
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
|
||||||
|
--park "$OUTSIDE" --out "$OUT" --exceptions "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "--tmdb with no key is reported as an unfinished plan, not a finished one" {
|
||||||
|
# A dead provider is invisible where it happens: every 401 is one line on
|
||||||
|
# stderr, and a terminal that scrolled past them shows a plan that reads as
|
||||||
|
# complete while every folder still carries the metadata provider's junk. The
|
||||||
|
# plan is still written — the answers that DID arrive are worth their rename —
|
||||||
|
# so the state has to ride in the summary line and in the review file.
|
||||||
|
# env -u keeps this offline no matter what the developer's shell exports.
|
||||||
|
run env -u TMDB_API_KEY python3 "$PLAN" --root "$LIB" --enum "$ENUM" \
|
||||||
|
--titles "$TITLES" --probed "$PROBED" --tmdb-cache "$CACHE" \
|
||||||
|
--movies "$MOVIES" --tmdb --out "$OUT" --exceptions "$EXC"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
grep -q 'tmdb NOT ASKED (TMDB_API_KEY unset): NOT provider-complete' <<<"$output"
|
||||||
|
grep -q 'WARNING: tmdb NOT ASKED (TMDB_API_KEY unset) — NOT provider-complete' "$EXC"
|
||||||
|
grep -qE '^plan: [1-9][0-9]* moves' <<<"$output"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the census exports the provider's episode numbering as a TSV" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/jellyfin.db"
|
||||||
|
python3 - "$DB" "$SEASON" <<'PY'
|
||||||
|
import os, sqlite3, sys, uuid
|
||||||
|
db, season = sys.argv[1], sys.argv[2]
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
pre = "/data/TVShows/" + os.path.basename(season)
|
||||||
|
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
|
||||||
|
ParentIndexNumber, ProductionYear, ExtraType,
|
||||||
|
IsVirtualItem, Data)""")
|
||||||
|
sid = str(uuid.uuid4())
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
|
||||||
|
pre, None, None, 2010, None, 0,
|
||||||
|
'{"ProviderIds":{"Tmdb":"1234","Tvdb":"78850"}}'))
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(str(uuid.uuid4()), sid, "MediaBrowser.Controller.Entities.TV.Episode",
|
||||||
|
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
|
||||||
|
None, None, 0, '{}'))
|
||||||
|
c.commit()
|
||||||
|
PY
|
||||||
|
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
|
||||||
|
[ "${lines[0]}" = "file series year season episode extra tmdb title tvdb" ]
|
||||||
|
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
|
||||||
|
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234 Pilot 78850$"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the census reads provider ids from the table newer Jellyfin keeps" {
|
||||||
|
DB="$BATS_TEST_TMPDIR/jellyfin12.db"
|
||||||
|
python3 - "$DB" "$SEASON" <<'PY'
|
||||||
|
import os, sqlite3, sys, uuid
|
||||||
|
db, season = sys.argv[1], sys.argv[2]
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
pre = "/data/TVShows/" + os.path.basename(season)
|
||||||
|
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
|
||||||
|
ParentIndexNumber, ProductionYear, ExtraType,
|
||||||
|
IsVirtualItem, Data)""")
|
||||||
|
# Jellyfin 12 moved the ids out of the Data blob and into their own table.
|
||||||
|
c.execute("create table BaseItemProviders (ItemId, ProviderId, ProviderValue)")
|
||||||
|
sid = str(uuid.uuid4())
|
||||||
|
eid = str(uuid.uuid4())
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
|
||||||
|
pre, None, None, 2010, None, 0, '{}'))
|
||||||
|
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||||
|
(eid, sid, "MediaBrowser.Controller.Entities.TV.Episode",
|
||||||
|
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
|
||||||
|
None, None, 0, '{}'))
|
||||||
|
c.execute("insert into BaseItemProviders values (?,?,?)", (sid, "Tvdb", "9999"))
|
||||||
|
c.commit()
|
||||||
|
PY
|
||||||
|
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
|
||||||
|
--out "$BATS_TEST_TMPDIR/exp12.tsv"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
tail -1 "$BATS_TEST_TMPDIR/exp12.tsv" \
|
||||||
|
| grep -q " Pilot 9999$"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an episode its own name states files as the titled standard name" {
|
||||||
|
plan
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv")" \
|
||||||
|
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the tvdb identity names the series folder, not the name a folder left behind" {
|
||||||
|
printf '{"tvdb:series:78850": {"name": "Show Box", "year": "2016"}}\n' > "$CACHE"
|
||||||
|
plan
|
||||||
|
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" \
|
||||||
|
= "$LIB/Show Box (2016)/Season 01/Show Box (2016) - S01E01 - Pilot.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a provider name that already carries the year is not given it twice" {
|
||||||
|
# TVDb answers the reboot as `Acapulco (2021)`. The folder contract is ONE year,
|
||||||
|
# and `Acapulco (2021) (2021)` is a folder neither matcher nor human reuses.
|
||||||
|
printf '{"tvdb:series:78850": {"name": "Show Box (2016)", "year": "2016"}}\n' > "$CACHE"
|
||||||
|
plan
|
||||||
|
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" \
|
||||||
|
= "$LIB/Show Box (2016)/Season 01/Show Box (2016) - S01E01 - Pilot.mkv" ]
|
||||||
|
run grep -F '(2016) (2016)' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a series no authority resolves keeps the provider's name and is reported" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
|
||||||
|
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
|
||||||
|
grep -A9 -i "no tvdb" "$EXC" | grep -q "SNL Pack"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a row whose series is the park folder is not evidence of a show" {
|
||||||
|
plan_park
|
||||||
|
run awk -F '\t' -v p="$PARK/" '$1 == "move" && index($3, p) == 1 {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
run awk -F '\t' '$1 == "move" && $2 ~ /Park Show S01E01/ {print $3}' "$OUT"
|
||||||
|
[[ "$output" == "$OUTSIDE/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the pile's sanitized display name is still the pile, not a series" {
|
||||||
|
# Jellyfin drops the leading underscore when it shows the hidden folder, so the
|
||||||
|
# row says the series is "unsorted" while the file sits in `_unsorted`. Match
|
||||||
|
# only the underscored spelling and the plan renames real files into a visible
|
||||||
|
# folder named after the review pile — which is how 617 of them got there.
|
||||||
|
title_row "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" "unsorted" "" 1 1 \
|
||||||
|
"" "" "Twisted Metal"
|
||||||
|
plan_park
|
||||||
|
run awk -F '\t' -v l="$LIB" \
|
||||||
|
'$1 == "move" && index($3, l "/unsorted/") == 1 {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
run awk -F '\t' '$1 == "move" && $2 ~ /Twisted\.Metal\.S01E01/ {print $3}' "$OUT"
|
||||||
|
[[ "$output" == "$OUTSIDE/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a subtitle follows its episode and keeps its language flag" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt")" \
|
||||||
|
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.en.srt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a subtitle two folders down inside Subs/<release-name>/ still follows its episode" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt")" \
|
||||||
|
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.jpn.srt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a subtitle that carries the container's word in its own name still follows the episode" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt")" \
|
||||||
|
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.forced.srt" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a season image keeps its own name; a series nfo keeps the series folder" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$SEASON/folder.jpg")" = "$LIB/The Examples (2010)/Season 01/folder.jpg" ]
|
||||||
|
[ "$(dst "$SEASON/tvshow.nfo")" = "$LIB/The Examples (2010)/tvshow.nfo" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "bonus material files as an extra, never as an episode" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$SEASON/Trailers/teaser.mkv")" = "$LIB/The Examples (2010)/trailers/teaser.mkv" ]
|
||||||
|
[ -z "$(awk -F '\t' '$3 ~ /Season .*teaser/ {print $3}' "$OUT")" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file only the provider numbers files at the episode it is matched to, title or no title" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$ANIME/[grp] Anime Pack - 05.mkv")" \
|
||||||
|
= "$LIB/Anime Pack/Season 01/Anime Pack - S01E05.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a bonus folder inside a release pack belongs to the pack's show, not to its own name" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$ANIME/01. Extras/NCOP 01.mkv")" = "$LIB/Anime Pack/extras/NCOP 01.mkv" ]
|
||||||
|
[ -z "$(awk -F '\t' '$3 ~ /\/01\. Extras\// {print $3}' "$OUT")" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a Season folder outvotes the stale row its files carry" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
|
||||||
|
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@test "one file holding two episodes keeps both numbers in its name" {
|
||||||
|
plan
|
||||||
|
[ "$(dst "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv")" \
|
||||||
|
= "$LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E02-E03 - Double Trouble.mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two files claiming one episode: one keeps the name, the other parks" {
|
||||||
|
plan
|
||||||
|
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E01 - Two Pilots.mkv " "$OUT")
|
||||||
|
[ "$n" -eq 1 ]
|
||||||
|
run dst "$LIB/Dupes/Show Two S01E01.mkv"
|
||||||
|
[[ "$output" == "$PARK/"* ]]
|
||||||
|
run dst "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv"
|
||||||
|
[[ "$output" == "$LIB/Show Two (2016)/Season 01/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a file no authority numbers is parked, never renamed into the library" {
|
||||||
|
plan
|
||||||
|
run dst "$LIB/Orphan Tapes/clip001.mp4"
|
||||||
|
[[ "$output" == "$PARK/Orphan Tapes/clip001.mp4" ]]
|
||||||
|
grep -q "clip001.mp4" "$EXC"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a long file the movie provider knows leaves the TV library for the movie one" {
|
||||||
|
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
|
||||||
|
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
|
||||||
|
python3 -c 'import json, sys
|
||||||
|
json.dump({"movie:some documentary":
|
||||||
|
{"name": "Some Documentary Special", "year": 2015, "id": 55}},
|
||||||
|
open(sys.argv[1], "w"))' "$CACHE"
|
||||||
|
plan
|
||||||
|
[ "$(dst "$LIB/Longform/Some Documentary Special (2015).mkv")" \
|
||||||
|
= "$MOVIES/Some Documentary Special (2015)/Some Documentary Special (2015).mkv" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a long file no movie provider knows is parked, not moved into Movies" {
|
||||||
|
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
|
||||||
|
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
|
||||||
|
plan
|
||||||
|
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
|
||||||
|
[[ "$output" == "$PARK/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a movie answer the file does not itself name stays in the review pile" {
|
||||||
|
# TMDb's top hit for a truncated query is a NEAR miss, not an answer: "mobile
|
||||||
|
# suit gundam 00" comes back as Special Edition I while the file is Special
|
||||||
|
# Edition III, and a one-word query "russia" comes back as a 1963 Bond film. A
|
||||||
|
# file leaves the TV library only when it states the title the provider names.
|
||||||
|
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
|
||||||
|
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
|
||||||
|
python3 -c 'import json, sys
|
||||||
|
json.dump({"movie:some documentary":
|
||||||
|
{"name": "Some Documentary Special Part Two", "year": "2015", "id": 55}},
|
||||||
|
open(sys.argv[1], "w"))' "$CACHE"
|
||||||
|
plan
|
||||||
|
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
|
||||||
|
[[ "$output" == "$PARK/"* ]]
|
||||||
|
run awk -F '\t' -v m="$MOVIES/" '$1 == "move" && index($3, m) == 1 {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a movie answer from another year is not that film" {
|
||||||
|
# The year the file states outranks a provider's eagerness: the 2015 special is
|
||||||
|
# not the 1999 one, however alike their names are.
|
||||||
|
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
|
||||||
|
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
|
||||||
|
python3 -c 'import json, sys
|
||||||
|
json.dump({"movie:some documentary":
|
||||||
|
{"name": "Some Documentary Special", "year": "1999", "id": 55}},
|
||||||
|
open(sys.argv[1], "w"))' "$CACHE"
|
||||||
|
plan
|
||||||
|
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
|
||||||
|
[[ "$output" == "$PARK/"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the plan creates every folder before the move that needs it" {
|
||||||
|
plan
|
||||||
|
python3 - "$OUT" "$LIB" "$MOVIES" "$PARK" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
plan, known = sys.argv[1], set(sys.argv[2:])
|
||||||
|
made, bad = set(), []
|
||||||
|
for line in open(plan).read().splitlines()[1:]:
|
||||||
|
action, _src, dest, _kind = line.split("\t")
|
||||||
|
if action == "mkdir":
|
||||||
|
made.add(dest)
|
||||||
|
elif action == "move":
|
||||||
|
d = os.path.dirname(dest)
|
||||||
|
while d not in known and d != os.path.dirname(d):
|
||||||
|
if d not in made:
|
||||||
|
bad.append(f"{dest}: {d} was never made")
|
||||||
|
d = os.path.dirname(d)
|
||||||
|
print("\n".join(bad[:5]))
|
||||||
|
sys.exit(1 if bad else 0)
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "no move lands outside the library or the movie library" {
|
||||||
|
plan
|
||||||
|
run awk -F '\t' -v l="$LIB" -v m="$MOVIES" \
|
||||||
|
'$1 == "move" && $3 !~ "^" l "/" && $3 !~ "^" m "/" {print $3}' "$OUT"
|
||||||
|
[ -z "$output" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the executor preflights the plan and no target is claimed twice" {
|
||||||
|
plan
|
||||||
|
run python3 "$APPLY" apply --plan "$OUT" --root "$BATS_TEST_TMPDIR" \
|
||||||
|
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
dup=$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)
|
||||||
|
[ -z "$dup" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "re-planning a library the plan already satisfies moves nothing" {
|
||||||
|
plan
|
||||||
|
# Applying the plan and re-censusing is exactly what the operator does next:
|
||||||
|
# every authority file is keyed by path, so all three are replayed through the
|
||||||
|
# plan the way a fresh census would see them. (The enum has no header; the
|
||||||
|
# titles export does, and its header row is not a path so it survives.)
|
||||||
|
python3 - "$OUT" "$ENUM" "$TITLES" "$PROBED" <<'PY'
|
||||||
|
import os, sys
|
||||||
|
plan = sys.argv[1]
|
||||||
|
moves = {}
|
||||||
|
for line in open(plan).read().splitlines()[1:]:
|
||||||
|
action, src, dest, _kind = line.split("\t")
|
||||||
|
if action == "move":
|
||||||
|
moves[src] = dest
|
||||||
|
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||||||
|
os.rename(src, dest)
|
||||||
|
for path, col in ((sys.argv[2], 2), (sys.argv[3], 0), (sys.argv[4], 1)):
|
||||||
|
if not os.path.getsize(path):
|
||||||
|
continue
|
||||||
|
rows = [line.rstrip("\n").split("\t") for line in open(path)]
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
for f in rows:
|
||||||
|
f[col] = moves.get(f[col], f[col])
|
||||||
|
fh.write("\t".join(f) + "\n")
|
||||||
|
PY
|
||||||
|
plan
|
||||||
|
[ -z "$(awk -F '\t' '$1 == "move" {print $2}' "$OUT")" ]
|
||||||
|
}
|
||||||
Executable
+212
@@ -0,0 +1,212 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/media-verify.py — the replay of a relayout journal against a
|
||||||
|
# media server's OWN database, which is the documented proof that a rename
|
||||||
|
# landed (a scan answers 204 whether or not it finds anything).
|
||||||
|
#
|
||||||
|
# Every test states what the database holds and asserts what the command prints
|
||||||
|
# and returns; nothing here reaches into the script's internals. The fixtures are
|
||||||
|
# minimal sqlite files with the columns the two servers really use, so the seam
|
||||||
|
# fails if the tool ever reads the wrong table, forgets Plex's `deleted_at`
|
||||||
|
# filter, or forgets that Jellyfin stores the container-side path (/data/…) while
|
||||||
|
# Plex stores the host path verbatim.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
VERIFY="$REPO_ROOT/scripts/media-verify.py"
|
||||||
|
|
||||||
|
# jf_mk <db> — the Jellyfin shape: one BaseItems table, Path is the column.
|
||||||
|
jf_mk() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sqlite3, sys
|
||||||
|
con = sqlite3.connect(sys.argv[1])
|
||||||
|
con.execute("create table BaseItems (Id INTEGER PRIMARY KEY, Path TEXT, Type TEXT,"
|
||||||
|
" ParentId INTEGER, Name TEXT)")
|
||||||
|
con.commit()
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# jf_row <db> <path> — one item the server believes it has.
|
||||||
|
jf_row() {
|
||||||
|
python3 - "$1" "$2" <<'PY'
|
||||||
|
import sqlite3, sys
|
||||||
|
con = sqlite3.connect(sys.argv[1])
|
||||||
|
con.execute("insert into BaseItems (Path, Type) values (?, 'x')", (sys.argv[2],))
|
||||||
|
con.commit()
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# px_mk <db> — the Plex shape: media_parts joined up to metadata_items.
|
||||||
|
px_mk() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sqlite3, sys
|
||||||
|
con = sqlite3.connect(sys.argv[1])
|
||||||
|
con.execute("create table metadata_items (id INTEGER PRIMARY KEY, title TEXT,"
|
||||||
|
" deleted_at TEXT)")
|
||||||
|
con.execute("create table media_items (id INTEGER PRIMARY KEY, metadata_item_id"
|
||||||
|
" INTEGER, container TEXT)")
|
||||||
|
con.execute("create table media_parts (id INTEGER PRIMARY KEY, media_item_id"
|
||||||
|
" INTEGER, file TEXT, deleted_at TEXT)")
|
||||||
|
con.commit()
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# px_row <db> <path> [DELETED_AT] — one part; a non-empty DELETED_AT is a row a
|
||||||
|
# deleted scan left behind, which must never count as proof.
|
||||||
|
px_row() {
|
||||||
|
python3 - "$1" "$2" "${3:-}" <<'PY'
|
||||||
|
import sqlite3, sys
|
||||||
|
db, path, gone = sys.argv[1], sys.argv[2], sys.argv[3] or None
|
||||||
|
con = sqlite3.connect(db)
|
||||||
|
cur = con.execute("insert into metadata_items (title, deleted_at) values ('t', ?)",
|
||||||
|
(gone,))
|
||||||
|
mi = cur.lastrowid
|
||||||
|
cur = con.execute("insert into media_items (metadata_item_id, container)"
|
||||||
|
" values (?, 'mkv')", (mi,))
|
||||||
|
con.execute("insert into media_parts (media_item_id, file, deleted_at)"
|
||||||
|
" values (?, ?, NULL)", (cur.lastrowid, path))
|
||||||
|
con.commit()
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# move_row <src> <dst> — one move in the journal.
|
||||||
|
move_row() {
|
||||||
|
printf '{"op": "move", "src": "%s", "dst": "%s"}\n' "$1" "$2" >> "$JOURNAL"
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
SHARE="$BATS_TEST_TMPDIR/mnt/entropy"
|
||||||
|
JFDB="$BATS_TEST_TMPDIR/jellyfin.db"
|
||||||
|
PXDB="$BATS_TEST_TMPDIR/plex.db"
|
||||||
|
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
|
||||||
|
: > "$JOURNAL"
|
||||||
|
jf_mk "$JFDB"
|
||||||
|
px_mk "$PXDB"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "every move that the database holds at its destination is reported landed" {
|
||||||
|
move_row "$SHARE/Movies/Old Name (1999).mkv" "$SHARE/Movies/Old Name (1999)/Old Name (1999).mkv"
|
||||||
|
move_row "$SHARE/Movies/Two.mkv" "$SHARE/Movies/Two/Two.mkv"
|
||||||
|
# Jellyfin stores the container-side path: /data stands for the share.
|
||||||
|
jf_row "$JFDB" "/data/Movies/Old Name (1999)/Old Name (1999).mkv"
|
||||||
|
jf_row "$JFDB" "/data/Movies/Two/Two.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "${lines[0]}" = "jellyfin: 2 moves checked" ]
|
||||||
|
[ "${lines[1]}" = "landed: 2" ]
|
||||||
|
[ "${lines[2]}" = "still at src: 0" ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a database that has not rescanned yet is refused, naming the stale rows" {
|
||||||
|
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
|
||||||
|
jf_row "$JFDB" "/data/Movies/Old.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[ "${lines[1]}" = "landed: 0" ]
|
||||||
|
[ "${lines[2]}" = "still at src: 1" ]
|
||||||
|
[[ "$output" == *"$SHARE/Movies/Old.mkv"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a moved file no row of the database names is reported missing" {
|
||||||
|
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 1" ]
|
||||||
|
[[ "$output" == *"$SHARE/Movies/Old/Old.mkv"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "Plex stores the host path verbatim, so a row at the destination lands" {
|
||||||
|
move_row "$SHARE/TVShows/Show/Season 01/Show S01E01.mkv" \
|
||||||
|
"$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
|
||||||
|
px_row "$PXDB" "$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
|
||||||
|
--share "$SHARE"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "${lines[0]}" = "plex: 1 moves checked" ]
|
||||||
|
[ "${lines[1]}" = "landed: 1" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a Plex row a deleted scan left behind is not proof a move landed" {
|
||||||
|
move_row "$SHARE/TVShows/Old S01E01.mkv" "$SHARE/TVShows/Show/Old S01E01.mkv"
|
||||||
|
px_row "$PXDB" "$SHARE/TVShows/Show/Old S01E01.mkv" "2026-10-30 00:00:00"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
|
||||||
|
--share "$SHARE"
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[ "${lines[1]}" = "landed: 0" ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 1" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a database without the columns that kind needs is an error, not a zero" {
|
||||||
|
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$PXDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"$PXDB"* ]]
|
||||||
|
[[ "$output" == *"BaseItems"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "only move rows are counted, so folder operations cannot inflate the tally" {
|
||||||
|
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
|
||||||
|
jf_row "$JFDB" "/data/Movies/Old/Old.mkv"
|
||||||
|
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
|
||||||
|
printf '{"op": "rmdir", "path": "%s/Movies/Empty"}\n' "$SHARE" >> "$JOURNAL"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "${lines[0]}" = "jellyfin: 1 moves checked" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a journal with no moves is clean, not a failure" {
|
||||||
|
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "${lines[1]}" = "landed: 0" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the offender list stops at --show so a stale library stays readable" {
|
||||||
|
for n in 1 2 3 4; do
|
||||||
|
move_row "$SHARE/Movies/F$n.mkv" "$SHARE/Movies/F$n/F$n.mkv"
|
||||||
|
done
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data --show 2
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 4" ]
|
||||||
|
[[ "$output" == *"$SHARE/Movies/F1/F1.mkv"* ]]
|
||||||
|
[[ "$output" != *"$SHARE/Movies/F4/F4.mkv"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the missing rows are counted by extension, so a sidecar is not mistaken for a lost film" {
|
||||||
|
move_row "$SHARE/Music/Album/Mandatory Fun.log" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.log"
|
||||||
|
move_row "$SHARE/Music/Album/Film.mkv" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Film.mkv"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 2" ]
|
||||||
|
[ "${lines[4]}" = "missing by extension: .log 1, .mkv 1" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "--ignore-ext keeps a file the server never indexes out of the verdict" {
|
||||||
|
move_row "$SHARE/Music/Album/Song.flac" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
|
||||||
|
jf_row "$JFDB" "/data/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
|
||||||
|
move_row "$SHARE/Music/Album/Mandatory Fun.cue" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.cue"
|
||||||
|
|
||||||
|
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
|
||||||
|
--share "$SHARE" --container /data --ignore-ext log,cue
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ "${lines[1]}" = "landed: 1" ]
|
||||||
|
[ "${lines[3]}" = "missing from the DB: 0" ]
|
||||||
|
[ "${lines[4]}" = "not indexed (extension ignored): 1" ]
|
||||||
|
}
|
||||||
+213
-8
@@ -673,26 +673,44 @@ PY
|
|||||||
|
|
||||||
window_fixture() {
|
window_fixture() {
|
||||||
D="$BATS_TEST_TMPDIR/win"
|
D="$BATS_TEST_TMPDIR/win"
|
||||||
mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora"
|
mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora" "$D/bin"
|
||||||
echo '{}' > "$D/campaign/kld-ref/kld.json"
|
echo '{}' > "$D/campaign/kld-ref/kld.json"
|
||||||
: > "$D/rad/qwen38-bf16.rad"
|
: > "$D/rad/qwen38-bf16.rad"
|
||||||
: > "$D/corpora/calib.jsonl"
|
echo "{\"p\":1}" > "$D/corpora/calib.jsonl"
|
||||||
|
# the window's preflight resolves the image digest through podman and asks about the
|
||||||
|
# port/device by host pid; these stubs answer for a quiet host
|
||||||
|
cat > "$D/bin/podman" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
case "${1:-}" in
|
||||||
|
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
|
||||||
|
esac
|
||||||
|
EOS
|
||||||
|
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/ss"
|
||||||
|
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/fuser"
|
||||||
|
chmod +x "$D/bin/podman" "$D/bin/ss" "$D/bin/fuser"
|
||||||
echo "$D"
|
echo "$D"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the calibration window refuses before touching a card without a reference recording" {
|
@test "the calibration window refuses before touching a card without a reference recording" {
|
||||||
D="$BATS_TEST_TMPDIR/win-less"
|
D="$(window_fixture)"
|
||||||
mkdir -p "$D/campaign/calib"
|
mv "$D/campaign/kld-ref/kld.json" "$D/campaign/kld-ref/kld.json.absent"
|
||||||
run env FRANK_QWEN38_CAMPAIGN="$D/campaign" bash "$WINDOW"
|
run env PATH="$D/bin:$PATH" \
|
||||||
|
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
|
||||||
|
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
|
||||||
|
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
|
||||||
|
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW"
|
||||||
[ "$status" -eq 2 ]
|
[ "$status" -eq 2 ]
|
||||||
[[ "$output" == *"no reference recording"* ]]
|
[[ "$output" == *"FAIL reference"* ]]
|
||||||
|
[[ "$output" == *"must not take the cards"* ]]
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the calibration window's dry run prints the gated plan, including what it would bank" {
|
@test "the calibration window's dry run prints the gated plan, including what it would bank" {
|
||||||
D="$(window_fixture)"
|
D="$(window_fixture)"
|
||||||
run env FRANK_QWEN38_CAMPAIGN="$D/campaign" \
|
run env PATH="$D/bin:$PATH" \
|
||||||
|
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
|
||||||
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
|
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
|
||||||
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" bash "$WINDOW" --dry-run
|
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
|
||||||
|
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW" --dry-run
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
|
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
|
||||||
[[ "$output" == *"else kill the rig"* ]]
|
[[ "$output" == *"else kill the rig"* ]]
|
||||||
@@ -731,3 +749,190 @@ PY
|
|||||||
[ "$status" -eq 2 ]
|
[ "$status" -eq 2 ]
|
||||||
[[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]]
|
[[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ---- the window must not mistake a dying production socket for the rig -------
|
||||||
|
# 2026-10-30: the runner stopped frank-archon, the rig container was still loading,
|
||||||
|
# and /health answered 200 from the production process draining its listener. The
|
||||||
|
# window logged `health ok waited=0s`, the pusher refused on the container name, and
|
||||||
|
# a 12M window died at second zero. The gate must prove the LISTENER is in the rig.
|
||||||
|
|
||||||
|
tap_window_fixture() {
|
||||||
|
export W="$BATS_TEST_TMPDIR/window"
|
||||||
|
mkdir -p "$W/scripts" "$W/campaign/calib" "$W/campaign/kld-ref" "$W/models/rad" "$W/corpora" "$W/bin"
|
||||||
|
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-calib-window.sh" "$W/scripts/"
|
||||||
|
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh" "$W/scripts/"
|
||||||
|
echo "{}" > "$W/campaign/kld-ref/kld.json"
|
||||||
|
: > "$W/models/rad/qwen38-bf16.rad"
|
||||||
|
echo "{\"p\":1}" > "$W/corpora/calib.jsonl"
|
||||||
|
# a gram so the empty-tap check passes on the positive path
|
||||||
|
: > "$W/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
|
||||||
|
|
||||||
|
cat > "$W/scripts/run-qwen38-bf16.sh" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ -n "${STUB_RIG_MARKER:-}" ] && : > "$STUB_RIG_MARKER"
|
||||||
|
sleep "${STUB_RIG_SLEEP:-3}"
|
||||||
|
exit 0
|
||||||
|
EOS
|
||||||
|
cat > "$W/bin/py" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ -n "${STUB_PUSH_SENTINEL:-}" ] && : > "$STUB_PUSH_SENTINEL"
|
||||||
|
exit 0
|
||||||
|
EOS
|
||||||
|
cat > "$W/bin/podman" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
case "${1:-}" in
|
||||||
|
inspect) echo true ;;
|
||||||
|
# the PREFLIGHT asks who owns :7770 (production, per LISTEN_PID); the WINDOW asks which
|
||||||
|
# pids belong to the rig (RIG_PID). Both stubs are the same binary, so keep them apart.
|
||||||
|
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
|
||||||
|
top) echo PID
|
||||||
|
# the WINDOW asks "which pids are the rig's", the PREFLIGHT asks "which are
|
||||||
|
# production's". One stub, two answers: that separation IS the identity lesson.
|
||||||
|
case "${2:-}" in
|
||||||
|
qwen38-bf16) echo "${RIG_PID:-424242}" ;;
|
||||||
|
*) echo "${LISTEN_PID:-424242}" ;;
|
||||||
|
esac ;;
|
||||||
|
*) : ;;
|
||||||
|
esac
|
||||||
|
EOS
|
||||||
|
printf '#!/usr/bin/env bash\nexit 0\n' > "$W/bin/fuser"
|
||||||
|
cat > "$W/bin/ss" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
echo "LISTEN 0 4096 *:7770 *:* users:((\"radiance\",pid=${LISTEN_PID:-424242},fd=9))"
|
||||||
|
EOS
|
||||||
|
cat > "$W/bin/curl" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
exit 0
|
||||||
|
EOS
|
||||||
|
chmod +x "$W/scripts/run-qwen38-bf16.sh" "$W/bin/py" "$W/bin/podman" "$W/bin/ss" "$W/bin/curl" "$W/bin/fuser"
|
||||||
|
}
|
||||||
|
|
||||||
|
tap_window_env() {
|
||||||
|
env PATH="$W/bin:$PATH" \
|
||||||
|
FRANK_QWEN38_CAMPAIGN="$W/campaign" \
|
||||||
|
FRANK_QWEN38_RAD="$W/models/rad/qwen38-bf16.rad" \
|
||||||
|
Q38_WINDOW_CALIB_CORPUS="$W/corpora/calib.jsonl" \
|
||||||
|
Q38_CALIB_MIN_FREE_MIB=1 \
|
||||||
|
Q38_WINDOW_HEALTH_TIMEOUT=1 \
|
||||||
|
PY="$W/bin/py" \
|
||||||
|
STUB_RIG_MARKER="$W/rig.marker" \
|
||||||
|
STUB_RIG_SLEEP=0 \
|
||||||
|
STUB_PUSH_SENTINEL="$W/push.ran" \
|
||||||
|
bash "$W/scripts/qwen38-calib-window.sh"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a health answer from a listener outside the rig is not the rig: no push, window fails" {
|
||||||
|
tap_window_fixture
|
||||||
|
# health 200, container "running", but the listener pid belongs to production
|
||||||
|
export LISTEN_PID=999999 RIG_PID=424242
|
||||||
|
run tap_window_env
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[ -z "$(ls "$W/push.ran" 2>/dev/null)" ]
|
||||||
|
[[ "$output" == *"before it ever answered /health"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "when the listener is inside the rig the window proceeds and banks the tap" {
|
||||||
|
tap_window_fixture
|
||||||
|
export LISTEN_PID=424242 RIG_PID=424242
|
||||||
|
run tap_window_env
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$W/push.ran" ]
|
||||||
|
[[ "$output" == *"TAP BANKED"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- the preflight: every fact by IDENTITY, and it must fail loudly -------------------------
|
||||||
|
# Each case here is a lie that cost a real window on 2026-10-30.
|
||||||
|
|
||||||
|
preflight_fixture() {
|
||||||
|
export PF="$BATS_TEST_TMPDIR/pf"
|
||||||
|
mkdir -p "$PF/campaign/calib" "$PF/campaign/kld-ref" "$PF/bin"
|
||||||
|
echo '{}' > "$PF/campaign/kld-ref/kld.json"
|
||||||
|
echo '{"p":1}' > "$PF/corpus.jsonl"
|
||||||
|
cat > "$PF/bin/podman" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
case "$1" in
|
||||||
|
image) # an image carries every digest it was pulled under; the pin is one of them
|
||||||
|
printf '%s\n' "docker.io/stilldeadcode/radiance@sha256:4beb81eea874e3e308fbffbe1a9402a2f845c19e68ea612ba949b43bd5cb7cb0"
|
||||||
|
printf '%s\n' "docker.io/stilldeadcode/radiance@${STUB_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}" ;;
|
||||||
|
top) echo HPID; echo "${STUB_PROD_PIDS:-4242}" ;;
|
||||||
|
esac
|
||||||
|
EOS
|
||||||
|
cat > "$PF/bin/ss" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ -n "${STUB_LISTENER_PID:-}" ] && echo "LISTEN 0 4096 *:7770 *:* users:((\"x\",pid=${STUB_LISTENER_PID},fd=9))"
|
||||||
|
exit 0
|
||||||
|
EOS
|
||||||
|
cat > "$PF/bin/fuser" <<'EOS'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ -n "${STUB_KFD_PID:-}" ] && echo "${STUB_KFD_PID}"
|
||||||
|
exit 0
|
||||||
|
EOS
|
||||||
|
chmod +x "$PF/bin/podman" "$PF/bin/ss" "$PF/bin/fuser"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_preflight() {
|
||||||
|
( cd "$BATS_TEST_DIRNAME/.." && env PATH="$PF/bin:$PATH" \
|
||||||
|
FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
|
||||||
|
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" \
|
||||||
|
Q38_CALIB_MIN_FREE_MIB=1 \
|
||||||
|
bash scripts/qwen38-preflight.sh "$@" )
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight passes when every fact is true, and says so" {
|
||||||
|
preflight_fixture
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"PREFLIGHT PASSED"* ]]
|
||||||
|
[[ "$output" == *"carries sha256:69add58f"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight accepts the service's own process on the port, not just any answer" {
|
||||||
|
preflight_fixture
|
||||||
|
export STUB_PROD_PIDS=4242 STUB_LISTENER_PID=4242 STUB_KFD_PID=4242
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"held by frank-archon (host pid 4242)"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight fails when the tag resolves to a different digest" {
|
||||||
|
preflight_fixture
|
||||||
|
export STUB_IMAGE_DIGEST="sha256:deadbeefdeadbeef"
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"does not carry sha256:69add58f"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight fails when :7770 belongs to a host pid the service does not own" {
|
||||||
|
preflight_fixture
|
||||||
|
export STUB_LISTENER_PID=999999
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *":7770 is held by host pid 999999"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight fails when another process holds the GPU" {
|
||||||
|
preflight_fixture
|
||||||
|
export STUB_KFD_PID=999998
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"foreign holders:"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight fails when the tap dir already holds grams from another run" {
|
||||||
|
preflight_fixture
|
||||||
|
: > "$PF/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
|
||||||
|
run run_preflight
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"rotate them to .AGENTTRASH"* ]]
|
||||||
|
run run_preflight --allow-existing-tap
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "preflight fails when the launcher is not in the repo root" {
|
||||||
|
preflight_fixture
|
||||||
|
run env PATH="$PF/bin:$PATH" FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
|
||||||
|
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" Q38_CALIB_MIN_FREE_MIB=1 \
|
||||||
|
bash -c "cd /tmp && bash '$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh'"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"not the repo root"* ]]
|
||||||
|
}
|
||||||
@@ -157,12 +157,15 @@ flag_value() {
|
|||||||
[[ "$output" != *"--vram-kv-mib"* ]]
|
[[ "$output" != *"--vram-kv-mib"* ]]
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
|
@test "the calib dry run opens the tap at the 12M floor, on the latest pin, and serves on 7770" {
|
||||||
run bash "$RUNNER" calib --dry-run
|
run bash "$RUNNER" calib --dry-run
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
flat="$(oneline)"
|
flat="$(oneline)"
|
||||||
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
|
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
|
||||||
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
|
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
|
||||||
|
# the tap producer is the campaign's own pin (latest radiance), not production's
|
||||||
|
[[ "$flat" == *"radiance@sha256:69add58f"* ]]
|
||||||
|
[[ "$flat" != *"radiance@sha256:d590e300"* ]]
|
||||||
[[ "$flat" == *"--port 7770"* ]]
|
[[ "$flat" == *"--port 7770"* ]]
|
||||||
[[ "$flat" == *":/data/calib "* ]]
|
[[ "$flat" == *":/data/calib "* ]]
|
||||||
# calib serves; it does not record
|
# calib serves; it does not record
|
||||||
|
|||||||
Reference in new issue
Block a user