feat(media): the movie-library relayout toolchain — census, plan, executor #93

Merged
dank merged 1 commits from feat/media-relayout into main 2026-10-30 19:38:03 -07:00
Owner

What lands

Four scripts, one seam, one runbook page. Nothing here is a deployable service —
scripts/deploy-plan.py returns an empty plan for both targets for every path
in this diff, so merging cannot restart anything that serves traffic.

File Role
scripts/media-census.py the read-only inputs: a stat pass, an ffprobe header pass (niced, so it is safe beside a live endpoint), and the Plex metadata-cache read (mode=ro&immutable=1) whose matched titles outrank any filename guess
scripts/media-titles.py the title authority: the filename parser + the normaliser every comparison runs through
scripts/media-plan.py the Jellyfin-format planner. Opens no media file, so a whole-library plan costs seconds, and it ends in gates — bad-names, occupied, double-claim, left-unmoved, unplanned — that must all read zero before a human points the executor at 24 TB
scripts/media-apply.py the executor: rename(2) only, intent fsynced to a write-ahead journal before the move it authorises
tests/media_relayout.bats 16 tests over a synthetic library: no ffprobe, no network, no server
serverAppConfigs-docs/Dankserver/Media Relayout.md the procedure, the gates, the rollback

Why

/mnt/entropy/Movies is release-folded: Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/
with the video inside named after the release. Jellyfin matches a movie by folder
and file name (its own contract:
one folder per movie, the video named exactly like the folder, extras in the
canonical subfolders, copies of one film as - 2160p / - 4K versions), so the
release noise is what costs posters, extras and versions.

The laws the fixture pins

Each of these was earned against the live library, and each is a test that was
proven red against a copy of the rule reverted:

  • A Plex year of 0 means indexed, never matched. Trusting it named 67
    folders Title (0) — Jellyfin items with no year, no poster, no identity.
  • A Plex title carrying / is sanitised. Plex's own title for V/H/S/2 is
    V/H/S/2; kept verbatim it became Movies/V/H/S/2 (2013)/, a five-level tree
    neither matcher reads as a movie.
  • A folder inside a movie's folder is bonus territory even when its name is
    misspelled.
    Saboteur (1942)/Exrtas/Saboteur - A Closer Look is a typo for
    Extras; the vocabulary rule never fired, the featurette became a movie folder
    of its own, and its words out-scored the real film and drew that box set's other
    extras into it. The replacement rule needs no metadata: the folder holding the
    longest film owns the shorter video filed under it, unless that file's own name
    carries a different year — that is another film speaking.
  • Matching words split on everything that is not a letter or a digit. The
    earlier tokenizer left (1960)/extras glued, so a film could not be found by
    the folder that holds it. A bare number counts only when it could be a release
    year, so 1080 and 2160 never pose as agreement between two names.
  • A name with no words in it is bonus material, not a folder.
    .Extra 1 ( 1H 27min multisub ).mkv parses to 1, and a movie folder named 1
    is not a movie; it becomes a review line telling a human to name the file.
  • The executor renames and nothing else, journals before every move, and
    refuses all-or-nothing: an occupied target, a vanished source, a duplicate
    destination or a cross-filesystem move stops the run before the first rename.
    --force relaxes only the opinion guards; it never permits overwriting.
  • The journal lives beside the library root, never inside it — a log inside
    the library is a file Jellyfin indexes. Rollback is one command off that
    journal, and it removes only the folders the run created and that are empty
    again
    .

Verification

  • ./tests/run.sh — 614/614, including the 16 new tests and the OKF/vault
    gates on the new page.
  • Red-capability: the suite was re-run against a copy of the planner with each
    rule reverted in turn (BAD_CHARS, the year gate, the words gate, the
    tokenizer, the subfolder law) and the matching test went red each time — no
    assertion here is vacuous.
  • Executor tests were red first (no media-apply.py), then green: dry run
    touches nothing, a full apply journals one line per move, an occupied target
    and a vanished source stop the run with nothing moved, an interrupted run
    finishes without moving anything twice, rollback restores every path and
    content, and rollback refuses when a moved file is gone.
  • scripts/deploy-plan.py --target {dankserver,dankclaw} over this diff →
    empty plan (no service restarts).

Not in this PR

Nothing runs on deploy and nothing moves by itself. The library relayout itself
is a separate, ordered act after merge: re-census, plan, read
jf-exceptions.txt
, dry-run, then --apply with the journal, verify, and a
Jellyfin + Plex scan. The census and plan for that run are on Dankserver in
/tmp; the audit verdict is posted as a comment on this PR.


cwd: /home/blank/.omp/wt/wt-20261030-163847-6d1fc1f · session: 01a18f2e-18b1-7573-bcd5-ac99636f8bcb

## What lands Four scripts, one seam, one runbook page. Nothing here is a deployable service — `scripts/deploy-plan.py` returns an **empty plan for both targets** for every path in this diff, so merging cannot restart anything that serves traffic. | File | Role | |---|---| | `scripts/media-census.py` | the read-only inputs: a stat pass, an `ffprobe` header pass (`nice`d, so it is safe beside a live endpoint), and the Plex metadata-cache read (`mode=ro&immutable=1`) whose matched titles outrank any filename guess | | `scripts/media-titles.py` | the title authority: the filename parser + the normaliser every comparison runs through | | `scripts/media-plan.py` | the Jellyfin-format planner. Opens no media file, so a whole-library plan costs seconds, and it ends in gates — `bad-names`, `occupied`, `double-claim`, `left-unmoved`, `unplanned` — that must all read zero before a human points the executor at 24 TB | | `scripts/media-apply.py` | the executor: `rename(2)` only, intent fsynced to a write-ahead journal **before** the move it authorises | | `tests/media_relayout.bats` | 16 tests over a synthetic library: no ffprobe, no network, no server | | `serverAppConfigs-docs/Dankserver/Media Relayout.md` | the procedure, the gates, the rollback | ## Why `/mnt/entropy/Movies` is release-folded: `Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after the release. Jellyfin matches a movie by folder and file name ([its own contract](https://jellyfin.org/docs/general/server/media/movies): one folder per movie, the video named exactly like the folder, extras in the canonical subfolders, copies of one film as `- 2160p` / `- 4K` versions), so the release noise is what costs posters, extras and versions. ## The laws the fixture pins Each of these was earned against the live library, and each is a test that was **proven red** against a copy of the rule reverted: * **A Plex year of `0` means *indexed, never matched*.** Trusting it named 67 folders `Title (0)` — Jellyfin items with no year, no poster, no identity. * **A Plex title carrying `/` is sanitised.** Plex's own title for *V/H/S/2* is `V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/`, a five-level tree neither matcher reads as a movie. * **A folder inside a movie's folder is bonus territory even when its name is misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for `Extras`; the vocabulary rule never fired, the featurette became a movie folder of its own, and its words out-scored the real film and drew that box set's other extras into it. The replacement rule needs no metadata: the folder holding the longest film owns the shorter video filed under it, unless that file's own name carries a different year — that is another film speaking. * **Matching words split on everything that is not a letter or a digit.** The earlier tokenizer left `(1960)/extras` glued, so a film could not be found by the folder that holds it. A bare number counts only when it could be a release year, so `1080` and `2160` never pose as agreement between two names. * **A name with no words in it is bonus material, not a folder.** `.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`, and a movie folder named `1` is not a movie; it becomes a review line telling a human to name the file. * **The executor renames and nothing else**, journals before every move, and refuses all-or-nothing: an occupied target, a vanished source, a duplicate destination or a cross-filesystem move stops the run *before the first rename*. `--force` relaxes only the opinion guards; it never permits overwriting. * **The journal lives beside the library root, never inside it** — a log inside the library is a file Jellyfin indexes. Rollback is one command off that journal, and it removes only the folders the run created *and that are empty again*. ## Verification * `./tests/run.sh` — **614/614**, including the 16 new tests and the OKF/vault gates on the new page. * Red-capability: the suite was re-run against a copy of the planner with each rule reverted in turn (`BAD_CHARS`, the year gate, the words gate, the tokenizer, the subfolder law) and the matching test went red each time — no assertion here is vacuous. * Executor tests were red first (no `media-apply.py`), then green: dry run touches nothing, a full apply journals one line per move, an occupied target and a vanished source stop the run with nothing moved, an interrupted run finishes without moving anything twice, rollback restores every path and content, and rollback refuses when a moved file is gone. * `scripts/deploy-plan.py --target {dankserver,dankclaw}` over this diff → empty plan (no service restarts). ## Not in this PR Nothing runs on deploy and nothing moves by itself. The library relayout itself is a separate, ordered act after merge: re-census, plan, **read `jf-exceptions.txt`**, dry-run, then `--apply` with the journal, `verify`, and a Jellyfin + Plex scan. The census and plan for that run are on Dankserver in `/tmp`; the audit verdict is posted as a comment on this PR. --- cwd: /home/blank/.omp/wt/wt-20261030-163847-6d1fc1f · session: 01a18f2e-18b1-7573-bcd5-ac99636f8bcb
dank added 1 commit 2026-10-30 19:36:54 -07:00
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
Author
Owner

Live audit of the plan this toolchain just produced (Dankserver, fresh census)

Census re-run with the shipped scripts at 19:31 PDT — enumerate 8193 files,
probe 4842 containers (6 workers, niced, beside the live frank-archon),
plex 3774 matched leaves — then planned:

groups=2675 folders=2669 moves=8189 mkdirs=4197 film-files=2849
kinds={'film': 2849, 'disc-file': 632, 'extra': 1768, 'track': 1883, 'misc': 634,
       'image': 264, 'nfo': 64, 'junk': 20, 'orphan': 5, 'tv': 44, 'iso': 1, 'watermark': 25}
problems=34 bad-names=0 occupied=0 double-claim=0 left-unmoved=0 unplanned=0

Every gate reads zero. The independent hazard audit over the plan TSV agrees:

Check Verdict
census files with no plan row 0 (8193 = 8189 moves + 4 already at target)
double-claimed destinations 0
targets occupied by a file that stays 0
bad name components (<>:"/|?*, over-long, whitespace-edged) 0
Title (0) folders (the Plex-year-0 class) 0
non-canonical children under a movie folder {}
diff vs the previously reviewed plan 8189 rows identical, 0 moved elsewhere, 0 added, 0 removed
bytes 21.92 TiB census, 21.86 TiB moving — a rename preserves them

The three bare-number paths the audit flagged are not movie folders: they are the
.pad/0, .pad/1, .pad/2 padding files of one release, filed as misc under
that film's other/. Correct as-is.

The 34 problem lines are the whole review list, and every one is meaningful:
20 macOS AppleDouble sidecars (junk pile), 5 ownerless sidecars, 3 Bruce Lee
extras whose names are numbers (UNNAMED — bonus pile until a human names them),
2 Nausicaa extras that live in a collection folder holding no film, 2 qBittorrent
.!qB partials refused as SKIPPED, 1 lone DVD .iso and 1 second
MAKEMKV master flagged REVIEW, and 2 README notes (one of them: Big Little
Lies
and a Scientology special are TV, parked in _TV/ with a note rather
than mis-filed as movies).

Nothing moves until the operator runs the executor with --apply; the journal
will sit at /mnt/entropy/media-relayout.jsonl, beside the library, never inside it.

## Live audit of the plan this toolchain just produced (Dankserver, fresh census) Census re-run with the shipped scripts at 19:31 PDT — `enumerate` 8193 files, `probe` 4842 containers (6 workers, `nice`d, beside the live `frank-archon`), `plex` 3774 matched leaves — then planned: ``` groups=2675 folders=2669 moves=8189 mkdirs=4197 film-files=2849 kinds={'film': 2849, 'disc-file': 632, 'extra': 1768, 'track': 1883, 'misc': 634, 'image': 264, 'nfo': 64, 'junk': 20, 'orphan': 5, 'tv': 44, 'iso': 1, 'watermark': 25} problems=34 bad-names=0 occupied=0 double-claim=0 left-unmoved=0 unplanned=0 ``` Every gate reads zero. The independent hazard audit over the plan TSV agrees: | Check | Verdict | |---|---| | census files with no plan row | **0** (8193 = 8189 moves + 4 already at target) | | double-claimed destinations | **0** | | targets occupied by a file that stays | **0** | | bad name components (`<>:"/\|?*`, over-long, whitespace-edged) | **0** | | `Title (0)` folders (the Plex-year-0 class) | **0** | | non-canonical children under a movie folder | **{}** | | diff vs the previously reviewed plan | **8189 rows identical, 0 moved elsewhere, 0 added, 0 removed** | | bytes | 21.92 TiB census, 21.86 TiB moving — a rename preserves them | The three bare-number paths the audit flagged are not movie folders: they are the `.pad/0`, `.pad/1`, `.pad/2` padding files of one release, filed as `misc` under that film's `other/`. Correct as-is. The 34 problem lines are the whole review list, and every one is meaningful: 20 macOS AppleDouble sidecars (junk pile), 5 ownerless sidecars, 3 Bruce Lee extras whose names are numbers (`UNNAMED` — bonus pile until a human names them), 2 Nausicaa extras that live in a collection folder holding no film, 2 qBittorrent `.!qB` partials refused as `SKIPPED`, 1 lone DVD `.iso` and 1 second `MAKEMKV` master flagged `REVIEW`, and 2 `README` notes (one of them: *Big Little Lies* and a Scientology special are **TV**, parked in `_TV/` with a note rather than mis-filed as movies). Nothing moves until the operator runs the executor with `--apply`; the journal will sit at `/mnt/entropy/media-relayout.jsonl`, beside the library, never inside it.
dank merged commit e1760e3316 into main 2026-10-30 19:38:03 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#93