Compare commits

...
Author SHA1 Message Date
Frank 7aa32d1b64 feat(media): the operator's map gives a folder the id its name cannot reach
Tvdb titles a record in the show's own language and keeps the English name in
aliases the name search never reads, so the rule that accepts only tvdb's own
spelling of a query can never resolve them: ブラッククローバー (331753) holds no
alias at all, and the bare alias "One Piece" belongs to 392276, the 2023 live
action, not to the 1999 record (81797) that 1,072-file pack is. Seventeen pack
folders on the array — 3,507 files — are in that class.

media-shows.py now reads scripts/media-shows-tvdb-ids.tsv by default: one line
per pack folder, folder <TAB> tvdb id <TAB> note, every id read from tvdb's own
record. The map outranks a provider id, because a box set's provider id is the
server's own guess at a folder name, and it merges the release folders that are
one show. Everything it touched goes to the exceptions: every folder it named,
every row that matched no folder, every disagreement with a provider id, every id
tvdb never answered. The file is read fail-closed — an id that is not a number,
or one folder named twice, stops the plan, because a typo there renames a pack of
1,268 files to the wrong show.

scripts/media-shows-tvdb.py is the sheet the map was built from, promoted out of
/tmp so the next run is reproducible: it reads a plan's mkdir rows, asks tvdb
about every destination folder carrying no [tvdbid-], and grades the answer
EXACT / PUNCT / NEAR / NONE by how much was guessed to reach it. Acceptance is
unchanged from the passes it replaces — bare-form equality plus the year-span
guard — and that gate is what rejected three confident picks: Fairy Tail's 2024
sequel (410031), the 2026 Hokuto no Ken remake (465216) and Mugyutto! Black
Clover (378762) offered for Black Clover. The sheet is for the operator; the map
is what the planner reads, and only after he approves the rows.

Tests: seven for the map's behaviour — rename to the record it names, two release
folders becoming one series, precedence over a provider id and the line stating
the disagreement, an id tvdb never answered, a row whose folder is gone, a
malformed id refusing the run, a folder named twice refusing the run — plus one
that the shipped file parses and never stops a plan.
2026-10-31 13:10:32 -07:00
dank 576c3421b8 Merge pull request 'feat(media): the folder carries its tvdb id; specials and year-seasons resolve' (#105) from feat/shows-ids-and-specials into main 2026-10-31 12:56:35 -07:00
Frank e4656cfd7a fix(media): a tvdb episode page is not the episode list
Tvdb's default episode query answers at most 500 rows and states no total.
One Piece (1999), Crayon Shin-chan (1992) and Detective Conan (1996) each
answer exactly 500 while holding over a thousand, so the contradiction rule
added in the previous commit read every long-running show as contradicted:
the pack books more episodes than "the authority holds", and the rule that
exists to stop a wrong anime title would have stripped the right one from
every file of the season.

A capped list now records that it is capped (`tvdb:capped:<id>`, so a warm
cache answers the same question a cold one did) and contradicts nothing;
titles for the episodes the page does hold still ride.

Test: a 500-row page with a 501-episode pack keeps ` - Episode 1` and lists
no contradiction. Red before the change, proved in a copy of the tree with
the cap raised past the fixture.
2026-10-31 12:40:49 -07:00
Frank acab4cef22 feat(media): the folder carries its tvdb id; specials and year-seasons resolve
Three defects in the TVShows planner, all three visible in the applied plan:

- A series folder named a show by words alone. Jellyfin's Tvdb agent and
  Plex's scanner both read `[tvdbid-NNNN]`, so the resolved id now rides the
  series folder — and only the folder: a file name gets 180 characters and
  the episode title needs the room.
- 1,412 files parked because their provider row said `season 0`. Season 0 is
  a legal season; Jellyfin's contract is `Season 00` for specials.
- 1,033 files parked because Jellyfin's date matcher writes the AIRDATE YEAR
  into the season column. Such a row is now renumbered by the title tvdb
  states for the series' id, read backwards off its own episode list. A
  title tvdb gives to two episodes names no episode, and that file parks
  with the reason it always carried.

A season tvdb's own episode list CONTRADICTS — a release pack books more
episodes there than the authority holds, the anime numbering hazard — keeps
its rename and loses the tvdb title, because the slot its number points at
is a different episode. Those series are listed in the exceptions file,
which is where the operator reads them.

Tests: six through the fixture (five red before the change; the ambiguous
title is a guard that the recovery must never guess). Full bats suite green.
2026-10-31 12:27:15 -07:00
dank 04802b3e48 Merge pull request 'fix(media): verify read the library the way the servers write it' (#103) from fix/media-verify-wal-read into main 2026-10-31 12:12:23 -07:00
dank 44e55ad172 Merge pull request 'feat(media): tvdb names the show — titled episode files, id-backed series folders, park outside' (#102) from feat/tv-tvdb-naming into main 2026-10-31 12:12:15 -07:00
dank 132805a25a Merge pull request 'fix(dankserver): mount /mnt/entropy from its LUKS mapper at boot' (#104) from fix/dankserver-entropy-automount into main 2026-10-31 12:04:08 -07:00
Frank 2406641bc6 fix(dankserver): mount /mnt/entropy from its LUKS mapper at boot
The crypttab entry named the md RAID array UUID (61bd6eaf-...) where a luks
entry takes the LUKS container UUID (cd86c728-...), so
systemd-cryptsetup@entropy.service could never produce /dev/mapper/entropy and
the fstab entry could not resolve either; the volume only appeared when a
desktop session let udisks2 unlock it, which kept the broken boot path hidden
for a year. The keyfile was wrong as well: 27 bytes, i.e. the 26-character
passphrase plus a trailing newline, and cryptsetup hashes the file whole.

Records the host config as dankserver/host/{crypttab,fstab} (the dankclaw/host
precedent), rewrites the Entropy Volume page to own the chain, the UUID law and
the keyfile contract, and comments out the now-unmountable /dev/md0 line.

Proven on the host: /etc/{crypttab,fstab} byte-identical to the repo copies,
reboot into 7.0.0-42-generic mounts /mnt/entropy from /dev/mapper/entropy, all
12 containers returned, frank-archon back to 141 t/s (was 6.7 t/s on a wedged
amdgpu queue). ./tests/run.sh: ALL TESTS PASSED (749).
2026-10-31 11:56:39 -07:00
Frank dbcf8a2c35 docs(media): the verifier prints the four verdict classes itself
The taxonomy was written as something an operator assembled by hand; the tool now
emits `on disk at dst` / `REAL` / `gone` lines, which is the difference between
reading `missing from the DB: 1851` as 1851 lost albums and reading it correctly.

cwd: /home/blank/.omp/wt/wt-20261030-163847-6d1fc1f · session: wt-20261030-163847-6d1fc1f
2026-10-31 10:29:33 -07:00
Frank a5a9c04da2 feat(media): the verdict splits what the database alone cannot
"No row at the destination" was one number, and it conflated three different
things: a sidecar the server never indexes, a file a later pass renamed again,
and a file that never left its source — the last one being actual damage. The
runbook already named the four classes; the tool never computed them, so an
operator reading `missing from the DB: 1851` had no way to tell a booklet.pdf
from a lost album and might have run a rollback over a false alarm.

`missing` rows are now stat'd on both paths and reported as
`on disk at dst, no row names it` / `REAL — file still at src` /
`gone — at neither path`, with offenders listed in that order. A row still
naming `src` stays a separate line: that is the server's own stale pointer, a
phantom, and it costs a re-scan rather than a restore.

Two new tests (REAL, and gone-vs-sidecar) plus two existing layout assertions
loosened from line positions to content. 14/14 in tests/media_verify.bats.
2026-10-31 10:28:44 -07:00
Frank c95ae20bfb Merge main into feat/tv-tvdb-naming: the arms moved to their own pages
main split the music and audiobook arms out of Media Relayout.md and put a
locator in their place, so the branch's trims inside those sections now live on
the two child pages, and the parent keeps main's locator. The two Jellyfin laws
the branch moved to Jellyfin.md stay moved; the sweep law keeps the branch's
wording ("until the next sweep"), which is the one the next line depends on.

Full suite: 758 tests, all green.
2026-10-31 10:22:34 -07:00
Frank a0dfa61623 docs(media): the TVShows id-map proposal — where it lives and what may be applied unreviewed 2026-10-31 10:18:16 -07:00
Frank 4b67c3c4d8 fix(media): the census read Plex the same wrong way
The two Plex arms of media-census.py — `plex` (the movie planner's title
authority) and `audiobooks` (the chapter rows the audiobook plan files by) —
opened the library `mode=ro&immutable=1` while the Jellyfin arms already used
plain `mode=ro`. Same defect as media-verify.py: an uncheckpointed row of a
running scan is invisible, so a just-scanned item is simply absent from the
authority file a plan is built from, and the pass cannot say why.

Both now go through open_plex_ro(), which tries plain ro first and keeps
immutable only as the fallback for a database whose -shm cannot be read. The
new audiobooks test leaves a chapter row committed-but-uncheckpointed and
asserts the export still carries it; red before the change, green after.
2026-10-31 10:13:17 -07:00
Frank 7ee831adee docs(media): what a verification verdict actually means, and what a rollback can undo
Three things the 2026-10-31 five-library relayout taught, written where the
next operator reads them:

- A verify run must open the servers' databases plain `mode=ro`. Both keep
  their libraries in WAL mode; `immutable=1` skips the write-ahead log, so a
  scanner's own committed rows read as missing and the read can fail as
  "database disk image is malformed". The census page claimed immutable — now
  it says what the code does.
- "No row at the destination" splits into four classes, and only REAL (no row
  at dst AND the file still at src) is the relayout's own damage. Phantom rows
  naming a pre-move path are normal behaviour for both servers. And the owning
  server decides whether a pass can convict: Jellyfin holds no audiobook
  library, so that journal verifies against Plex or it proves nothing.
- A chain of passes rolls back newest journal first, or the tool refuses; the
  refusals measured that day are the tool working, not failing.

The music and audiobook arms moved to their own pages (progressive disclosure;
the parent drops 175 lines and stays the owner of the shared gates, the
executor, the journal law and the rollback). Stale per-run counts destroyed:
episode counts, byte sizes, move tallies for a pass that already happened.
2026-10-31 10:13:11 -07:00
Frank afad21dda5 fix(media): verify read the library the way the servers write it
A verify run beside a running scan reported rows the scanner had just
written as MISSING, and could fail outright as "database disk image is
malformed". Both libraries live in WAL mode, and the verifier opened them
with immutable=1, which tells sqlite the file never changes: it skips the
write-ahead log, so every committed-but-uncheckpointed row is invisible.
134 MB of jellyfin.db-wal was riding past the verifier's eyes during the
2026-10-31 rescan.

Open plain mode=ro first — sqlite then reads the WAL through its -shm —
and keep immutable=1 only as the fallback for a database whose -shm
cannot be read. The bats fixture leaves a journaled move uncheckpointed
and proves the verifier still counts it landed.
2026-10-31 10:00:11 -07:00
Frank f430e2264d docs(plex): the scan recipe kept its own promise about the token
The block said the token rides stdin and never argv, then handed it to curl as
`-H "X-Plex-Token: $T"` — world-readable in /proc/<pid>/cmdline. It now writes
a mode-600 curl config inside the container, which is also how the overnight
rescan was actually issued (five sections, all accepted). Two truths the old
block got wrong are stated: a section answers 200 with a Directory body when
Accept: application/json is sent, and a backslash-newline inside a quoted
printf format is literal, so the config builder writes two printfs.
2026-10-31 09:50:22 -07:00
Frank 82a2183d40 docs(media): a film handoff needs the file's own words and year
The provider's first hit for a salvaged query is a near miss, and the page said
only that a provider must name the file. The bullet now states the two checks and
the two real misfiles they exist to stop.
2026-10-31 07:39:45 -07:00
Frank c70f9b3f6d fix(media): a film leaves the TV library only for a film it actually names
The first tvdb-grade plan handed 12 files to the movie library, and two of them
were wrong in a way the exceptions file could not show: an episode of the
documentary series `Russia 1985-1999: TraumaZone` was renamed to the 1963 Bond
film `From Russia with Love`, and `Gundam 00 Special Edition III - Return The
World` became `Special Edition I - Celestial Being`.

Two guards, each alone insufficient:

- `title_of` salvages a short query from a long release name ("russia", "mobile
  suit gundam 00"), and tmdb() took the FIRST search result. The call site then
  compared the provider's words against that salvaged query, so any title merely
  CONTAINING the query passed. The provider's title must now be a run of words the
  FILE itself states: `Special Edition I: Celestial Being` is not in a filename
  that says `Special Edition III - Return The World`.
- tmdb() read the year from `year`/`first_air_date`, which the MOVIE search does
  not return — it returns `release_date`. Every film answer therefore arrived
  year-less and the year guard at the call site could never fire. That is how a
  1963 answer survived a filename stating 1991.

Tests: a provider answer the file does not name parks, and nothing lands in the
movie library; an answer from another year is refused. The year-from-the-wire half
is proven by the live re-plan, which now dates every movie answer.
2026-10-31 07:38:54 -07:00
Frank 32b8901bff docs(media): the pile is matched with its underscore stripped
The server drops the leading underscore from the hidden folder's display name, so
the park-echo rule that compares spelled names lets the review pile survive as a
visible series. The law now says which spelling is compared; the mechanism stays
with the guard's docstring.
2026-10-31 07:28:22 -07:00
Frank 6d03bbc8e7 fix(media): the review pile is not a series, and a folder carries the year once
Two destination defects the first tvdb-grade plan exposed in its own output —
both visible by reading the plan's folder names, neither caught by any gate.

1. Jellyfin sanitizes a hidden folder's display name: `_unsorted` comes back as
   `unsorted`. The park-echo guard compared the row's series against the folder
   with the underscore, so 1,091 pile rows survived as a real series and the plan
   renamed 617 files out of the pile into a VISIBLE folder named `unsorted` —
   inventing the junk show the guard exists to prevent. The comparison now strips
   leading underscores from both sides.

2. TVDb hands back some series with the year already on the name (`Acapulco
   (2021)`). label_of appended the year anyway, producing `Acapulco (2021)
   (2021)` — a folder no matcher and no human reaches for twice. A name that
   already ends in `(YYYY)` is left alone.

Both are covered by fixtures: a pile row whose series is the sanitized spelling
must park, never file under the pile's name; a provider name carrying its year
produces one year in the path and the plan contains no `(YYYY) (YYYY)` anywhere.
2026-10-31 07:27:47 -07:00
Frank 32b58dd787 fix(media): TMDb's v3 key travels as a query parameter; Bearer is a JWT only
The first tvdb-grade TV plan built 49,771 moves with not ONE TMDb answer: every
search and every external_ids call came back 401. The credential on this box is
the v3 key (~/.config/secrets/tmdb, `api_key=`), and TMDb takes a v3 key only as
the `api_key` query parameter — a v4 request token is the only thing the Bearer
header accepts. Measured on the same credential in one pass: the query form
resolves `New Girl (2011)`, the Bearer form raises HTTP 401.

tmdb_request() now picks the transport from the shape it was handed (three
dot-separated segments = a JWT → Bearer, anything else → query parameter), and
the old header comment's reasoning is corrected: a token in argv is
world-readable in /proc/<pid>/cmdline, but this URL is built inside the process
and never reaches argv, so the query form leaks nothing.

The failure mode that let it look like a finished plan is fixed with it. The
401s were stderr-only, so the summary printed a confident move count while every
series folder still carried the metadata provider's folder-echo name — the exact
junk this contract exists to undo. Provider health is now part of the summary
line and of the exceptions file, and a test covers it offline with
`env -u TMDB_API_KEY`: a plan that asked and could not ask says NOT
provider-complete in both places, and still writes the moves it can justify,
because the answers that did arrive are worth their rename.
2026-10-31 07:16:46 -07:00
Frank 0a365274b6 docs(agent): a PR comment posts to the issues endpoint, and an empty jq body is silent
Measured while commenting on PR 102: POST /pulls/102/comments answers 405 to a
plain body (it is the review-comment store); the conversation thread is
POST /issues/<PR-number>/comments, which answers 201 and returns an html_url on
the PR. Gitea maintains one issue index, so the PR number is the same number.

The trap that hid it: building the body with jq --rawfile b /dev/stdin inside a
command substitution feeds jq nothing (the substitution does not inherit the
heredoc), Gitea answers 422, and jq -r .id on the error prints nothing — a
silent no-op that looked like a posted comment. The body goes in a file.
2026-10-31 07:11:13 -07:00
Frank e531e90278 docs(media): a scan request belongs to the server's own page
Media Relayout.md hit the vault's 500-line gate the moment the tvdb contract
landed, and the fat in it was not the new text: the Plex and Jellyfin scan
blocks (35 lines) sat inside the relayout procedure even though they are the
way you talk to those servers after ANY change, and two Laws bullets repeated
the same facts a third time. Both blocks now live on [[Plex]] and
[[Jellyfin]]; the relayout page keeps the two laws an operator needs at that
moment — 204 is acceptance and anything else queued nothing, and one accepted
refresh is the whole procedure, because a second moves `Scan Media Library` to
`Cancelling` and every later 204 is discarded.

Same pass, the dead numbers the vault's own law says to destroy: an applied
music pass's date, an applied audiobook pass's move/park counts, and the 146
sweep stragglers of one specific first sweep. The laws they taught stay.

490 lines, `bats tests/docs_completeness.bats` 6/6 (length violation gone,
link integrity clean).
2026-10-31 07:10:28 -07:00
Frank c3ab565d9b feat(media): tvdb names the show — episode titles, id-backed folders, park outside
The TV relayout that landed copied Jellyfin's Series names verbatim, and the
naming census showed what that launders in: one Series item per release folder
(1,107 items for 1,107 top-level dirs), the same show spelled six ways, and
box sets living in 346 separate folders for 821 identities. Jellyfin names a
Series after the folder it scanned, so `24 (2001) Season 1` is a folder echo,
not a show name.

The name authority is now tvdb, reached through the id the server matched:

  * media-census.py jellyfin emits two more columns — `title` (the episode row's
    own Name) and `tvdb` (the series id, read from BaseItemProviders where
    Jellyfin 12 keeps provider ids and from the row's Data blob where older
    builds do, so the census does not care which shape the live DB has);
  * media-shows.py resolves each series to tvdb — the row's tvdb id, else its
    TMDb id through TMDb's external ids, else an exact tvdb name search — and
    writes tvdb's name and year. That also collapses the provider's spellings
    into ONE identity, which is what turns a box set split over five release
    folders into one folder and what makes two files claiming one episode MEET
    in the duplicate rule;
  * the episode file becomes `<Series> (<year>) - SxxEyy - <Title>`: the title
    comes from tvdb's episode list, else the row's own title, else nowhere — no
    title means no separator, never an invented one.

Two refusals come with it. A row whose series is the hidden top-level folder the
file itself sits in is NOT evidence: the review pile parked inside the library
was indexed as one show named `_unsorted`, and its rows would have renamed real
shows to it. And inside such a folder a file's own words may MATCH a registered
show but never INVENT one — invention is what filled the pile. `--park`
therefore belongs outside the library, which is also what makes the next plan
re-file the pile instead of skipping it.

Both providers answer offline-reproducibly: every answer is cached in one JSON
file, so a plan can be re-planned without a key. Lookups stay behind --tmdb and
--tvdb, and the tvdb token is read from a mode-600 file, never argv.

tests/media_shows.bats is 23 tests, none of which touch the network — the
fixture Jellyfin DB carries the providers table and the name cache is an input
file. ./tests/run.sh: 753 green.
2026-10-31 07:01:50 -07:00
Frank 5dbbfcb8a0 docs(media): an accepted refresh can still be dropped by a wedged task runner
Tonight's sequence, measured: the post-sweep refresh answered 204 while
`Scan Media Library` sat in `Cancelling`, and Jellyfin's stale-path counts froze
for an hour at 0 % CPU — the request was accepted and discarded. Restarting the
user unit cleared the runner; one POST after that reported State `Running` at
130 % CPU and the counts began to move. So the status code is not the evidence,
and the restart is `systemctl --user restart jellyfin.service`, never
`deploy.sh`, which would run the never-tested metadata-view cutover.
2026-10-31 04:13:12 -07:00
Frank 02a568fb97 docs(media): the Jellyfin scan 401 was curl-config quoting, not the server
Corrects the lesson written an hour ago, which said a non-204 was transient and
should simply be re-issued. It is reproducible and mechanical: `printf` with
`\"` inside the single-quoted format string drops the backslash, curl then ends
the quoted value at the first bare quote, and Jellyfin receives
`Authorization: MediaBrowser Token=` with an empty key. Both forms were run
against the host in one pass — `\\"` answers 204, `\"` answers 401 — so the
block's double backslashes are load-bearing, and the handoff script that deviated
left a 58k-move library unscanned for 40 minutes.

Also the trap that diagnosis walked into: a second accepted refresh while a scan
is running moves `Scan Media Library` to `Cancelling` and the pass starts over.
One accepted request is the procedure; wait it out.
2026-10-31 03:41:22 -07:00
Frank 1e1527d101 docs(media): the SSD metadata view has never been deployed on Dankserver
The vault described the overlay as the live shape. It is not: `media-view.sh
status` answers NOT MOUNTED, /var/lib/media-meta does not exist, and the
installed quadlets under ~/.config/containers/systemd still carry the pre-view
binds (/mnt/entropy:/data:ro for Jellyfin, /mnt/entropy:/mnt/entropy:ro for
Plex) while git binds the view with ConditionPathIsMountPoint. The deploy gate
(deploy-remote.sh:ensure_media_view) has therefore never run: the first
`deploy.sh dankserver --service jellyfin|plex` is an untested cutover of both
media servers, and Jellyfin's read-only sidecar error is today's expected state,
not a fault.

Same commit, the scan-acceptance law from tonight's TV handoff: it logged
`jellyfin scan: 401`, so Jellyfin never rescanned a 58k-move library until the
request was re-issued with the identical key and header form (204). Anything
that is not 204 means no scan was queued — re-issue, do not diagnose.
2026-10-31 03:13:53 -07:00
Frank 9eb85ea1f9 fix(media): order the sidecar reap only where the view is mounted
Every apply and rollback printed "next: scripts/media-view.sh reap", but on
Dankserver there is nothing to reap: the metadata view has never been installed
(`media-view.sh status` -> NOT MOUNTED, `/var/lib/media-meta` absent) and the
live quadlets still bind the array read-only. All three relayout arms tonight
issued that order — 242,084 moves — and it pointed at a directory that does not
exist. Where no overlay is mounted the servers bind the array itself, so the
sidecars moved with the media.

`--metadata-view` (default /var/lib/media-meta/view) names the overlay and the
note now rides only when that path is a mountpoint. Red first: two new cases in
tests/media_relayout.bats (apply and rollback); the shared helper names a real
mountpoint so the pre-existing presence assertion keeps its meaning. 37 ok.
2026-10-31 03:13:53 -07:00
dank 96944f0b17 feat(media): media-verify.py replays a journal against the servers' own tables
A scan answers 204 whether or not it found anything, so the fleet's documented
proof — every move row has a row at its dst and zero rows still name its src —
was a hand-written sqlite query every time anyone needed it. That is the step a
half-finished relayout is judged on, so it is now a command with an exit code.

It encodes the two doors that bite: Jellyfin stores the container-side path
(/data stands for the share) while Plex stores the host path verbatim and hides
deleted scans behind deleted_at on two tables. The report breaks the missing rows
down by extension, because a .cue or .log has no row to land in and must not read
as a lost album; --ignore-ext keeps those out of the verdict.
2026-10-31 01:02:48 -07:00
dank 33f97cd3b3 docs(media): the TV library arm gets its own journal, and its plan lives on the array
Two relayouts writing one journal cannot be rolled back apart: the mover
(Movies -> TVShows) and the library re-filer both pointed at
media-relayout-tv.jsonl. The re-filer now writes media-relayout-shows.jsonl,
and its plan/exceptions move off /tmp, which a reboot eats and which is not a
record.
2026-10-31 00:55:32 -07:00
dank 254fb41fd0 docs(media): the music relayout is mid-flight, and the array takes one writer
The page said the music plan "was applied" while 62k of its 131k journal rows
were still being written; a runbook line that gets believed is worse than none.
Also lands the single-writer law the audiobook handoff exists to honour, and the
durable plan path (/tmp is not a record).
2026-10-31 00:40:30 -07:00
dank 3613f69593 Merge pull request 'feat(media): the audiobook relayout, filed by the book Plex matched' (#101) from feat/media-audiobook-relayout into main 2026-10-31 00:38:02 -07:00
dank 6bf70c6a64 fix(media): a shelf needs a name, so a mistagged chapter number parks
Plex's artist row for 272 chapters is a chapter number the ripper put in the
artist tag (0.4, 31, 05), which would have filed them under a shelf called 0.4.
A narrator is a person, so it must contain a letter; the same test is deliberately
not applied to a book title, which may honestly be 1984.
2026-10-31 00:34:40 -07:00
dank f1b8b60817 fix(media): a tag-scan placeholder is not a book identity
The first real plan filed 1,106 chapters from 14 unrelated top-level folders
onto one shelf, because all 1,106 carry the album tag [Unknown Album] and the
same artist tag. Plex's album title is the authority for the shelf only when it
names a book; a placeholder names nothing, and unrelated books share it. Those
chapters now park with the path they have today, timestamp-glued variants
included; Various Artists stays a legitimate multi-cast narrator.
2026-10-31 00:31:30 -07:00
dank 4ae2bde9ab docs(media): fit the audiobook section inside the OKF caps, refresh the index
The linter caps a page at 500 lines and a description at 120 chars: the section
lost four lines of prose (no law went with them) and the description lost its
restatement of the format names. gen-vault-index.py regenerated the locator row.
2026-10-31 00:25:58 -07:00
dank ceb5617a42 docs(media): the audiobook arm, and the music plan's applied state
One page owns the relayout facts: the new section carries the Plex-only reader,
the shape, the four commands, and the two laws a future operator would otherwise
re-litigate (never renumber a chapter; the shelf is the narrator). The music
section said its plan was NOT applied — it was applied on 2026-10-31, and a stale
line in a runbook gets believed, so it now names the journal and the rollback.
2026-10-31 00:24:50 -07:00
dank a7ceda95fd feat(media): the audiobook relayout, filed by the book Plex matched
The audiobook tree is the one library Jellyfin does not read at all and Plex
reads completely: 58,975 chapters matched into Artist/Album/Track, where the
album is the BOOK and the book's single artist row is the NARRATOR. That is the
authority the new planner files by, never the collection folder a pack made.

  Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>

Two laws are the opposite of the music tool's, both forced by the rows: files
are never renumbered (72% of the books carry chapters with no usable index, and
playback order comes from the embedded track tag either way, so an invented
number can only lie), and the shelf is the narrator, not the author (every book
has exactly one artist row; the author survives in 10% of album titles and in no
column Plex stores, so one rule beats a shelf half authors and half narrators).

media-census.py gains the audiobooks export: the chapters Plex matched under a
root, host paths straight through, nothing outside the root and nothing Plex
deleted. 22 tests in tests/media_audiobooks.bats pin the export's join and the
plan's output.
2026-10-31 00:24:45 -07:00
dank a32361ee5e Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-31 00:10:58 -07:00
dank ce6b3b1d3b docs(media): how to prove a relayout landed, read from both servers' tables
A scan answers 204 whether or not it found anything, so proof is the journal
replayed against Jellyfin's BaseItems.Path and Plex's media_parts.file: every
move has a row at its dst, no row names its src. Records the four wrong doors:
no sqlite3 binary on the hosts or in either container, Jellyfin's single
BaseItems table (no MediaItems), its container-side /data path prefix, and
Plex's HTTPD closing on ?path= / on /search matching titles not paths.
2026-10-31 00:10:50 -07:00
dank 382e3eb0b8 Merge pull request 'feat(media): the music library's own relayout, filed by its tags' (#100) from feat/jellyfin-provider-keys into main 2026-10-31 00:04:41 -07:00
Frank 54a393edab Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-30 23:58:34 -07:00
Frank a5459b0b1e docs(media): the music library's own relayout, with the laws it earned 2026-10-30 23:58:34 -07:00
Frank b03931f7ce feat(media): a music relayout that files by the tag row Jellyfin matched
The music library is rips and discography packs whose filenames say nothing the
matcher needs. The authority is Jellyfin's own audio rows: one per track, with
the album and album-artist ids it carries in BaseItemProviders (the item JSON
holds an empty ProviderIds for albums in this build).

Grouping is what the tags assert, not where files sit: (album-artist tag, album,
year), case-folded. A compilation is one album whose tracks name many performers,
so it files once under Various Artists; two album-artist tags on one album name
are two albums. The folder spelling is the dominant one, because a name taken
from whichever row sorted first depends on row order and re-names albums that are
already filed.

An occupied path is never a destination: the executor refuses occupied targets,
so a plan that aims at one fails after the renames meant to clear it. A file
standing on its own target, or already carrying a collision suffix, is a keep -
otherwise the plan shuffles instead of being a fixed point.
2026-10-30 23:58:34 -07:00
dank 82f6ec55c2 Merge pull request 'feat(media): plan a TV relayout from the numbering Jellyfin already has' (#99) from feat/media-tv-relayout into main 2026-10-30 23:38:36 -07:00
dank 1a1dbac268 Merge pull request 'Jellyfin metadata providers: keys ride a host file the deployer installs' (#98) from feat/jellyfin-provider-keys into main 2026-10-30 22:48:21 -07:00
25 changed files with 4434 additions and 250 deletions

No files matched your search

+1 -1
View File
@@ -851,6 +851,6 @@ history.
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. |
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
+1
View File
@@ -0,0 +1 @@
entropy UUID=cd86c728-459e-4f3a-bcc2-41ecf84685ec /etc/entropy nofail,luks
+18
View File
@@ -0,0 +1,18 @@
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/nvme0n1p2 during curtin installation
/dev/disk/by-uuid/4f3731d2-fb7b-48e7-a79f-0e33cb8a4162 / ext4 defaults 0 1
# /boot/efi was on /dev/nvme0n1p1 during curtin installation
/dev/disk/by-uuid/183C-442D /boot/efi vfat defaults 0 1
#/dev/disk/by-label/archive-one /mnt/archive-one auto nosuid,nodev,nofail,x-gvfs-show 0 0
#/dev/disk/by-label/archive-two /mnt/archive-two auto nosuid,nodev,nofail,x-gvfs-show 0 0
# /dev/md0 is a LUKS container, not a filesystem; its ext4 is mounted via
# /dev/mapper/entropy on the next line. Mounting /dev/md0 directly fails.
#/dev/md0 /mnt/md0 ext4 auto,nofail,nosuid,nodev 0 0
/dev/mapper/entropy /mnt/entropy ext4 auto,nosuid,nofail,nodev,x-gvfs-show 0 0
/swapfile none swap sw 0 0
+25 -5
View File
@@ -32,6 +32,12 @@ cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
--force relaxes only the guards that are opinion (journal already rolled back,
paths outside --root); it never allows overwriting a file.
--metadata-view names the overlay the media servers read their library through
(default /var/lib/media-meta/view). The sidecar follow-up printed after an apply
or rollback is conditioned on that path being a mountpoint: where no view is
mounted the servers bind the array itself, the sidecars moved with the media,
and a reap order would point at a directory that is not there.
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
inside it: a log inside the library is a file Jellyfin will index.
"""
@@ -114,13 +120,24 @@ def read_plan(path, root, force):
return mkdirs, keeps, moves
# A relayout moves bytes on the raw array; the sidecars the media servers wrote
# live in the SSD overlay and stay at the old coordinates. Naming the follow-up
# here is the point of use for that fact — see the Media Metadata View page.
# The media servers may read the library through the SSD overlay view
# (scripts/media-view.sh). A relayout moves bytes on the raw array, so sidecars
# the servers wrote through that view stay at the old coordinates — naming the
# follow-up here is the point of use for that fact, but only where the view
# exists: where it is not mounted the servers bind the array itself, the
# sidecars moved with the media, and a reap order points at a directory that
# is not there. See the Media Metadata View page for the host's state.
METADATA_VIEW = "/var/lib/media-meta/view"
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
" these moves — next: scripts/media-view.sh reap")
def metadata_note(args):
"""Print the sidecar follow-up only when the overlay is really mounted."""
if os.path.ismount(args.metadata_view):
print(METADATA_HINT)
def do_apply(args, root, journal, mkdirs, keeps, moves):
"""Preflight everything, then folders, then renames. Never a partial start."""
problems = []
@@ -202,7 +219,7 @@ def do_apply(args, root, journal, mkdirs, keeps, moves):
f" {dirs_new} folders created, {len(keeps)} files stayed put")
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
f" --journal {journal.path} --root {root} --apply")
print(METADATA_HINT)
metadata_note(args)
return 0
@@ -386,7 +403,7 @@ def do_rollback(args, root, journal):
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
print(f"rolled back: {undone} files back where the census found them,"
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
print(METADATA_HINT)
metadata_note(args)
return 0
@@ -398,6 +415,9 @@ def main():
ap.add_argument("--journal")
ap.add_argument("--apply", action="store_true", help="actually change the library")
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
ap.add_argument("--metadata-view", default=METADATA_VIEW,
help="overlay the media servers read through; the sidecar"
" note prints only when this path is a mountpoint")
args = ap.parse_args()
root = os.path.abspath(args.root)
+425
View File
@@ -0,0 +1,425 @@
#!/usr/bin/env python3
r"""media-audiobooks.py — a plan that re-files the audiobook library into one folder per book.
media-audiobooks.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Plex reads this library and Jellyfin does not: its music scanner has no audiobook
concept, and the tree is indexed entirely by Plex, which matched 58,975 of the
62,879 files here into its Artist/Album/Track shape. So the authority is
`scripts/media-census.py audiobooks` — the BOOK (album row), its NARRATOR (that
book's single artist row) and its CHAPTERS (track rows) — and never the folder a
release pack made. The plan is:
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>.<ext>
... the same folder, " (2)" on the name, when two chapters
of one book carry the same name
What that buys is the one property both media servers' book scanners assume: a
folder holds exactly one book. Today 212 top-level folders are a mix of
author-named shelves and collection packs, and the books inside the packs are
split across `… [Part 1]`, `… [Part 2]` folders three or four deep.
Three laws here are the opposite of what a music shelf does, and each is forced
by what the rows actually hold:
* A chapter is NEVER renumbered. 7,556 of the 10,485 books contain at least one
chapter whose index Plex does not know, so a number written into a name would
be invented — and it would buy nothing, because both servers order playback by
the embedded track tag, not by the file name. A wrong number in a name is worse
than none: it is the number a human later trusts. A file already carrying its
own number keeps it, untouched.
* The shelf is the NARRATOR, not the author. Every book has exactly one artist row
and it is the narrator; the author survives only inside 10% of the album titles
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving the
author would mean parsing titles for one book in ten and inventing it for the
rest. One rule that always resolves beats a shelf that is half author and half
narrator; the author stays visible in the book folder's own name, which is the
title Plex shows.
* Nothing is guessed at. A chapter row with no book or no narrator parks — a
tag-scan placeholder (`[Unknown Album]`) names neither, since unrelated books
share it and filing on one merges them, and so does a chapter number the ripper
mistagged as the artist (`0.4`). An audio file Plex never matched parks, a
video container parks, and a loose file in a folder holding two books parks.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What a chapter file can be. .m4b is the audiobook container; the rest are the
# audio a pack ships when it was ripped as music.
AUDIO_EXT = {".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav", ".flac",
".wma", ".aiff", ".aif", ".m4v", ".aac", ".mka"}
# Never an audiobook, whatever the library it sits in says: Plex reads these as
# video, and renaming one into an audio extension would claim it is a chapter.
VIDEO_EXT = {".mkv", ".mp4", ".webm", ".avi", ".mov", ".mpg", ".mpeg", ".ts",
".m2ts", ".vob", ".ifo", ".wmv", ".iso", ".img", ".bdmv", ".ogm",
".divx"}
# Tag-scan placeholders. These strings are not identities: 1,106 chapters from
# 14 unrelated top-level folders carry the album `[Unknown Album]`, so filing on
# them would merge books that have nothing to do with each other — the exact
# shape this relayout exists to make impossible. Such a chapter parks with the
# path it has today and the fix belongs in its tags. `Various Artists` is NOT a
# placeholder: that is what a multi-cast read carries, and the one book here
# with 1,189 chapters under it is genuinely one book.
PLACEHOLDER_BOOK = {"[unknown album]", "unknown album", "no title", "untitled",
"[untitled]", "unknown title", "n/a", "-"}
PLACEHOLDER_NARRATOR = {"[unknown artist]", "unknown artist", "unknown author",
"<unknown>", "[unknown]", "unknown"}
def identity(value, placeholders, needs_letter=False):
"""The tag value Plex matched, when it names something; '' when it does not.
A trailing parenthesized group is stripped for the test only, because rippers
glue timestamps to the placeholder (`Unknown album (03/03/2015 21:39:55)`).
`needs_letter` is for a NARRATOR alone: a shelf is a person's name, and 272
chapters here carry a chapter number in the artist tag (`0.4`, `31`). The same
test must never judge a BOOK title, which may honestly be `1984`.
"""
v = (value or "").strip()
bare = re.sub(r"\s*\([^()]*\)\s*$", "", v).strip().casefold()
if bare in placeholders or (needs_letter and not any(c.isalpha() for c in v)):
return ""
return v
# The characters Plex documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the chapter row Plex matched
PLACED = {} # chapter path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
AUDIO_BY_DIR = defaultdict(set) # source dir -> chapter files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding chapters
FOLDERS_SEEN = {} # (dir, depth) -> book folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its chapters may sit
BOOKS = {} # (narrator, book, year) -> the book group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the rows mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing unprintable,
no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 60k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ------------------------------------------------------------------------ books
def group_books():
"""Every matched chapter into one group per (narrator, book, year).
The key is the identity Plex asserts in its own database, not the folder the
file sits in: one book arrives as four part folders inside a collection pack,
and two copies of one book in one pack are one book to Plex, whose album item
IS that narrator, title and year. The year is part of the key so a 1965
recording and a 2020 recording that share a title stay two books.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above Part 1/Part 2 sees both
narrator = identity(r["narrator"], PLACEHOLDER_NARRATOR, needs_letter=True)
book = identity(r["book"], PLACEHOLDER_BOOK)
if not book:
park(path, "a chapter row that names no book — Plex's album title is a "
"placeholder like [Unknown Album], so filing on it would "
"merge unrelated books onto one shelf")
continue
if not narrator:
park(path, "a chapter row that names no narrator — the artist row is a "
"placeholder like [Unknown Artist] or a chapter number a "
"ripper mistagged as the artist, so no shelf owns it")
continue
key = (narrator.casefold(), book.casefold(), (r["year"] or "").strip())
g = BOOKS.setdefault(key, {"narrators": [], "books": [], "year": key[2],
"members": [], "indexed": 0})
g["members"].append(path)
g["narrators"].append(narrator)
g["books"].append(book)
if str(r["track"] or "").isdigit():
g["indexed"] += 1
def book_folder(g):
"""The shelf Plex's own rows name: <narrator>/<book> (<year>). None when a
name is made of characters no path can hold — inventing a shelf is filing it
wrong, so those chapters park instead."""
narrator = safe(dominant(g["narrators"]))
book = safe(dominant(g["books"]))
if not narrator or not book:
return None
return os.path.join(ARGS.root, narrator,
f"{book} ({g['year']})" if g["year"] else book)
def place_books():
"""One folder per book; every chapter keeps the name it has today."""
for key in sorted(BOOKS):
g = BOOKS[key]
folder = book_folder(g)
if folder is None:
for path in sorted(g["members"]):
park(path, "a book or narrator Plex spells with characters no "
"folder name can hold")
continue
for path in sorted(g["members"]):
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
# The name the ripper wrote, unchanged: the track tag inside the file
# is what orders playback, and 72% of the books here have chapters no
# row numbers, so any number added here would be invented.
PLACED[path] = assign(path, folder, base, ext, "chapter",
"two chapters of one book carried the same name")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The book folders the chapters in `d`, or within `depth` directories below
it, were placed in. Memoised, because every loose file in a big collection
folder asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The book folder that owns this directory — the first place up the tree
where chapters sit, in the directory itself or within SUBTREE_DEPTH levels
below it, which is how a pack whose parts are `… [Part 1]`/`… [Part 2]` owns
the cover art lying above them. None when no chapter is near enough to say,
AMBIGUOUS when the chapters there belong to more than one book."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_other():
"""Everything that is not a chapter Plex matched: covers, booklets, EPUBs,
logs, checksums, playlists — and the clutter that is none of those."""
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
base = os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the audiobook library: Plex reads "
"it as video, and renaming it would claim it is a chapter")
elif ext in AUDIO_EXT:
park(path, "an audio file no chapter row names — it is not in the "
"library Plex matched, so nothing says which book it is")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a file whose folder holds more than one book, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one book folder would have carried two files of the same"
" sidecar name")
else:
park(path, "not an audiobook file, and no book's folder lies with"
" it to carry it")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Audiobook relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
unindexed = sorted(f"{safe(dominant(g['narrators']))}/{safe(dominant(g['books']))}"
for g in BOOKS.values()
if 0 < g["indexed"] < len(g["members"]))
if unindexed:
fh.write(f"{len(unindexed):>6} books have chapters Plex does not number. Their\n")
fh.write(" files keep the names they carry: playback order comes from the\n"
" track tag inside each file, and a number written here would be\n"
" invented for the chapters with no index. Fix it in the tags, not\n"
" in the names — Plex re-reads tags on a scan and never names files.\n\n")
fh.write(" Playlists move with their book and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"))
ap.add_argument("--enum", default="/tmp/audiobooks-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/plex-audiobooks.tsv",
help="the chapter rows scripts/media-census.py audiobooks wrote")
ap.add_argument("--out", default="/tmp/audiobooks-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/audiobooks-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
group_books()
place_books()
place_other()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a chapter row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+252 -30
View File
@@ -5,6 +5,7 @@ r"""Read-only census of a media library — the inputs a relayout plan is built
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
media-census.py jellyfin [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
media-census.py audiobooks [--root DIR] [--db PATH] [--out TSV]
Run it on the machine that mounts the library. Nothing here writes inside the
library and nothing is deleted; the only outputs are the TSVs below, and the
@@ -20,11 +21,28 @@ scripts/media-plan.py and scripts/media-shows.py read them instead of touching m
the titles Plex already matched — the authority over any filename guess
jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
<TAB> tmdb (see JF_COLS)
<TAB> tmdb <TAB> title <TAB> tvdb (see JF_COLS)
one row per episode Jellyfin has matched and numbered, its path mapped
from the container's view back to the host; `year` and `tmdb` are the
SERIES' values, `extra` is Jellyfin's own extra type when the row is
bonus material rather than an episode
from the container's view back to the host; `year`, `tmdb` and `tvdb`
are the SERIES' values, `title` is the episode title the row carries,
`extra` is Jellyfin's own extra type when the row is bonus material
rather than an episode
music file <TAB> artist <TAB> album <TAB> year <TAB> disc <TAB> track <TAB> title
<TAB> mb <TAB> performer (see MUSIC_COLS)
one row per track Jellyfin has tagged: the tag read is what the row IS,
so `artist` is the album artist (the track artist when that is empty) and
`album`/`year` are the album item's provider values when the folder is an
album item and the tags' otherwise; `mb` is the album's MusicBrainz
release group, `performer` the track artist the tags carry
audiobooks file <TAB> narrator <TAB> book <TAB> year <TAB> track <TAB> chapter
(see AB_COLS)
one row per chapter Plex matched under the audiobook root: `book` is the
album row Plex built from the files' own tags, `narrator` that book's
artist row — Plex keeps exactly one artist per book, which is why it, and
not an author parsed out of a title, names the shelf. `track` is empty
wherever Plex has no chapter number, which is most of this library.
`probe` is the slow step (every container is opened for its header, ~10 min for
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
@@ -66,13 +84,71 @@ left join metadata_items par on par.id = mi.parent_id
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
"""
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
JF_COLS = ["file", "series", "year", "season", "episode", "extra", "tmdb"]
JF_COLS = ["file", "series", "year", "season", "episode", "extra",
"tmdb", "title", "tvdb"]
# Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
"MediaBrowser.Controller.Entities.TV.Season",
"MediaBrowser.Controller.Entities.TV.Series")
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType, IsVirtualItem
from BaseItems where Type = ? and Path like ?""")
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType,
IsVirtualItem, Name from BaseItems where Type = ? and Path like ?""")
# The music tree is these two entity classes. The track row carries the tags
# Jellyfin read out of the file; the album row carries what the provider matched,
# which is the better folder name of the two.
JF_AUDIO = "MediaBrowser.Controller.Entities.Audio.Audio"
JF_ALBUM = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
MUSIC_COLS = ["file", "artist", "album", "year", "disc", "track", "title", "mb",
"performer"]
MUSIC_SQL = ("""select b.Path, b.AlbumArtists, b.Artists, b.Album, b.ProductionYear,
b.ParentIndexNumber, b.IndexNumber, b.Name, b.Id,
a.Name, a.ProductionYear, a.Id
from BaseItems b
left join BaseItems a on a.Id = b.ParentId and a.Type = ?
where b.Type = ? and b.Path like ?""")
# The album's identity, in preference order. A release group is the album; a bare
# release id is one reissue of it.
MUSIC_PROVIDERS = ("MusicBrainzReleaseGroup", "MusicBrainzAlbum")
# Plex's audiobook tree is the same Artist/Album/Track shape as its music tree:
# the leaf (10) is a CHAPTER, its parent (9) is the BOOK, and that book's artist
# (8) is the NARRATOR. The track's own `index` is exported because the exceptions
# report counts how many books have chapters Plex does not number — the reason the
# relayout refuses to invent numbers into file names.
PLEX_ARTIST, PLEX_ALBUM, PLEX_TRACK = 8, 9, 10
AB_COLS = ["file", "narrator", "book", "year", "track", "chapter"]
AB_SQL = f"""
select p.file, coalesce(ar.title,''), coalesce(al.title,''), coalesce(al.year,0),
coalesce(mi."index",-1), coalesce(mi.title,'')
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
join metadata_items al on al.id = mi.parent_id and al.metadata_type = {PLEX_ALBUM}
left join metadata_items ar on ar.id = al.parent_id
and ar.metadata_type = {PLEX_ARTIST}
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
and al.deleted_at is null
"""
def open_plex_ro(db):
"""Plex's library, read the way Plex writes it.
`immutable=1` tells sqlite the file never changes, so it skips the
write-ahead log: every row a scan has committed but not yet checkpointed
is invisible, and reading a live database that way can fail outright as
"database disk image is malformed". Plain ro reads the WAL through its
-shm; immutable is only the fallback for a database whose -shm cannot be
read. The same law scripts/media-verify.py states.
"""
last = None
for opt in ("mode=ro", "mode=ro&immutable=1"):
try:
return sqlite3.connect(f"file:{db}?{opt}", uri=True)
except sqlite3.Error as e:
last = e
sys.exit(f"cannot open the Plex database read-only ({db}): {last}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
def enumerate_lib(root, out):
@@ -173,13 +249,7 @@ def probe(enum, out, jobs):
def plex(root, db, out):
"""Plex's matched titles for files under root, from its DB, read-only."""
uri = f"file:{db}?mode=ro&immutable=1"
try:
con = sqlite3.connect(uri, uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
con = open_plex_ro(db)
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(PLEX_COLS)
@@ -200,9 +270,10 @@ def jellyfin(root, db, container, out):
The numbering is what matters, not the name: a release group's file name says
nothing about which episode a file is, and Jellyfin has already answered that
question against a provider. Season/episode come from the episode row; the
series name, its year and its TMDb id come from the Series row the episode
climbs to, because those are the only place Jellyfin keeps them.
question against a provider. Season/episode and the episode's own title come
from the episode row; the series name, its year and its TMDb and Tvdb ids come
from the Series row the episode climbs to, because those are the only place
Jellyfin keeps them.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
@@ -223,21 +294,38 @@ def jellyfin(root, db, container, out):
" where Type in (?,?,?)", JF_TV):
node[i] = (pid, typ, name, year, data)
# Jellyfin 12 keeps provider ids in their own table; older builds kept them in
# the row's Data blob. Both shapes are read, because the tvdb id the series
# naming depends on is wherever this build happened to write it.
prov = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in ('Tmdb', 'Tvdb')"):
prov[(iid, pid)] = val
def provider_of(item_id, want):
got = prov.get((item_id, want))
if got:
return str(got)
try:
blob = json.loads(node[item_id][4] or "{}").get("ProviderIds") or {}
except (KeyError, ValueError, TypeError):
return ""
return str(blob.get(want, "") or "")
def series_of(item_id):
"""Climb episode -> season -> series; empty when the chain is broken."""
for _ in range(4):
if item_id not in node:
return "", "", ""
pid, typ, name, year, data = node[item_id]
return "", "", "", ""
pid, typ = node[item_id][0], node[item_id][1]
if typ.endswith(".Series"):
try:
tmdb = (json.loads(data or "{}")
.get("ProviderIds") or {}).get("Tmdb", "")
except ValueError:
tmdb = ""
return name or "", year or "", tmdb
return (node[item_id][2] or "", node[item_id][3] or "",
provider_of(item_id, "Tmdb"), provider_of(item_id, "Tvdb"))
item_id = pid
return "", "", ""
return "", "", "", ""
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
@@ -248,21 +336,134 @@ def jellyfin(root, db, container, out):
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(JF_COLS)
for iid, path, idx, pidx, extra, virt in con.execute(JF_SQL,
(JF_TV[0], prefix + "%")):
for iid, path, idx, pidx, extra, virt, epname in con.execute(
JF_SQL, (JF_TV[0], prefix + "%")):
if virt or not path or not path.startswith(prefix):
skipped += 1 # a virtual item is a variant, not a file
continue
sname, syear, tmdb = series_of(iid)
sname, syear, tmdb, tvdb = series_of(iid)
w.writerow([root + "/" + path[len(prefix):],
sname.replace("\t", " "), syear,
"" if pidx is None else pidx,
"" if idx is None else idx,
(extra or "").replace("\t", " "), tmdb])
(extra or "").replace("\t", " "), tmdb,
(epname or "").replace("\t", " "), tvdb])
written += 1
print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
def music(root, db, container, out):
"""The tracks Jellyfin has already tagged, for the library at root.
Jellyfin reads music metadata out of the file itself, so an Audio row IS the
tag read, done once, by the vendor's own parser: its album-artist, album,
year, disc and track columns are what the embedded tags say and its Name is
the track title. That makes this export the whole authority for a music
relayout — the path a ripper chose is not evidence of anything, and the
folder Jellyfin put a track under is often a release pack, not an album.
Two gaps this export answers from the row rather than the tree. Many tracks
sit under a folder that is not an album item at all (a disc folder, or a
mixed one), so the album name and year fall back to the tags when the album
row is absent. And an album's MusicBrainz ids are not in the row's own Data
— the current schema leaves that empty for music — they are rows of
BaseItemProviders keyed by the item's Id.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
"It is a WAL database: copy it with its -wal and -shm beside it and"
" point --db at the copy. Never sudo it — the data dir belongs to"
" the container's owner, and rootless storage is per-user.")
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
need = {"Id", "ParentId", "Type", "Path", "Name", "IndexNumber",
"ParentIndexNumber", "ProductionYear", "Album", "AlbumArtists",
"Artists"}
if not need <= have:
sys.exit(f"{db} is not a Jellyfin library database with music columns"
f" (BaseItems has no {','.join(sorted(need - have))})")
# Provider ids, best identity first: the release group is the album, a bare
# release id is one reissue of it and never overrides a group.
mb = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
rank = {p: i for i, p in enumerate(MUSIC_PROVIDERS)}
best = {}
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in (?, ?)", MUSIC_PROVIDERS):
if iid not in best or rank[pid] < best[iid][0]:
best[iid] = (rank[pid], val)
mb = {k: v[1] for k, v in best.items()}
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
if not container:
container = "/data/" + os.path.basename(os.path.normpath(root))
prefix = container.rstrip("/") + "/"
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(MUSIC_COLS)
for (path, aal, arts, tag_album, tag_year, disc, track, name, iid,
alb_name, alb_year, alb_id) in con.execute(
MUSIC_SQL, (JF_ALBUM, JF_AUDIO, prefix + "%")):
if not path or not path.startswith(prefix):
skipped += 1 # a row for a file outside this library
continue
# The album artist owns the folder; the track artist is the fallback
# for the compilations whose album-artist tag was left empty.
w.writerow([root + "/" + path[len(prefix):],
((aal or "").strip() or (arts or "").strip()),
(alb_name or tag_album or "").replace("\t", " "),
alb_year or tag_year or "",
"" if disc is None else disc,
"" if track is None else track,
(name or "").replace("\t", " "),
mb.get(alb_id or "") or mb.get(iid or "") or "",
(arts or "").replace("\t", " ")])
written += 1
print(f"tracks={written} outside-root-or-blank={skipped} -> {out}")
def audiobooks(root, db, out):
"""The chapters Plex matched inside the audiobook library, read-only.
Plex stores HOST paths here (unlike Jellyfin, which stores the container's
view), so no prefix mapping is needed: the part's `file` is the path the
operator has today. Rows for files outside `root` are skipped, because one
Plex library DB carries every library on the box.
The chapter's `index` is exported as an empty field when Plex has none. That
is not a gap to fill in here: three quarters of the books on this box contain
at least one un-indexed chapter, so the relayout that reads this file must
not renumber anything from it.
"""
con = open_plex_ro(db)
tables = {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}
missing = {"media_parts", "media_items", "metadata_items"} - tables
if missing:
sys.exit(f"{db} is not a Plex library database (no "
f"{', '.join(sorted(missing))})")
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(AB_COLS)
for file, narrator, book, year, track, chapter in con.execute(
AB_SQL, (root.rstrip("/") + "/%",)):
if not file or not file.startswith(root.rstrip("/") + "/"):
skipped += 1 # a row for a file outside this library
continue
w.writerow([file.replace("\t", " "), narrator.replace("\t", " "),
book.replace("\t", " "), "" if not year else year,
"" if track is None or int(track) < 0 else track,
chapter.replace("\t", " ")])
written += 1
print(f"chapters={written} outside-root={skipped} -> {out}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
@@ -289,6 +490,23 @@ def main():
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
m = sub.add_parser("music",
help="export the tracks Jellyfin has tagged for the root")
m.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"),
help="music library root on the host")
m.add_argument("--db", default=JF_DB)
m.add_argument("--container", default=None,
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
m.add_argument("--out", default="/tmp/jellyfin-music.tsv")
b = sub.add_parser("audiobooks",
help="export the chapters Plex matched for the audiobook root")
b.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"),
help="audiobook library root on the host")
b.add_argument("--db", default=PLEX_DB)
b.add_argument("--out", default="/tmp/plex-audiobooks.tsv")
a = ap.parse_args()
if a.cmd == "enumerate":
enumerate_lib(a.root, a.out)
@@ -296,6 +514,10 @@ def main():
probe(a.enum, a.out, a.jobs)
elif a.cmd == "plex":
plex(a.root, a.db, a.out)
elif a.cmd == "music":
music(a.root, a.db, a.container, a.out)
elif a.cmd == "audiobooks":
audiobooks(a.root, a.db, a.out)
else:
jellyfin(a.root, a.db, a.container, a.out)
+445
View File
@@ -0,0 +1,445 @@
#!/usr/bin/env python3
r"""media-music.py — a plan that re-files the music library into Jellyfin's own shape.
media-music.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Jellyfin's music contract (https://jellyfin.org/docs/general/server/media/music/)
is not the movie one. Jellyfin reads the metadata out of the file, so a folder is
an ALBUM and nothing else, every disc of an album belongs in that one folder, the
tags decide the track order, and a file name only matters when it carries a
character known to break playback (`< > : " / \ | ? *`) or when a lyric file has
to match the basename of the track it belongs to. The plan is therefore:
Music/<album artist>/<Album> (<year>)/NN - <Title>.<ext>
... the same folder, D.NN - <Title> when the album has discs
Music/Various Artists/... when its tracks name
more than one artist
The authority is the tag row Jellyfin already read — `scripts/media-census.py
music` — never the release-pack folder a ripper made. A file with no row is not
guessed at: it parks under `<root>/_unsorted` with the relative path it has today.
The rules below exist because of what a downloaded discography pack actually
contains:
* A cuesheet beside exactly one audio file means the album is one image with a
track list inside it. That file keeps its own name: numbering it `01` would
claim it is one track, and Jellyfin shows one track either way.
* Sidecars — images, cuesheets, EAC logs, checksums, booklets, playlists —
follow the album the audio in their own folder belongs to. A folder holding
two albums gives a sidecar no single owner, so it parks instead of being
guessed into one.
* A video container (.mkv, .vob, .ifo, …) inside a music library is not music.
Renaming it to .mka/.m4a would tell Jellyfin to index video as audio, so it
parks for a human decision.
* Playlists move with their album and their CONTENTS are never rewritten: the
paths inside an .m3u go stale. Editing a file the operator owns to keep a
convenience file tidy is not this tool's job.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What Jellyfin will index as music. .mp4/.mkv/.webm are deliberately NOT here:
# the vendor doc says an audio-only stream in those containers is never read as
# music, and this tool renames for a reason it can prove, not to guess at one.
AUDIO_EXT = {".flac", ".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav",
".wma", ".aiff", ".aif", ".ape", ".dsf", ".dff", ".wv", ".mka",
".mpc", ".aac"}
VIDEO_EXT = {".mkv", ".vob", ".ifo", ".bup", ".mpg", ".mpeg", ".avi", ".mov",
".ts", ".m2ts", ".wmv", ".webm", ".mp4", ".m4v", ".iso", ".img",
".bdmv", ".divx", ".ogm"}
# A lyric has to carry the basename of its track, so it is named after the audio
# file and not after itself. .txt counts only when its stem matches an audio stem:
# a loose .txt in an album folder is a note.
LYRIC_EXT = {".lrc", ".elrc", ".txt"}
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tiff"}
# The album-artist strings that mean "compilation", so such an album is not filed
# under one artist who does not own it.
VARIOUS = {"various", "various artists", "unknown", "unknown artist", "unknow"}
# The characters Jellyfin documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the tag row
PLACED = {} # audio path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
CUE_DIRS = set() # source dirs holding a .cue
AUDIO_BY_DIR = defaultdict(set) # source dir -> audio files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding audio
FOLDERS_SEEN = {} # (dir, depth) -> album folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its audio may sit
ALBUMS = {} # (artist, album, year) -> the album group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the tags mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing
unprintable, no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 90k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ---------------------------------------------------------------------- albums
def group_albums():
"""Every tagged track into one group per (album-artist tag, album, year).
The key is the identity the tags assert, not the folder the file sits in: one
album arrives as three discs in three folders, and two copies of an album in
one pack are one album to Jellyfin, whose album item is that name, artist and
year. The artist in the key is the ALBUM-artist tag and never the performer,
because a compilation is one album whose tracks name many performers — keying
on the performer would file one album once per artist.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above CD1/CD2 sees both
tag = (r["artist"] or "").strip()
album = (r["album"] or "").strip()
performer = (r["performer"] or "").strip()
if not album:
park(path, "a track no album names")
continue
if not tag and not performer:
park(path, "a track no artist names")
continue
key = (tag.casefold(), album.casefold(), (r["year"] or "").strip())
g = ALBUMS.setdefault(key, {"tags": [], "albums": [], "year": key[2],
"members": [], "performers": set(),
"mb": set()})
g["members"].append(path)
g["tags"].append(tag)
g["albums"].append(album)
g["performers"].add(performer or tag)
if r.get("mb"):
g["mb"].add(r["mb"])
def folder_artist(g):
"""Whose shelf the album sits on. The album-artist tag owns it when it names
one; when the tags name only performers, a single performer names the album
and several make it a compilation, which belongs to no one of them. The key is
case-folded, so the spelling that reaches the folder is the one the tracks
mostly carry — see dominant()."""
tags = [t for t in g["tags"] if t]
if tags:
name = dominant(tags)
return "Various Artists" if name.casefold() in VARIOUS else name
if len({p.casefold() for p in g["performers"]}) == 1:
return dominant(g["performers"])
return "Various Artists"
def album_folder(g):
name = safe(dominant(g["albums"])) or "Untitled"
return os.path.join(ARGS.root, safe(folder_artist(g)) or "Unknown Artist",
f"{name} ({g['year']})" if g["year"] else name)
def is_cue_image(path):
"""A folder holding this one audio file and a cuesheet is an album image."""
d = os.path.dirname(path)
return len(AUDIO_BY_DIR.get(d, ())) == 1 and d in CUE_DIRS
def number_of(value, width):
return f"{int(value):0{width}d}" if str(value or "").isdigit() else ""
def place_albums():
for key in sorted(ALBUMS):
g = ALBUMS[key]
folder = album_folder(g)
discs = {g0["disc"] for g0 in (ROWS[p] for p in g["members"]) if g0["disc"]}
tracks = [int(ROWS[p]["track"]) for p in g["members"]
if str(ROWS[p]["track"] or "").isdigit()]
# More than one disc, or a disc numbered past the first: the track number
# alone cannot name a file uniquely, so the disc goes in front of it.
multi = len(discs) > 1 or any(str(d).isdigit() and int(d) > 1 for d in discs)
width = 3 if multi or (max(tracks, default=0) > 99) else 2
for path in sorted(g["members"]):
r = ROWS[path]
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
if is_cue_image(path):
name = safe(base) or "image"
else:
title = safe(r["title"] or base) or "untitled"
num = number_of(r["track"], 2 if multi else width)
if multi and num:
num = f"{r['disc'] or '1'}.{num}"
name = f"{num} - {title}" if num else title
PLACED[path] = assign(path, folder, name, ext, "track",
"two tracks claimed one name inside one album")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The album folders the audio in `d`, or within `depth` directories below it,
was placed in. Memoised, because every loose file in a big discography folder
asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The album folder that owns this directory — the first place up the tree
where audio sits, in the directory itself or within SUBTREE_DEPTH levels below
it, which is how a box with its discs in CD1/CD2 owns the cover lying above
them. None when no audio is near enough to say, AMBIGUOUS when the audio there
belongs to more than one album."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_sidecars():
"""Everything that is not a tagged track: lyrics, images, cuesheets, logs,
booklets, playlists — and the clutter that is none of those."""
# The lyric rule needs the audio's stem as it is named TODAY, before the plan
# renames it, because that is what the sidecar's own name was copied from.
stem = {(d, os.path.splitext(os.path.basename(a))[0]): a
for d, audios in AUDIO_BY_DIR.items() for a in audios}
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d, base = os.path.dirname(path), os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the music library: Jellyfin will "
"not read it as music, and renaming it would claim it is")
elif ext in AUDIO_EXT:
park(path, "an audio file no provider row names — its tags are not in "
"the library Jellyfin has matched")
elif ext in (LYRIC_EXT | IMAGE_EXT) and (d, base) in stem:
# A lyric, or a picture named after its track: it follows that track
# and takes the name the track now has, or Jellyfin cannot find it.
audio = stem[(d, base)]
dst = PLACED.get(audio)
if dst:
folder, name = os.path.dirname(dst), os.path.splitext(
os.path.basename(dst))[0]
assign(path, folder, name, ext, "lyric",
"a sidecar whose name collided with another in its album")
else:
park(path, "a sidecar of a track the plan could not place")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a sidecar whose folder holds more than one album, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one album folder would have carried two files of the same "
"sidecar name")
else:
park(path, "not music, or music with nothing beside it to belong to")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Music relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
no_year = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if not g["year"])
if no_year:
fh.write(f"{len(no_year):>6} albums carry no year in any tag, so their folders\n"
" are named without one; Jellyfin still matches them.\n")
fh.write(" " + ", ".join(no_year[:300]) + "\n\n")
clash = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if len(g["mb"]) > 1)
if clash:
fh.write(f"{len(clash):>6} albums put one folder name on more than one\n"
" MusicBrainz release group — different releases, one name.\n"
" They share the folder Jellyfin's own album item names.\n")
fh.write(" " + ", ".join(clash[:200]) + "\n\n")
fh.write(" Playlists move with their album and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"))
ap.add_argument("--enum", default="/tmp/music-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/jellyfin-music.tsv",
help="the tag rows scripts/media-census.py music wrote")
ap.add_argument("--out", default="/tmp/music-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/music-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
for path, (_size, ext) in HELD.items():
if ext == ".cue":
CUE_DIRS.add(os.path.dirname(path))
group_albums()
place_albums()
place_sidecars()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a tag row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+35
View File
@@ -0,0 +1,35 @@
# The operator's tvdb id map: `folder <TAB> tvdb id <TAB> note`.
#
# These are the pack folders whose show tvdb's NAME SEARCH cannot reach: the
# record exists, but tvdb titles it in the show's own language and the English
# name rides in aliases the search never reads. Without a row here the folder
# keeps the release folder's words and no episode title; with one, both matchers
# read `[tvdbid-NNNN]` and the files rename to tvdb's own name and titles.
#
# Every id was read from tvdb's own record (`/series/<id>/extended`) on
# 2026-10-31, and the note states what that record says plus the neighbouring
# record a name search would have picked instead — the row is checkable without
# re-searching, because one wrong id renames a whole pack to the wrong show. The
# plan refuses a line whose id is not a number and lists every folder the map
# touched in the exceptions file.
#
# Folders the name search DOES reach (the EXACT and PUNCT classes of the proposal
# table) stay off this file on purpose: a map row would hide a live lookup behind
# a number nobody re-checks.
[Anime Time] Pokemon Complete Series (Season 01-25+Movies+Specials+Shorts+OST) [DVD+BD][1080p][HEVC 10bit x265][AAC] 76703 tvdb: ポケットモンスター (1997), the umbrella record for every era this pack spans; 1268 files
[Anime Time] One Piece (0001-1071+Movies+Specials) [BD+CR] [Dual Audio] [1080p][HEVC 10bit x265][AAC][Multi Sub] 81797 tvdb: ワンピース (1999), alias ONE PIECE; the bare alias "One Piece" belongs to 392276, the 2023 live action; 1072 files
[Judas] Naruto (Complete Series) [BD 1080p][HEVC x265 10bit][Multi-Audio][Multi-Subs] 78857 tvdb: NARUTO-ナルト- (2002), alias "Naruto" carries the full-width dashes; 220 files
Star Trek Deep Space 9 Deluxe DVD Boxset + Extras DVDRip TSV 72073 tvdb: Star Trek: Deep Space Nine (1993), alias "Star Trek Deep Space 9"; the box set's own words match nothing; 279 files
Top Gear UK 1-17 74608 tvdb: Top Gear (2002), no "UK" alias — the US version is 199051; 131 files
Black Clover 001 331753 tvdb: ブラッククローバー (2017), which holds NO alias at all; 378762 is Mugyutto! Black Clover (2019), a different show; 108 files
Shin Chan - Complete Eng Dub Series 2160p Ai Upscaled -Mesc 437356 tvdb: Shin Chan (2006) — the English dub record this pack is; 79654 is the 1992 Japanese original; 78 files
Fairy Tail (2009-2014) (Seasons 1-8 + OVAs) (BD 1080p)(HEVC x265 10bit)(Dual-Audio)(Eng-Subs)-Judas[TGx] 114801 tvdb: フェアリーテイル (2009), alias FAIRY TAIL; 410031 is FAIRY TAIL 100 NEN QUEST (2024), the sequel; 59 files
(CBB) Tokyo Ghoul - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-10bit-x265) 281249 tvdb: 東京喰種トーキョーグール (2014) — :re and √A are seasons of THIS record, confirmed by search; 58 files
[Anime Time] Demon Slayer - Kimetsu no Yaiba (S01+S02+S03+S04+Mugen Train Arc+Movie) [BD] [Dual Audio][1080p][HEVC 10bit x265][AAC][Eng Sub] 348545 tvdb: 鬼滅の刃 (2019), alias "Demon Slayer: Kimetsu no Yaiba" — the folder's hyphen splits it differently; 56 files
(CBB) Tengen Toppa Gurren Lagann - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-x265-10bit) 80096 tvdb: 天元突破グレンラガン (2007), alias Tengen Toppa Gurren Lagann; 51 files
Fullmetal Alchemist - Brotherhood 85249 tvdb: 鋼の錬金術師 FULLMETAL ALCHEMIST (2009), alias "Full Metal Alchemist Brotherhood" — spaced, unlike this folder; 64 files
[NewbSubs] Attack on Titan~Shingeki no Kyojin Series (1080p Blu-ray x265 Dual Audio) 267440 tvdb: 進撃の巨人 (2013), aliases carry "Attack on Titan" and "Shingeki no Kyojin", joined here by a tilde; 27 files
Kimetsu no Yaiba 348545 the same record as the Anime Time pack above: two release folders, ONE show; 12 files
Rick and Morty The Anime 420279 tvdb: リック・アンド・モーティ:ジ・アニメ (2024), no English alias on the record; 10 files
[Anime Time] Dorohedoro (Season 1+OVA+Extra) [Dual Audio][BD][1080p][HEVC 10bit x265][AAC][Eng Sub] 370761 tvdb: ドロヘドロ (2020), alias Dorohedoro; 9 files
Ken Il Guerriero HOKUTO NO KEN 79156 tvdb: 北斗の拳 (1984), alias "Hokuto no Ken"; the Italian title matches nothing and 465216 is the 2026 remake; 5 files
Can't render this file because it contains an unexpected character in line 20 and column 182.
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env python3
r"""Ask tvdb what it calls the shows a plan renamed WITHOUT a tvdb id.
media-shows-tvdb.py [--plan TSV] [--root DIR] [--park DIR] [--map TSV]
[--out TSV] [--credentials FILE]
`media-shows.py` renames a series to tvdb's own words only when an id backs the
name; everything else keeps the metadata provider's words and no episode title,
and the plan's mkdir rows are where those folders show up. This script turns that
list into a decision sheet: one row per destination series folder that carries no
`[tvdbid-NNNN]`, with what tvdb's search answers for it and a verdict that says
HOW MUCH was guessed to get there.
It changes nothing. The sheet is for the operator; the file that changes
behaviour is `scripts/media-shows-tvdb-ids.tsv`, whose rows he approves. That is
the loop: plan -> this sheet -> approved map rows -> re-plan.
Verdicts, weakest last:
EXACT tvdb's own name equals the folder's words as they stand.
PUNCT it equals them only after release junk is stripped (pack numbers,
codecs, group names, `Complete Series`); the difference is spelling,
not identity.
NEAR tvdb answered something whose name is NOT the query. The best hit is
printed so a human can judge it, and it is never applied by anything.
NONE nothing left to ask with, or no answer at all.
Acceptance is deliberately unforgiving — a hit counts only when tvdb's name
equals the query with every non-alphanumeric dropped from both sides, and the
series' year still has to fall inside any year span the folder names. The
cleaner is the only thing that got smarter over the passes this tool replaces;
the acceptance test never moved, because a wrong id renames a whole pack to the
wrong show. A query that needs the cleaner is reported as PUNCT for exactly that
reason: the machine stripped words, so a human reads the row.
Read-only against tvdb. The credential file is the planner's own
(`api_key=`, optional `pin=`), read here and never printed.
"""
import argparse
import collections
import importlib.util
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
formatter_class=argparse.RawDescriptionHelpFormatter)
ARGS.add_argument("--plan", default="/tmp/tv-plan.tsv",
help="the plan media-shows.py just wrote (its mkdir rows are "
"the folders this sheet asks about)")
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
help="the TV library the plan renames")
ARGS.add_argument("--park", default="",
help="the review pile inside it (its folders are not shows)")
ARGS.add_argument("--map",
default=os.path.join(os.path.dirname(os.path.abspath(__file__)),
"media-shows-tvdb-ids.tsv"),
help="the operator's approved map; its folders are reported as "
"MAPPED and never asked again")
ARGS.add_argument("--credentials",
default=os.path.expanduser("~/.config/secrets/tvdb"),
help="file with an api_key= line (and a pin= for a user-issued "
"key); mode 600, and never printed")
ARGS.add_argument("--out", default="/tmp/tvdb-proposal.tsv",
help="the decision sheet")
ARGS = ARGS.parse_args()
ROOT = os.path.abspath(ARGS.root)
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
_spec = importlib.util.spec_from_file_location(
"eng", os.path.join(os.path.dirname(os.path.abspath(__file__)),
"media-titles.py"))
eng = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(eng)
# Release metadata, not the show: everything a scene folder name says about the
# ENCODING, the packaging or the languages, which tvdb has never heard of.
JUNK = re.compile(
r"complete|series|dual|multi|audio|sub|subs|multisub|dualaudio|"
r"\d{3,4}p|1080p|720p|2160p|480p|4k|hd|hdtv|cam|hdcam|telesync|x264|x265|hevc|avc|"
r"\d+bit|bit|bluray|blu-?ray|bdrip|web-?rip|web-?dl|webrip|hdtvrip|"
r"dvdrip|dvdr|dvdscr|tsr|tsv|mkv|avi|mp4|aac|ac3|dts|eac3|flac|ddp?5[.1]1|"
r"bd|dvd|rip|boxset|box|season|seasons|episodes?|specials?|shorts|ost|"
r"ova|ovas|oad|repack|proper|internal|mi?xed|budgetbits|deluxe|remux|"
r"remastered|uncut|collector|edition|extras?|volume|volumes|part|"
r"disc|disk|cd|ntsc|pal|region|fixed|patched|proof|english|esp|"
r"espagnol|french|vostfr|subtitul\w*|spanish|german|italian|ita|eng|fre|spa|lat|"
r"upscaled?|ai|dd|movie|movies",
re.I)
GROUP = re.compile(r"KONTRAST|TGx|Judas|Silence|FreetheFilms|NTb|DEADPOOL|Xploit|"
r"SM737|TSA|3OISL3R|PLAYWEB|EPiC|FLUX|HONEY|Zaxton|Cca3rW|"
r"monikeres|frank|hdh|psi|rtfpemaker|cbm|web|asser|astro|"
r"yfn|uindex|www|org|com|net|mesc|ataraxiaprime|seph\d*|"
r"[a-z]+subs?|edge", re.I)
BRACKET = re.compile(r"[\[({]([^\])}]*)[\])}]")
NUM_RANGE = re.compile(r"(?<!\w)\d{1,4}\s*(?:-|–|to|through)\s*\d{1,4}(?!\w)", re.I)
INSIDE = re.compile(r"all|complete|full|dual|multi|audio|sub|subs|ova|ovas|"
r"movies?|episodes?|extras?|bonus|pack|remux|region\s*\d*|"
r"bd|dvd|cd|ntsc|pal|hr-?dvd", re.I)
def drop_bracket(m):
"""A bracket says nothing when its words are junk; keep it when they aren't."""
toks = [t for t in re.split(r"[\s_.,+~&]+", m.group(1)) if t]
if not toks or all(INSIDE.fullmatch(t) or JUNK.fullmatch(t)
or GROUP.fullmatch(t)
or re.fullmatch(r"\d+[a-z]?", t, re.I)
or re.fullmatch(r"(19|20)\d\d", t) for t in toks):
return " "
return " " + m.group(1) + " "
def clean(name):
"""The folder's words with everything about the RELEASE taken out, so what is
left is what a person would type into tvdb. `Star Trek Deep Space 9 Deluxe
DVD Boxset + Extras DVDRip TSV` becomes `Star Trek Deep Space 9`; `SNL 16-20`
becomes `SNL`."""
name = re.sub(r"^\s*www\b.*?\b(org|com|net)\b\s*[-–]\s*", " ", name, flags=re.I)
name = re.sub(r"\bS\d{1,2}([ .Ee]\d+)?[-–Ee\d.]*", " ", name)
name = re.sub(r"\b\d{1,2}[xX]\d{1,3}\b", " ", name)
name = NUM_RANGE.sub(" ", name)
name = re.sub(r"(19|20)\d\d(?:\s*[-/–]\s*(?:19|20)\d\d)?", " ", name)
name = re.sub(r"\bby\s+[A-Za-z0-9_.-]+$", " ", name)
name = BRACKET.sub(drop_bracket, name)
name = re.sub(r"[\[()\]:,;!?*]", " ", name)
toks = re.split(r"[\s_.\-+~&]+", name)
out = []
for i, t in enumerate(toks):
if not t or JUNK.fullmatch(t) or GROUP.fullmatch(t):
continue
if re.fullmatch(r"\d+", t):
after = sum(1 for x in toks[i + 1:] if x
and not JUNK.fullmatch(x) and not GROUP.fullmatch(x)
and not re.fullmatch(r"\d+", x))
# A pack index (`001`) and a lead-in number over a real title
# (`01 Charles Barkley`) are noise; `30 Rock`, `24`, `Deep Space 9`
# are the title itself.
if len(t) >= 3 or (not out and after >= 2):
continue
out.append(t)
while out and out[-1].lower() in ("and", "with", "the", "a", "an", "to", "of"):
out.pop()
return " ".join(out).strip()
def years(name):
return [int(y) for y in re.findall(r"\b(19\d\d|20\d\d)\b", name)]
def bare(s):
return re.sub(r"[^a-z0-9]", "", s.lower())
CRED = {}
try:
for _line in open(ARGS.credentials):
if "=" in _line:
k, v = _line.strip().split("=", 1)
CRED[k.strip()] = v.strip()
except OSError:
print(f"tvdb: no credentials at {ARGS.credentials} — the sheet would be all "
"NEAR and NONE, which is not a decision sheet", file=sys.stderr)
sys.exit(2)
if not CRED.get("api_key"):
print(f"tvdb: no api_key in {ARGS.credentials}", file=sys.stderr)
sys.exit(2)
TOKEN = {"jwt": ""}
def call(path):
"""One tvdb v4 GET, logging in again once when the JWT expires mid-run."""
for attempt in (1, 2):
if not TOKEN["jwt"]:
body = {"apiKey": CRED["api_key"]}
if CRED.get("pin"):
body["token"] = CRED["pin"]
req = urllib.request.Request("https://api4.thetvdb.com/v4/login",
data=json.dumps(body).encode(),
headers={"Content-Type":
"application/json"})
TOKEN["jwt"] = json.load(
urllib.request.urlopen(req, timeout=30))["data"]["token"]
try:
req = urllib.request.Request("https://api4.thetvdb.com/v4" + path,
headers={"Authorization":
"Bearer " + TOKEN["jwt"]})
return json.load(urllib.request.urlopen(req, timeout=30))["data"]
except urllib.error.HTTPError as e:
if e.code == 401 and attempt == 1:
TOKEN["jwt"] = ""
continue
raise
MAPPED = {}
if ARGS.map and os.path.exists(ARGS.map):
for _line in open(ARGS.map):
_line = _line.rstrip("\n")
if not _line.strip() or _line.startswith("#"):
continue
cols = _line.split("\t")
if len(cols) > 1 and cols[1].strip().isdigit():
MAPPED[cols[0].strip()] = cols[1].strip()
# The folders the plan created inside the library, with the file count each one
# will hold. The review pile is not a show: its files belong to no identity, and
# naming the pile would put one junk series on the sheet for all of them.
shows = collections.Counter()
for line in open(ARGS.plan).read().splitlines()[1:]:
parts = line.split("\t")
if len(parts) < 4 or parts[0] != "mkdir":
continue
dst = os.path.abspath(parts[2])
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
continue
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
if top.startswith((".", "_")):
continue
if "[tvdbid-" in top:
continue # an id already backs it
shows.setdefault(top, 0)
for line in open(ARGS.plan).read().splitlines()[1:]:
parts = line.split("\t")
if len(parts) < 4 or parts[0] != "move":
continue
dst = os.path.abspath(parts[2])
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
continue
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
if "[tvdbid-" in top:
continue
shows[top] += 1
tally = collections.Counter()
rows = []
for folder in sorted(shows, key=lambda f: (-shows[f], f)):
files, tag, note = shows[folder], "NONE", ""
if folder in MAPPED:
rows.append([files, "MAPPED", folder, "", "", "", MAPPED[folder],
"the operator's map already states this id"])
tally["MAPPED"] += 1
continue
query, untouched = clean(folder), clean(folder) == folder
ys = years(folder)
if not query:
rows.append([files, "NONE", folder, "", "", "", "",
"nothing is left of the folder's name once release junk"
" is gone"])
else:
try:
hits = call("/search?query=" + urllib.parse.quote(query)
+ "&type=series") or []
except Exception as e: # noqa: BLE001
hits = []
note = f"tvdb did not answer: {e}"
want = bare(query)
pick = None
for h in hits:
if bare(h.get("name") or "") != want:
continue
hy = str(h.get("year") or "")
if ys and hy.isdigit() and not (min(ys) <= int(hy) <= max(ys)):
continue
pick = h
break
if pick:
tag = "EXACT" if untouched else "PUNCT"
rows.append([files, tag, folder, query, pick.get("name", ""),
str(pick.get("year", "") or ""),
str(pick.get("tvdb_id", "") or ""), ""])
elif hits:
best = hits[0]
rows.append([files, "NEAR", folder, query, best.get("name", ""),
str(best.get("year", "") or ""),
str(best.get("tvdb_id", "") or ""),
f"{len(hits)} answer(s); tvdb's own name is not the "
"query, so nothing would take it"])
else:
rows.append([files, "NONE", folder, query, "", "", "",
note or "tvdb has no series by that name"])
tally[rows[-1][1]] += 1
time.sleep(0.12)
# Two folders mapping to one record is not two shows; it is one show the release
# folders split. Say it on every row involved, or the operator approves the same
# id twice and reads the second as a mistake.
by_id = collections.Counter(r[6] for r in rows if r[6] and r[1] != "MAPPED")
for r in rows:
if r[6] and r[1] != "MAPPED" and by_id[r[6]] > 1:
r[7] = (f"MERGE: {by_id[r[6]]} folders map to tvdb {r[6]} — one show, "
"the releases split it; " + r[7]).strip("; ")
with open(ARGS.out, "w") as fh:
fh.write("files\tverdict\tfolder\tquery\ttvdb_name\ttvdb_year\ttvdb_id\tnote\n")
for r in rows:
fh.write("\t".join(str(c) for c in r) + "\n")
print(f"sheet: {len(rows)} folders"
+ "".join(f" {t}={n}" for t, n in sorted(tally.items()))
+ f" -> {ARGS.out}")
+580 -50
View File
@@ -3,32 +3,51 @@ r"""Plan a Jellyfin-format relayout of a TV library. DRY RUN ONLY: nothing moves
media-shows.py [--root DIR] [--enum TSV] [--titles TSV] [--probed TSV]
[--tmdb-cache JSON] [--tmdb] [--movies DIR] [--park DIR]
[--out TSV] [--exceptions TXT]
[--tvdb-ids TSV] [--out TSV] [--exceptions TXT]
Inputs are the TSVs scripts/media-census.py exports: `enumerate` lists what
exists, `jellyfin` says which episode each file IS, and `probe` is optional
duration evidence for the movie rule. The TMDb cache is a plain JSON file so a
plan stays reproducible: answers are read from it, and a miss is looked up only
with --tmdb and TMDB_API_KEY in the environment (the key never enters argv, and
exists, `jellyfin` says which episode each file IS (its series' TMDb and Tvdb
ids, and the episode title the row carries), and `probe` is optional duration
evidence for the movie rule. Every name and title answer lives in the cache JSON
so a plan stays reproducible: answers are read from it, and a miss is looked up
only with --tmdb/--tvdb and their credentials (a key never enters argv, and
never the cache).
Target contract (jellyfin.org/docs/general/server/media/shows):
<root>/<Series> (<year>)/Season <NN>/<Series> SxxEyy.ext
<root>/<Series> (<year>)/Season <NN>/<Series> (<year>) - SxxEyy - <Title>.ext
one file holding two episodes SxxE01-E02 specials Season 00
a documented part of an episode SxxE03 Part 1 sidecars <base>.<lang>.srt
bonus material <series>/trailers|featurettes|… — never a season
The series name and the episode title are the ones the IDENTITY authority — tvdb
— states for the id the metadata server matched, not the words a release folder
or a server's folder-derived Series name happens to carry.
A folder tvdb's search cannot state is not a folder tvdb does not know: the
record exists, it is titled in the show's own language, and the English name
rides in aliases the search never reads. --tvdb-ids is the human door for those:
one line per pack folder, `folder <TAB> tvdb id <TAB> note`, every id checked by
hand against the record it names. It is an input file like the census, so a plan
stays reproducible, and every folder it touched is listed in the exceptions —
one wrong row is a whole pack renamed to the wrong show.
PLAN is the TSV scripts/media-apply.py replays: `action src dst kind`, every
mkdir row before any move, and `keep` for a file already where the contract puts
it. EXCEPTIONS is the review list: everything parked, and why.
The authority chain, strongest first, for one video file:
1. a provider row — Jellyfin matched this exact path and numbered it.
1. a provider row — Jellyfin matched this exact path and numbered it. The
row's NAME is only evidence once an id backs it: the server names a Series
after the folder it scanned, so `24 (2001) Season 1` is a folder echo, and
the series' tvdb id is what undoes it.
2. the file's own SxxEyy / 1x04 marker, the series taken from a sibling's
provider row, else from the nearest ancestor folder that has one, else from
the release folder's own words.
a registered show its own words name, else — outside the review pile — from
the release folder's words. Inside a hidden top-level folder a file may
MATCH a registered show but never INVENT one: invention is what filled the
pile to begin with.
3. nothing. An unnumbered file is PARKED under --park with its relative path
kept: an invented `Season 01` steals an episode number from the file that
really is S01E01, and a wrong number is worse than no number.
@@ -47,8 +66,11 @@ Guards, and why each is here:
* Series art and the show nfo follow the SERIES, a season image follows its
SEASON, a subtitle or episode nfo follows the episode's new base — an episode
moved without its subtitles is a half-move.
* Hidden files, macOS forks, qBittorrent resume parts and the park pile itself
are never moved. Skipping the park is what makes re-planning stable.
* Hidden files, macOS forks, qBittorrent resume parts are never moved. A file
already inside --park is never moved either: that is what makes re-planning
stable. --park belongs OUTSIDE the library — parked files inside it get
indexed as one show named after the park folder, and a row like that is a
folder echo, never evidence (the guard in `row_names_the_park`).
The plan crosses three trees (the library, its park, the movie library), so the
executor runs with --root at the share that holds them, on its own journal —
@@ -56,7 +78,8 @@ one journal is one undo:
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
--out /mnt/entropy/media-tv-plan.tsv
--tvdb --tmdb --out /mnt/entropy/media-tv-plan.tsv \
--park /mnt/entropy/media-park/TVShows
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
"""
@@ -80,14 +103,31 @@ ARGS.add_argument("--titles", default="/tmp/jellyfin-tv.tsv",
help="the provider's numbering (media-census.py jellyfin)")
ARGS.add_argument("--probed", default="",
help="ffprobe census for the movie rule; absent = no durations")
ARGS.add_argument("--tmdb-cache", default="/tmp/tmdb-cache.json",
help="TMDb answers as an input file, so a plan is reproducible")
ARGS.add_argument("--tmdb-cache", default="/tmp/tv-names.json",
help="name answers (TMDb and Tvdb) as ONE input file, so a plan "
"is reproducible; keys carry the provider they came from")
ARGS.add_argument("--tmdb", action="store_true",
help="look up cache misses (needs TMDB_API_KEY)")
help="look up cache misses at TMDb (needs TMDB_API_KEY)")
ARGS.add_argument("--tvdb", action="store_true",
help="look up cache misses at Tvdb (needs the credentials file)")
ARGS.add_argument("--tvdb-credentials",
default=os.path.expanduser("~/.config/secrets/tvdb"),
help="file with an api_key= line (and a pin= line for a "
"user-issued key); mode 600, and never the cache")
ARGS.add_argument("--tvdb-ids",
default=os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"media-shows-tvdb-ids.tsv"),
help="the operator's map: folder <TAB> tvdb id <TAB> note, for the\n"
"packs whose name tvdb's search cannot state (anime is titled\n"
"by its own record, the English name rides in aliases). Used\n"
"only when the file exists; a line without a numeric id stops\n"
"the plan, because a typo renames a whole pack to the wrong\n"
"show.")
ARGS.add_argument("--movies", default="/mnt/entropy/Movies",
help="movie library a mis-filed film moves into")
ARGS.add_argument("--park", default="",
help="review pile (default: <root>/_unsorted)")
help="review pile OUTSIDE the library (default: <root>/_unsorted)")
ARGS.add_argument("--out", default="/tmp/tv-plan.tsv")
ARGS.add_argument("--exceptions", default="/tmp/tv-exceptions.txt")
ARGS = ARGS.parse_args()
@@ -105,6 +145,7 @@ try:
except (OSError, ValueError):
CACHE = {}
API_KEY = os.environ.get("TMDB_API_KEY", "")
TMDB = {"tries": 0, "failed": 0}
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".rm",
@@ -218,6 +259,20 @@ def leading_index(name):
return int(m.group(1)) if m else None
def tmdb_request(url):
"""One TMDb call, authenticated. The two credential shapes are NOT
interchangeable and the difference is a 401 on every single call: a v3 key
travels as the `api_key` query parameter, and only a v4 request token is
accepted as a Bearer header. A token in argv is world-readable in
/proc/<pid>/cmdline — this URL is built inside the process and never reaches
argv, so the query form is safe here."""
TMDB["tries"] += 1
if API_KEY.count(".") == 2: # three segments: a v4 JWT
return http_json(url, headers={"Authorization": "Bearer " + API_KEY})
return http_json(url + ("&" if "?" in url else "?") + "api_key="
+ urllib.parse.quote(API_KEY))
def tmdb(kind, query):
"""A provider answer for a query: the cache first, TMDb once, then never again."""
ck = f"{kind}:{query}"
@@ -231,26 +286,215 @@ def tmdb(kind, query):
path = "movie" if kind == "movie" else "tv"
url = ("https://api.themoviedb.org/3/search/" + path + "?query="
+ urllib.parse.quote(query) + "&include_adult=false")
req = urllib.request.Request(url, headers={
# The key rides a header, never the URL: a token in argv is world-readable.
"Authorization": "Bearer " + API_KEY,
"Accept": "application/json"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
found = json.load(r).get("results") or []
found = tmdb_request(url).get("results") or []
except Exception as e: # noqa: BLE001
TMDB["failed"] += 1
print(f"tmdb {kind} lookup failed for {query!r}: {e}", file=sys.stderr)
return None
if not found:
CACHE[ck] = None
return None
top = found[0]
# The two search endpoints date their rows differently: movies carry
# `release_date`, tv carries `first_air_date`. Reading only one of them leaves
# every film answer year-less, and a year-less answer can never be refused by
# the year guard below.
when = str(top.get("year") or "") or \
str(top.get("release_date") or top.get("first_air_date") or "")[:4]
CACHE[ck] = {"name": (top.get("title") or top.get("name") or "").strip(),
"year": str(top.get("year") or "") or (top.get("first_air_date") or "")[:4],
"id": top.get("id")}
"year": when, "id": top.get("id")}
return CACHE[ck]
# ---------------------------------------------------------------- tvdb, the identity
# tvdb names the show and the episode: it is the only provider whose strings are
# the ones the operator asked for, so its answer replaces a folder-derived Series
# name. The login mints a JWT per run from a mode-600 file; neither the key nor
# the token reaches argv, the cache, or the plan. A provider that will not answer
# never invents a name: every lookup here returns None, and None keeps the
# provider's own words and lands in the exceptions.
TVDB = {"jwt": None, "dead": False}
def http_json(url, body=None, headers=None):
req = urllib.request.Request(
url, data=json.dumps(body).encode() if body is not None else None,
headers={"Accept": "application/json", "Content-Type": "application/json",
**(headers or {})})
with urllib.request.urlopen(req, timeout=30) as r:
return json.load(r)
def tvdb_login():
"""The run's JWT, or None: a project key needs only api_key=, a user-issued
key also needs the pin."""
if TVDB["jwt"] is not None:
return TVDB["jwt"] or None
TVDB["jwt"] = ""
try:
lines = open(ARGS.tvdb_credentials).read().splitlines()
except OSError:
print(f"tvdb: no credentials at {ARGS.tvdb_credentials}", file=sys.stderr)
return None
sec = {}
for line in lines:
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
sec[k.strip()] = v.strip()
if not sec.get("api_key"):
print(f"tvdb: no api_key in {ARGS.tvdb_credentials}", file=sys.stderr)
return None
body = {"apiKey": sec["api_key"]}
if sec.get("pin"):
body["token"] = sec["pin"]
try:
TVDB["jwt"] = (http_json("https://api4.thetvdb.com/v4/login", body)
.get("data") or {}).get("token") or ""
except Exception as e: # noqa: BLE001
print(f"tvdb login failed: {e}", file=sys.stderr)
return None
return TVDB["jwt"] or None
def tvdb(path):
"""One Tvdb v4 GET, or None when it cannot be answered."""
if not ARGS.tvdb or TVDB["dead"]:
return None
jwt = tvdb_login()
if not jwt:
TVDB["dead"] = True
return None
try:
return http_json("https://api4.thetvdb.com/v4" + path,
headers={"Authorization": "Bearer " + jwt}).get("data")
except Exception as e: # noqa: BLE001
print(f"tvdb {path} failed: {e}", file=sys.stderr)
return None
def tvdb_series(sid):
"""{name, year} for a tvdb series id."""
ck = f"tvdb:series:{sid}"
if ck in CACHE:
return CACHE[ck]
data = tvdb(f"/series/{sid}")
if not data:
return None
CACHE[ck] = {"name": (data.get("name") or "").strip(),
"year": (data.get("firstAired") or "")[:4]}
return CACHE[ck]
def tvdb_id_for_tmdb(mid):
"""The tvdb id TMDb records for a series id — the bridge for the bulk of this
library, which the server matched by TMDb."""
ck = f"tmdb:external:{mid}"
if ck in CACHE:
return CACHE[ck] or ""
if not (ARGS.tmdb and API_KEY):
return ""
try:
got = tmdb_request("https://api.themoviedb.org/3/tv/" + mid + "/external_ids")
except Exception as e: # noqa: BLE001
TMDB["failed"] += 1
print(f"tmdb external ids failed for {mid}: {e}", file=sys.stderr)
return ""
CACHE[ck] = str(got.get("tvdb_id") or "")
return CACHE[ck]
def tvdb_search(name):
"""The tvdb series a provider name states, or None: only tvdb's own spelling
counts, so a near-miss never renames a show."""
ck = f"tvdb:search:{name}"
if ck in CACHE:
return CACHE[ck]
for hit in tvdb("/search?query=" + urllib.parse.quote(name) + "&type=series") or []:
if eng.norm(hit.get("name") or "") != eng.norm(name):
continue
CACHE[ck] = {"name": (hit.get("name") or "").strip(),
"year": str(hit.get("year") or ""),
"id": str(hit.get("tvdb_id") or "")}
return CACHE[ck]
CACHE[ck] = None
return None
# Tvdb's default episode query answers AT MOST this many rows and states no
# total: One Piece (1999), Crayon Shin-chan (1992) and Detective Conan (1996)
# each answer exactly 500 while holding over a thousand. A list that long bounds
# nothing above itself, so it must never be read as "the series holds 500".
EPISODE_PAGE_CAP = 500
def tvdb_episodes(sid):
"""`season/episode` -> title for one series, in the one call tvdb answers.
Whether that call came back a cut-off page is recorded in the cache beside
it, so a warm cache answers the same question a cold one did."""
ck = f"tvdb:episodes:{sid}"
if ck not in CACHE:
data = tvdb(f"/series/{sid}/episodes/default")
if not data:
return None
rows = data.get("episodes") or []
CACHE[ck] = {f"{e.get('seasonNumber')}/{e.get('number')}":
(e.get("name") or "").strip() for e in rows}
CACHE[f"tvdb:capped:{sid}"] = len(rows) >= EPISODE_PAGE_CAP
return CACHE[ck]
def tvdb_list_capped(sid):
"""True when tvdb's episode answer for this series is a page, not a list."""
rows = tvdb_episodes(sid)
if not rows:
return False
return CACHE.get(f"tvdb:capped:{sid}", len(rows) >= EPISODE_PAGE_CAP)
# tvdb's titles read backwards, for the rows that carry a title and no numbering.
_TITLE_IDX = {}
def tvdb_title_index(sid):
"""tvdb's episode list as `title -> (season, episode)`. A title two episodes
share names no episode: both halves are dropped, because a file that parks is
honest and a file filed at the wrong number is a lie the server believes."""
if sid in _TITLE_IDX:
return _TITLE_IDX[sid]
index, tie = {}, set()
for key, name in (tvdb_episodes(sid) or {}).items():
season, _, ep = key.partition("/")
t = eng.norm(name)
if not t or t in tie:
continue
if t in index:
del index[t]
tie.add(t)
continue
try:
n_season, n_ep = int(season), int(ep)
except ValueError:
continue # half-numbers ("12.5") state no slot
index[t] = (n_season, n_ep)
_TITLE_IDX[sid] = index
return index
def tvdb_number(t):
"""The (season, episode) tvdb states for a provider row whose own numbering is
not a season at all. Jellyfin's date matcher files specials by AIRDATE, so the
row says season 2002 — a YEAR — and only the title the provider recorded says
which episode it is. No id, no title, or a title tvdb gives twice: no number,
and the file parks with the reason it always carried."""
sid = (t.get("tvdb") or "").strip()
title = (t.get("title") or "").strip()
if not sid or not title:
return None
return tvdb_title_index(sid).get(eng.norm(title))
# ------------------------------------------------------------------- the inputs
rows = []
for line in open(ARGS.enum):
@@ -264,6 +508,26 @@ for line in open(ARGS.enum):
rows.append({"bytes": int(size), "ext": ext.lower(), "path": p, "base": base})
held = {r["path"]: r for r in rows}
def row_names_the_park(path, name):
"""The library's own review pile is not a show. When the park sat INSIDE the
library the server indexed the whole folder as one series named after it, and
a row like that would rename a real show to the pile — so a row whose series is
the hidden top-level folder the file itself sits in is dropped. The server also
sanitizes the display name, dropping the leading underscore, so `unsorted` and
`_unsorted` are the same pile: compare them without their leading punctuation,
or the pile survives as a visible series folder."""
top = rel(path).split("/")[0]
return top.startswith("_") and eng.norm(name.lstrip("_")) == eng.norm(top.lstrip("_"))
def in_review_pile(path):
"""A file under the library's hidden top-level folder is review material by
that folder's own convention: its filename may MATCH a show a provider names
somewhere else, but it may never INVENT one — inventing one from a file's own
words is what filled the pile to begin with."""
return rel(path).split("/")[0].startswith("_")
titles = {}
with open(ARGS.titles) as fh:
cols = fh.readline().rstrip("\n").split("\t")
@@ -272,7 +536,7 @@ with open(ARGS.titles) as fh:
f += [""] * (len(cols) - len(f))
row = dict(zip(cols, f))
p = os.path.abspath(row.get("file", ""))
if p in held:
if p in held and not row_names_the_park(p, row.get("series", "")):
titles[p] = row
dur = {}
@@ -365,6 +629,63 @@ def extras_kind(path, extra_type):
return "extras"
# --------------------------------------------------------- the operator's map
# The map is the human answer to a name the authority cannot be searched by. It
# is read as an input file — like the census and the cache — so a plan stays
# reproducible from the files beside it, and it outranks a provider id: a box
# set's provider id is the server's own guess at a folder name, while a map row
# is an id someone checked against the record it names.
MAP = {} # top-level pack folder -> {"id", "note"}
MAP_SEEN, MAP_CLASH = set(), [] # what the map actually touched
if ARGS.tvdb_ids and os.path.exists(ARGS.tvdb_ids):
for _n, _line in enumerate(open(ARGS.tvdb_ids), 1):
_line = _line.rstrip("\n")
if not _line.strip() or _line.startswith("#"):
continue
cols = _line.split("\t")
folder = cols[0].strip()
sid = cols[1].strip() if len(cols) > 1 else ""
if not folder or not sid.isdigit():
print(f"{ARGS.tvdb_ids}:{_n}: not a numeric tvdb id: {_line}",
file=sys.stderr)
sys.exit(2)
if folder in MAP:
print(f"{ARGS.tvdb_ids}:{_n}: folder named twice: {folder}",
file=sys.stderr)
sys.exit(2)
MAP[folder] = {"id": sid,
"note": cols[2].strip() if len(cols) > 2 else ""}
def top_folder(path):
"""The library folder a file's pack sits in. An id belongs to a show, and a
show is a folder here, so that is what the map keys on."""
return os.path.relpath(path, ROOT).split(os.sep)[0]
def map_id_for(path, provider_id):
"""The id the operator states for this file's folder. The map wins over the
provider's — it is the only door for a name tvdb cannot state — and a
disagreement between the two is written where the operator reads it, never
resolved in silence."""
if not MAP:
return provider_id
top = top_folder(path)
entry = MAP.get(top)
if not entry:
return provider_id
MAP_SEEN.add(top)
if provider_id and provider_id != entry["id"]:
MAP_CLASH.append((top, provider_id, entry["id"]))
return entry["id"]
def map_note():
if not MAP:
return ""
return f" (+{len(MAP_SEEN)} folders from the operator's map)"
# ----------------------------------------------------------- classify every file
series = {} # norm(name) -> {name, year, tmdb}
claimants = [] # (row, name, season, [episodes])
@@ -379,14 +700,17 @@ def series_name(name):
return clean(re.sub(r"\s{2,}", " ", PACK_IDX_RE.sub("", name)).strip(" ."))
def note(name, year="", tmdb_id=""):
"""Register a series; the first name wins, the first year and id stick."""
def note(name, year="", tmdb_id="", tvdb_id=""):
"""Register a series; the first name wins, the first year and ids stick."""
k = eng.norm(name)
s = series.setdefault(k, {"name": series_name(name), "year": "", "tmdb": ""})
s = series.setdefault(k, {"name": series_name(name), "year": "", "tmdb": "",
"tvdb": ""})
if year and not s["year"]:
s["year"] = str(year)
if tmdb_id and not s["tmdb"]:
s["tmdb"] = str(tmdb_id)
if tvdb_id and not s["tvdb"]:
s["tvdb"] = str(tvdb_id)
@@ -395,7 +719,7 @@ def as_extra(r, owner, extra_type):
if not owner:
unknown.append((r, "bonus material with no series to own it"))
return
note(owner)
note(owner, tvdb_id=map_id_for(r["path"], ""))
extras.append((r, owner, extras_kind(r["path"], extra_type)))
for r in video:
@@ -413,7 +737,9 @@ for r in video:
# every file under SNL 16-20/Season 16..19 carries season 16 episode 20.
owner = (t["series"] if t and t["series"]
else series_of_dir(os.path.dirname(p))
or title_of(r["base"][:marked[2]])[0])
or known_in(r["base"])
or ("" if in_review_pile(p)
else title_of(r["base"][:marked[2]])[0]))
if not eng.norm(owner).replace(" ", ""):
unknown.append((r, "an episode marker with no series name to hang it on"))
continue
@@ -422,17 +748,25 @@ for r in video:
else:
season = fs if fs is not None else num(t["season"])
ep = leading_index(r["base"]) or num(t["episode"])
if not 0 < season <= 200 or ep <= 0:
unknown.append((r, f"a provider row of season {season} episode {ep}"))
continue
if not 0 <= season <= 200 or ep <= 0:
# Season 0 IS a season: specials live in Season 00 by Jellyfin's own
# contract. A season that is a YEAR is the date matcher's doing, and
# the only authority on it is tvdb's episode list for the series.
recovered = tvdb_number(t)
if not recovered:
unknown.append(
(r, f"a provider row of season {season} episode {ep}"))
continue
season, ep = recovered
eps = [ep]
claimants.append((r, owner, season, eps))
note(owner, t["year"] if t else "", t["tmdb"] if t else "")
note(owner, t["year"] if t else "", t["tmdb"] if t else "",
map_id_for(r["path"], t["tvdb"] if t else ""))
continue
owner = series_of_dir(os.path.dirname(p))
if fs is not None and owner and leading_index(r["base"]):
claimants.append((r, owner, fs, [leading_index(r["base"])]))
note(owner)
note(owner, tvdb_id=map_id_for(r["path"], ""))
continue
if eng.is_extra(p, title_of(r["base"])[0], dur.get(p, 0.0), ROOT):
as_extra(r, owner, "")
@@ -450,17 +784,123 @@ for k, s in series.items():
if got and eng.norm(got["name"]) == k:
s["year"] = got["year"]
if ARGS.tmdb:
# tvdb is the identity authority. The row's own tvdb id is the door; a TMDb id
# crosses through TMDb's external ids; a bare name counts only when tvdb spells
# it the same and the years do not disagree. A series nothing resolves keeps the
# provider's words AND says so in the exceptions, which is where the operator
# finds the folders the plan did not rename to authority.
for s in series.values():
got = tvdb_series(s["tvdb"]) if s["tvdb"] else None
if not got and s["tmdb"]:
sid = tvdb_id_for_tmdb(s["tmdb"])
if sid:
s["tvdb"] = sid
got = tvdb_series(sid)
if not got:
hit = tvdb_search(s["name"])
if hit and hit["id"] and (not s["year"] or not hit["year"]
or hit["year"] == s["year"]):
s["tvdb"] = hit["id"]
got = hit
if got and got["name"]:
s["canon"], s["canon_year"] = got["name"], got["year"]
# One identity, one folder: the provider's spellings collapse to the shows tvdb
# says they are, so a box set split over five release folders is ONE series and
# two files claiming one episode MEET inside the duplicate rule below.
alias, canon = {}, {}
for k, s in series.items():
key = f"id:{s['tvdb']}" if s["tvdb"] else k
alias[k] = key
keep = canon.setdefault(key, s)
if keep is not s:
for f in ("year", "tvdb", "canon", "canon_year"):
if not keep.get(f) and s.get(f):
keep[f] = s[f]
if ARGS.tmdb or ARGS.tvdb:
with open(ARGS.tmdb_cache, "w") as fh:
json.dump(CACHE, fh, indent=1, sort_keys=True)
def identity(s):
"""(name, year) a folder and its files carry: tvdb's whenever it answered,
the provider's own words only for a series nothing resolved."""
if s.get("canon"):
return s["canon"], s["canon_year"] or s["year"]
return s["name"], s["year"]
def series_key(name):
"""The merged identity a provider spelling belongs to."""
return alias.get(eng.norm(name), eng.norm(name))
def label_of(s):
"""`<Series> (<year>)`, the form both matchers read. A provider's name can
arrive WITH the year already on it — TVDb hands back `Acapulco (2021)` for the
reboot — and appending a second one names a folder nothing will match again."""
nm, yr = identity(s)
nm = clean(nm)
if not yr or re.search(r"\(\d{4}\)\s*$", nm):
return nm
return f"{nm} ({yr})"
def contradicted_seasons(s):
"""Seasons whose numbering tvdb's own episode list contradicts — the one place
a tvdb title can be a confident lie. An anime pack numbered by its release group
books S05E01 in a season tvdb does not hold, or books more episodes there than
tvdb lists, and the slot its number points at is a DIFFERENT episode. Books <=
tvdb's count stays plausible: a partial pack is the normal shape. A list the
authority cut off at its page limit (EPISODE_PAGE_CAP) contradicts nothing —
it says nothing about the episodes it never sent."""
if "contradicted" in s:
return s["contradicted"]
bad = set()
eps = None if not s["tvdb"] or tvdb_list_capped(s["tvdb"]) \
else tvdb_episodes(s["tvdb"])
if eps:
have = collections.Counter()
for k in eps:
season, _, _ep = k.partition("/")
if season.isdigit():
have[int(season)] += 1
booked = collections.defaultdict(set)
for _r, name, season, claimed in claimants:
if canon.get(series_key(name)) is s:
booked[season].update(claimed)
bad = {n for n, seen in booked.items() if len(seen) > have.get(n, 0)}
s["contradicted"] = bad
return bad
def episode_title(r, s, season, ep):
"""The title this episode carries: tvdb's for the id the series resolved to,
else the title the matched row states, else nothing at all — a dangling ` - ` is
not a title, and an invented one is worse than none. A season tvdb's own list
contradicts gets no tvdb title: its number points somewhere else, and a wrong
title is worse than a bare one."""
if s["tvdb"] and season not in contradicted_seasons(s):
t = ((tvdb_episodes(s["tvdb"]) or {}).get(f"{season}/{ep}") or "").strip()
if t:
return clean(t)
t = ((titles.get(r["path"]) or {}).get("title") or "").strip()
return clean(t) if t else ""
def series_dir(name):
s = series.get(eng.norm(name))
"""One series, one folder — and the folder CARRIES the tvdb id. Both matchers
read `[tvdbid-NNNN]` (Jellyfin's Tvdb agent, Plex's scanner), so a show whose
title is ambiguous resolves by number instead of by a human guessing which
folder is which. The id rides the FOLDER only: Jellyfin gives a file name 180
characters and the episode title needs the room. A series nothing resolved
keeps the provider's words with no tag at all."""
s = canon.get(series_key(name))
if not s:
s = {"name": clean(name), "year": ""}
label = f"{s['name']} ({s['year']})" if s["year"] else s["name"]
return f"{ROOT}/{clean(label)}"
return f"{ROOT}/{clean(series_name(name))}"
tag = f" [tvdbid-{s['tvdb']}]" if s["tvdb"] else ""
return f"{ROOT}/{label_of(s)}{tag}"
# ------------------------------------------------------------------- the plan
@@ -500,25 +940,28 @@ def park(r, reason):
by_ep = collections.defaultdict(list)
for r, name, season, eps in claimants:
for e in eps:
by_ep[(eng.norm(name), season, e)].append(r)
by_ep[(series_key(name), season, e)].append(r)
winners, losers = set(), {}
for (key, season, e), rs in by_ep.items():
rs.sort(key=lambda r: (-r["bytes"], r["path"]))
winners.add(rs[0]["path"])
for r in rs[1:]:
losers.setdefault(r["path"],
f"duplicate of {series[key]['name']} S{season:02d}E{e:02d}"
f"duplicate of {canon[key]['name']} S{season:02d}E{e:02d}"
" (the bigger copy keeps the standard name)")
for r, name, season, eps in claimants:
if r["path"] in losers:
park(r, losers[r["path"]])
continue
s = series[eng.norm(name)]
label = f" S{season:02d}E" + "-E".join(f"{e:02d}" for e in eps)
s = canon[series_key(name)]
stem = f"{label_of(s)} - S{season:02d}E" + "-E".join(f"{e:02d}" for e in eps)
title = episode_title(r, s, season, eps[0])
if title:
stem += f" - {title}"
part = PART_RE.search(os.path.splitext(r["base"])[0])
tail = f" Part {part.group(1).upper()}" if part and len(eps) == 1 else ""
book(r, f"{series_dir(name)}/Season {season:02d}", s["name"] + label + tail + r["ext"],
"tv")
if part and len(eps) == 1:
stem += f" Part {part.group(1).upper()}"
book(r, f"{series_dir(name)}/Season {season:02d}", stem + r["ext"], "tv")
for r, name, kind in extras:
book(r, f"{series_dir(name)}/{kind}",
@@ -530,8 +973,13 @@ for r, name, kind in extras:
for r in films:
parsed, year = title_of(r["base"])
got = tmdb("movie", parsed)
words = eng.norm(got["name"]).split() if got else []
if not got or not eng.mentions(parsed.split(), words) \
# The provider's title must be a run of words the FILE itself states. The query
# is only what title_of could salvage from the name, and a short one ("russia")
# lets the top hit be a different work entirely: a 1963 Bond film is not a 1991
# documentary episode, and Special Edition I is not Special Edition III.
said = eng.norm(os.path.splitext(r["base"])[0]).split()
named = eng.norm(got["name"]).split() if got else []
if not got or not eng.mentions(named, said) \
or (year != "?" and got["year"] and got["year"] != year):
park(r, "long file with no episode evidence, and no movie provider names it")
continue
@@ -655,6 +1103,30 @@ for r, reason in unknown:
# --------------------------------------------------------------------- output
def provider_broken():
"""TMDb was asked for names and did not answer: the plan is not the plan the
operator asked for, whatever the move count says."""
return bool(ARGS.tmdb and (not API_KEY or TMDB["failed"]))
def provider_note():
"""Provider health on the summary line. The 401s are already on stderr, and a
terminal that scrolled past them shows a plan that looks finished: the count
belongs where it cannot be missed, because a half-resolved plan renames
folders with the folder's own junk words."""
if not (ARGS.tmdb or ARGS.tvdb):
return ""
if not ARGS.tmdb:
return ", tmdb off (names from the cache and tvdb only)"
if not API_KEY:
return ", tmdb NOT ASKED (TMDB_API_KEY unset): NOT provider-complete"
if not TMDB["tries"]:
return ", tmdb cache-only (every name was already answered)"
done = TMDB["tries"] - TMDB["failed"]
return (f", tmdb {done}/{TMDB['tries']} answered"
+ (" — NOT provider-complete" if TMDB["failed"] else ""))
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
@@ -676,16 +1148,25 @@ def write_plan():
sized = sum(held[s]["bytes"] for s, _d, _k in moves if s in held)
print(f"plan: {len(moves) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(keeps)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
f" {len(dirs)} folders{map_note()}{provider_note()} -> {ARGS.out}")
def write_exceptions():
no_year = sorted(s["name"] for s in series.values() if not s["year"])
no_year = sorted({s["name"] for s in canon.values() if not s["year"]})
no_tvdb = sorted({identity(s)[0] for s in canon.values() if not s.get("canon")})
clash = sorted({(identity(s)[0], tuple(sorted(contradicted_seasons(s))))
for s in canon.values() if contradicted_seasons(s)})
with open(ARGS.exceptions, "w") as fh:
fh.write(f"TV relayout exceptions for {ROOT}\n\n")
fh.write(f"Everything below is parked under {PARK}, keeping the relative path it\n")
fh.write("has today. Nothing is deleted and nothing is overwritten: a parked file\n")
fh.write("is one rename from wherever the plan is pointed next.\n\n")
if provider_broken():
fh.write("WARNING: " + provider_note().strip(", ").replace(": NOT", " — NOT")
+ "\n Names and episode titles that should have come from TMDb are\n"
" missing, so folders keep the metadata provider's words. Fix the\n"
" credential and re-plan: the cache keeps every answer that did\n"
" arrive, so the re-run costs only the lookups that failed.\n\n")
for reason in sorted(reasons, key=lambda r: -len(reasons[r])):
rs = reasons[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
@@ -698,6 +1179,55 @@ def write_exceptions():
fh.write(f"{len(no_year):>6} series carry no year from any provider, so their\n")
fh.write(" folders are named without one; Jellyfin still matches them.\n")
fh.write(" " + ", ".join(no_year[:300]) + "\n")
if no_tvdb:
fh.write(f"\n{len(no_tvdb):>6} series carry no tvdb identity, so their folders\n")
fh.write(" keep the name the metadata provider gave them and no episode\n")
fh.write(" title was taken. Nothing was invented: give these a resolvable\n")
fh.write(" id (--tvdb) and re-plan; a rename is idempotent.\n")
fh.write(" " + ", ".join(no_tvdb[:300]) + "\n")
if clash:
fh.write(f"\n{len(clash):>6} series carry seasons whose numbering tvdb"
" contradicts:\n")
fh.write(" a pack books more episodes in a season than tvdb holds"
" there — an anime\n pack numbered by its release group is"
" the usual shape. Those files still\n rename to the"
" standard shape, but they do NOT ride with tvdb's titles:"
"\n the slot tvdb answers for that number is a different"
" episode. Correct the\n numbering and re-plan; a rename"
" is idempotent.\n")
fh.write(" " + "; ".join(
f"{n} S{'/'.join(f'{x:02d}' for x in ss)}"
for n, ss in clash[:300]) + "\n")
map_ids = {e["id"] for e in MAP.values()}
unseen = sorted(f for f in MAP if f not in MAP_SEEN)
unanswered = sorted({f"{identity(s)[0]} ({s['tvdb']})"
for s in canon.values()
if s.get("tvdb") in map_ids and not s.get("canon")})
if MAP:
fh.write(f"\n{len(MAP_SEEN):>6} folders took their tvdb id from the\n")
fh.write(" operator's map — the human answer to a name tvdb's search\n"
" cannot state. A wrong row renames a whole pack to the wrong\n"
" show, so every folder it touched is listed here; fix the map\n"
" and re-plan, because a rename is idempotent.\n")
fh.write(" " + "; ".join(
f"{f} -> {MAP[f]['id']}" for f in sorted(MAP_SEEN)) + "\n")
if unseen:
fh.write(f"\n{len(unseen):>6} map rows name no folder on the array: the\n")
fh.write(" pack moved, or the row carries a name the array never held.\n")
fh.write(" " + "; ".join(unseen[:300]) + "\n")
if MAP_CLASH:
fh.write(f"\n{len(MAP_CLASH):>6} folders carry a provider tvdb id the map\n")
fh.write(" contradicts. The map won: a box set's provider id is the\n")
fh.write(" server's guess at a folder name. Read each one:\n")
fh.write(" " + "; ".join(
f"{f} provider {p} map {m}"
for f, p, m in sorted(MAP_CLASH)[:300]) + "\n")
if unanswered:
fh.write(f"\n{len(unanswered):>6} map ids tvdb never answered, so those\n")
fh.write(" folders carry the id and the provider's own name; both\n")
fh.write(" matchers resolve by the id in the folder tag, so the next\n"
" re-plan takes tvdb's title. Nothing was invented.\n")
fh.write(" " + "; ".join(unanswered[:300]) + "\n")
print(f"exceptions -> {ARGS.exceptions}")
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env python3
"""Prove a relayout landed, using the media server's own tables.
A scan answers `204` whether or not it found anything, and a rename that the
scanner has not followed looks exactly like a rename that never happened from
the outside. The only honest proof is the run's journal replayed against the
database the server actually serves from: every `move` row must have a row at
its destination, and NO row may still name its source — a source row is a scan
that has not caught up, and it is the one signature of a library pointing at
files that no longer exist.
Two doors, both read-only, both without a lock while the server runs (there is
no `sqlite3` binary on these hosts). Open `mode=ro` PLAIN: both servers keep
their libraries in WAL mode, and `immutable=1` — which claims the file never
changes — makes sqlite skip the write-ahead log entirely, so every row the
server has committed but not yet checkpointed reads as MISSING, and a read
beside a running scanner can even fail as "database disk image is malformed".
`immutable=1` is only the fallback, for a database whose `-shm` cannot be read.
jellyfin `BaseItems.Path`. The container binds the share at `/data`, so the
database holds `/data/<library>/…` where the journal holds
`/mnt/entropy/<library>/…`. Translate, or every row reads as missing.
plex `media_parts.file`, joined up to `metadata_items`. Plex is bound at
the host path, so its rows match the journal verbatim. Filter
`deleted_at is null` on BOTH tables — a deleted scan leaves its parts
behind, and one of those rows is not proof anything landed.
Neither server holds a row for everything the journal renames: a `.log`, `.cue`
or cover image has no row to land in, so its absence is not a fault. The report
therefore breaks the missing rows down by extension, and `--ignore-ext` names the
extensions this server does not index, which are then counted separately and kept
out of the verdict. What is left is split by what the filesystem says, because
the database alone cannot: a file still at its `src` is `REAL` (this pass's own
damage), a file at neither path is `gone` (a later pass or the server renamed it),
and a file sitting at its `dst` with no row naming it is simply something the
server does not index. A row that still names `src` is none of these: it is the
server's own stale pointer — a phantom — and it costs a re-scan, never a restore.
Exit codes: 0 every move landed; 1 something is stale or missing (the details are
printed); 2 the journal or the database is not usable at all.
"""
import argparse
import json
import os
import sqlite3
import sys
from collections import Counter
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
# Per kind: the table whose columns are required, those columns, and the query
# that returns every path the server believes it has.
KINDS = {
"jellyfin": {
"table": "BaseItems",
"columns": {"Path"},
"sql": "select Path from BaseItems",
"default_db": JF_DB,
},
"plex": {
"table": "media_parts",
"columns": {"file", "media_item_id", "deleted_at"},
"sql": ("select p.file from media_parts p"
" join media_items m on m.id = p.media_item_id"
" join metadata_items mi on mi.id = m.metadata_item_id"
" where p.deleted_at is null and mi.deleted_at is null"),
"default_db": PLEX_DB,
},
}
def die(msg):
print(f"media-verify: {msg}", file=sys.stderr)
sys.exit(2)
def read_moves(path):
"""The (src, dst) pairs of a journal, in the order the run performed them."""
moves = []
try:
fh = open(path, encoding="utf-8")
except OSError as e:
die(f"cannot read the journal {path}: {e.strerror}")
with fh:
for n, line in enumerate(fh, 1):
line = line.strip()
if not line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
die(f"{path} line {n} is not a journal record")
if rec.get("op") == "move":
if not rec.get("src") or not rec.get("dst"):
die(f"{path} line {n} is a move without both src and dst")
moves.append((rec["src"], rec["dst"]))
return moves
def server_paths(db, kind):
"""Every path the server holds, or a refusal that names what is missing."""
spec = KINDS[kind]
con = None
err = None
# Plain ro reads the WAL the server commits through; immutable is the last
# resort, because it cannot see a row the server has not checkpointed.
for opt in ("mode=ro", "mode=ro&immutable=1"):
try:
con = sqlite3.connect(f"file:{db}?{opt}", uri=True)
break
except sqlite3.Error as e:
err = e
if con is None:
die(f"cannot open {db}: {err}")
with con:
have = {r[1] for r in con.execute(f"pragma table_info({spec['table']})")}
if not spec["columns"] <= have:
die(f"{db} is not a {kind} library database ({spec['table']} has no "
f"{','.join(sorted(spec['columns'] - have))})")
try:
return {r[0] for r in con.execute(spec["sql"]) if r[0]}
except sqlite3.Error as e:
die(f"{db} refused the {kind} query: {e}")
def to_host(path, container, share):
"""The database's path in the journal's terms; identity when it is not a
container-side path."""
if not container:
return path
prefix = container.rstrip("/") + "/"
if path.startswith(prefix):
return share.rstrip("/") + "/" + path[len(prefix):]
return path
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--journal", required=True,
help="the run's journal (scripts/media-apply.py writes it)")
ap.add_argument("--db-kind", choices=sorted(KINDS), required=True)
ap.add_argument("--db", default=None, help="the server's database on this host")
ap.add_argument("--share", default="/mnt/entropy",
help="the share every journal path lives under")
ap.add_argument("--container", default=None,
help="what the share is called inside the container; empty"
" compares verbatim (default /data for jellyfin, nothing"
" for plex)")
ap.add_argument("--show", type=int, default=20,
help="how many offenders to name before saying 'and N more'")
ap.add_argument("--ignore-ext", default="",
help="comma-separated extensions this server does not index,"
" kept out of the verdict (e.g. log,cue,nfo,srt)")
args = ap.parse_args()
db = args.db or KINDS[args.db_kind]["default_db"]
container = ("/data" if args.db_kind == "jellyfin" else "") \
if args.container is None else args.container
ignore = set()
for ext in args.ignore_ext.split(","):
ext = ext.strip().lower()
if ext:
ignore.add(ext if ext.startswith(".") else "." + ext)
moves = read_moves(args.journal)
stored = {to_host(p, container, args.share) for p in server_paths(db, args.db_kind)}
landed, stale, missing, ignored = 0, [], [], 0
for src, dst in moves:
if os.path.splitext(dst)[1].lower() in ignore:
ignored += 1
continue
at_dst, at_src = dst in stored, src in stored
if at_dst and not at_src:
landed += 1
continue
if at_src:
stale.append(src)
if not at_dst:
missing.append((src, dst))
# The database cannot tell a lost file from one it never indexes: both read
# as "no row at the destination". Stat both paths, so the report separates
# the classes the runbook names — `REAL` is the only one that is this pass's
# own damage, and `gone` is a later pass or the server's own rename.
on_disk, real, gone = [], [], []
for src, dst in missing:
if os.path.exists(dst):
on_disk.append(dst)
elif os.path.exists(src):
real.append(src)
else:
gone.append(dst)
print(f"{args.db_kind}: {len(moves)} moves checked")
print(f"landed: {landed}")
print(f"still at src: {len(stale)}")
print(f"missing from the DB: {len(missing)}")
print(f" on disk at dst, no row names it: {len(on_disk)}")
print(f" REAL — file still at src: {len(real)}")
print(f" gone — at neither path: {len(gone)}")
if ignored:
print(f"not indexed (extension ignored): {ignored}")
if missing:
exts = Counter(os.path.splitext(d)[1].lower() or "(no extension)"
for _, d in missing)
print("missing by extension: "
+ ", ".join(f"{e} {n}" for e, n in sorted(exts.items())))
for label, items in (("real", real), ("gone", gone), ("missing", on_disk),
("stale", stale)):
for path in items[:args.show]:
print(f" {label}: {path}")
if len(items) > args.show:
print(f" {label}: … and {len(items) - args.show} more")
return 1 if (stale or missing) else 0
if __name__ == "__main__":
sys.exit(main())
@@ -48,6 +48,14 @@ Use the `Authorization: token` header for everything.
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
- **Comment on a PR**: the conversation thread is `POST $API/issues/<PR-number>/comments`
(201, and the answer's `html_url` points at `pulls/<n>`; Gitea's issue index is
unified, so the PR's number is the same number). `$API/pulls/<n>/comments` is the
**review**-comment store and answers `405` to a plain body.
- **A body built by `jq --rawfile b /dev/stdin` inside `$(…)` is empty**: the command
substitution does not inherit the heredoc, so Gitea receives `{}` and answers 422 —
and `jq -r '.id'` on the error prints nothing, which reads as success. Write the
body to a file and pass `--rawfile` that file.
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
@@ -1,51 +1,68 @@
---
type: reference
description: "The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at"
description: "Dankserver's /mnt/entropy volume: the LUKS/array chain, the crypttab + fstab boot path, and the containers bound to it"
tags:
- dankserver
timestamp: 2026-08-28
timestamp: 2026-10-31
---
# Entropy Volume
The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at
`/mnt/entropy`. The command-line helper reuses the host's desktop-integrated
`udisks2`, `crypttab`, and `fstab` configuration; it does not store or handle
the passphrase itself.
`/mnt/entropy` is the ext4 filesystem on the LUKS container holding `/dev/md0`
(RAID6, seven disks). The media stack's data lives there, so the mount is a boot
dependency for those containers rather than a convenience.
## Command-line mount
```bash
/home/dank/serverAppConfigs/scripts/mount-entropy.sh
```
Run it as the normal desktop user (`dank`), not from a root shell. `udisksctl`
will securely prompt for the LUKS passphrase. `sudo mount /mnt/entropy` then
uses the existing `/etc/fstab` entry, preserving its options:
## Chain
```text
nosuid,nodev
/dev/md0 <- mdadm (its mdadm.conf ARRAY is what makes md0 exist at boot)
-> LUKS UUID=cd86c728-459e-4f3a-bcc2-41ecf84685ec keyfile: /etc/entropy
-> /dev/mapper/entropy <- systemd-cryptsetup@entropy.service
-> ext4 at /mnt/entropy <- mnt-entropy.mount, from /etc/fstab
```
The existing mountpoint is owned by `dank:dankserver` with mode `0755`; the
helper deliberately does not run `chmod` or `chown`, so it cannot silently
change those permissions.
## Files that own it
## Device path
| Repo file | Installs to | Produces at boot |
|---|---|---|
| `dankserver/host/crypttab` | `/etc/crypttab` | `systemd-cryptsetup@entropy.service` → `/dev/mapper/entropy` |
| `dankserver/host/fstab` | `/etc/fstab` | `mnt-entropy.mount` (mountpoint `dank:dankserver`, `0755`) |
The current storage chain is:
Deploy a change by copying the file onto the host, then
`systemctl daemon-reload && update-initramfs -u`. The initramfs step is not
optional: the cryptsetup hook bakes `/etc/crypttab` into the initrd, and the
array is unlocked there, before the real root mounts. `scripts/mount-entropy.sh`
(`udisksctl unlock` + `mount`) stays as the manual path for a volume that was
never unlocked; on a correct boot it is redundant.
```text
/dev/md0 -> LUKS UUID cd86c728-459e-4f3a-bcc2-41ecf84685ec
-> ext4 label entropy
-> /mnt/entropy
```
## Law
The helper checks that `/dev/md0` exists and exits successfully if the volume
is already mounted. It is intentionally specific to this host and volume:
that avoids accidentally unlocking or mounting a different disk.
**`crypttab`'s device field takes the LUKS UUID, never the md array UUID.**
`blkid` prints both on adjacent lines for a RAID-backed container, and the file
shipped since 2025 carried the array UUID (`61bd6eaf-…`) where the LUKS container
is `cd86c728-…`. The generator produced a perfectly valid unit pointing at a
device that never appears, so the mapper came into existence only through a
desktop `udisks2` session — the volume looked mounted on every login while every
boot failed to mount it. The matching `fstab` line named the same wrong UUID
directly instead of the mapper.
## Related
**A LUKS keyfile is hashed whole, so a trailing newline is a different
passphrase.** The shipped `/etc/entropy` was 27 bytes — 26 characters plus `\n` —
and `cryptsetup` rejected it even after the UUID was corrected. Keyfiles stay
`0600` with no trailing newline; prove it with
`cryptsetup open --test-passphrase --key-file /etc/entropy UUID=…` *before*
relying on it for a boot.
- [[Storage Policy]]
- [[Dankserver Stack]]
**Prove a mount fix with the artifacts systemd generates, not with a live
mount.** The regenerated `systemd-cryptsetup@entropy.service` naming the right
device, `findmnt /mnt/entropy` sourcing `/dev/mapper/entropy`, and a matching
checksum between the repo copy and the host copy are the evidence. A mount that
exists because someone logged in proves nothing about the boot path — that
exactly describes the year this bug survived.
## Consumers
`jellyfin`, `copyparty`, and `plex` bind paths under `/mnt/entropy`; with the
volume absent they start `active (healthy)` against empty directories, which is
how a missing 91 TiB volume hides. See [[Dankserver Stack]] for the stack and
[[Storage Policy]] for what may be written here.
+59 -13
View File
@@ -18,7 +18,7 @@ clients.
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
## Facts (verified from the live unit, 2026-09-03)
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
| Item | Value |
|------|-------|
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**; sidecars land on the NVMe, the [[Entropy Volume]] array underneath stays read-only and untouched |
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
| Health | `curl -f http://localhost:8096/health` every 30 s |
@@ -36,23 +36,27 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
```
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
/var/lib/media-meta/view -> /data (the SSD-backed overlay view)
/var/lib/media-meta/view -> /data (git: the overlay view)
/mnt/entropy -> /data (live today: the array, read-only)
```
Library folders point at `/data` paths, which are the view's paths: stable
across container recreation, and identical to the raw array layout below it.
Library folders point at `/data` paths: stable across container recreation and
identical to the raw array layout below. Which side of that path the container
is on is the open question on this host — the installed unit binds the read-only
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
`Local metadata` — save metadata to media folders — is enabled, and it works
because `/data` is the [[Media Metadata View]] rather than the read-only array:
`movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork are written once, on the
NVMe upper layer, and [[Plex]] reads the same files. Jellyfin is the fleet's
only metadata writer.
`Local metadata` — save metadata to media folders — is enabled, and it is the
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
written once on the NVMe upper layer are files [[Plex]] can read too. While
`/data` is bound read-only every item update logs the error below and no sidecar
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
that raises the view.
**Failure signature:** `System.IO.IOException: Read-only file system :
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
instead of the view — the view is unmounted or the quadlet drifted. That used to
be the expected state (pre-view, every item update logged it); today it is a
fault. Run `scripts/media-view.sh status`.
instead of the view. On Dankserver that is today's expected state — the view has
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
same line is a fault: the quadlet drifted or the mount died.
## Ports & access
@@ -178,6 +182,48 @@ scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
2. Set metadata/language preferences (`LANG=en_US.UTF-8` pinned in the unit).
3. Configure clients to use `http://192.168.0.235:8096` or the guarded vhost.
## Triggering a scan
The scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same value is
`JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a mode-600 curl
config, never a header argument, in the `MediaBrowser` form: 12.4 authenticates
only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token`
for every key, valid or not. Before believing a 401, prove the key against the
server's own table — `select count(*) from ApiKeys where AccessToken = ?` on
`jellyfin.db` opened `mode=ro&immutable=1`. The backslashes below matter:
```bash
umask 077
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
"$(cat ~/.config/secrets/jellyfin)"
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
echo 'request = "POST"'; } >/tmp/jf.hdr
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
shred -u /tmp/jf.hdr
```
`204` is acceptance; anything else queued no scan, so re-issue the identical
request instead of diagnosing. A `401` from a key that answers `200` on a
hand-typed request is this block's quoting, not the server: bash `printf`
collapses `\"` to `"`, curl ends the quoted value at that quote, and Jellyfin
receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answers
204, `\"` answers 401 — both run on this host).
One accepted refresh is the whole procedure. A second one while a scan runs moves
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
later `204` is accepted and then discarded — the status code stops meaning
anything. The fix is `systemctl --user restart jellyfin.service` (never
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
response code.
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
is the incremental pass, and inotify does not cross a rootless podman bind, so
folders created on the host stay invisible to it. The move is
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
record's `RefreshStatus` is the progress read.
## See Also
- [[Plex]] — co-resident media server on the same volume
@@ -0,0 +1,74 @@
---
type: reference
title: Media Audiobook Relayout
description: "Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals."
tags: [dankserver, media-server, plex, media-relayout]
timestamp: 2026-10-31T00:00:00Z
---
# Media Audiobook Relayout
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
is the NARRATOR, both matched out of the files' own tags. Today one book is split
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
re-files the tree so the property both servers' book scanners assume is true —
Audiobooks/<Narrator>/<Book> (<year>)/<the file keeps its own name>
The shared gates, the executor, the journal law and the rollback live in
[[Media Relayout]]; this page is the audiobook arm only.
```bash
# 1. The authority: Plex's chapter rows for this root (the DB opens read-only).
python3 scripts/media-census.py audiobooks --root /mnt/entropy/Audiobooks \
--out /tmp/ab_rows.tsv
# 2. What is on disk (stat only).
python3 scripts/media-census.py enumerate --root /mnt/entropy/Audiobooks \
--out /tmp/ab_enum.tsv
# 3. The plan.
python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
--enum /tmp/ab_enum.tsv --rows /tmp/ab_rows.tsv --out /tmp/ab_plan.tsv \
--exceptions /tmp/ab_exceptions.txt
# 4. Read the exceptions file FIRST. Then the same executor, journal and rollback
# as [[Media Relayout]] shows, on its own journal, and refresh the Plex section
# as in step 5 of the movie Procedure — from inside the container, never over
# the host port.
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
```
The group is the identity Plex asserts in its own database — **(narrator, book,
year)** — so one book arriving in four part folders becomes one folder, and a
1965 and a 2020 recording that share a title stay two books.
Two laws are forced by what the rows actually hold:
+- **A chapter is never renumbered.** Most books contain chapters whose index Plex
does not know, so a number written into a name would be invented — and it would
buy nothing, because both servers order playback by the embedded track tag, not
by the file name. A wrong number in a name is the number a human later trusts.
Fix order in the tags: a Plex scan re-reads tags and never renames a file.
+- **The shelf is the narrator, not the author.** Every book has exactly one artist
row and it is the narrator; the author survives only inside some album titles
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving it would
mean parsing titles where a ripper wrote one and inventing it everywhere else.
The author stays visible in the folder name Plex itself shows.
What it refuses, the music arm's list with one addition: a video container, audio
no chapter row names, and a loose file in a folder holding two books all park under
`_unsorted` with the path they have today — and so does a chapter whose row names
**no book or no narrator**: a placeholder (`[Unknown Album]`, `[Unknown Artist]`)
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
`Various Artists` is not a placeholder — that is a multi-cast read.
The reviewed plan lives at `/mnt/entropy/media-relayout-audiobooks-plan.tsv`; it
applies on its own journal, behind the music writer, and then refreshes the Plex
section. Verify it with `--db-kind plex` — Jellyfin holds no rows here, so a
Jellyfin pass proves nothing either way.
## See Also
[[Media Relayout]], [[Media Music Relayout]], [[Plex]]
@@ -20,6 +20,19 @@ The mount point is not a volume and holds nothing while unmounted. It sits insid
merged door are one directory — and so the mount table grows no new `/mnt` entry
next to the array.
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
`/var/lib/media-meta` does not exist, and the installed quadlets under
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
is the shape git describes, not the shape the servers run. The cutover is an
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
`ensure_media_view` (`install` then `verify`) before it restarts the container,
so it must not be fired through the middle of a library scan. Until that deploy
lands there is no upper layer, so no relayout can leave ghost metadata and
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
view really is a mountpoint.
## Shape
| Item | Value |
@@ -0,0 +1,68 @@
---
type: reference
title: Media Music Relayout
description: "Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals."
tags: [dankserver, media-server, jellyfin, media-relayout]
timestamp: 2026-10-31T00:00:00Z
---
# Media Music Relayout
`/mnt/entropy/Music` is rips and discography packs, and Jellyfin has matched and
tagged almost all of it. `scripts/media-music.py` re-files it into the shape
Jellyfin's music scanner reads —
Music/<album artist>/<Album> (<year>)/NN - <Title>.ext
— and the authority is the tag row of the track, never the name the ripper gave
the file. The shared gates, the executor, the journal law and the rollback live in
[[Media Relayout]]; this page is the music arm only.
```bash
# 1. The authority: Jellyfin's audio rows, one per track, with the album and
# artist ids its own database carries (the DB opens read-only).
python3 scripts/media-census.py music --root /mnt/entropy/Music \
--out /tmp/music_rows.tsv
# 2. What is on disk (stat only).
python3 scripts/media-census.py enumerate --root /mnt/entropy/Music \
--out /tmp/music_enum.tsv
# 3. The plan.
python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.tsv \
--rows /tmp/music_rows.tsv --out /tmp/music_plan.tsv \
--exceptions /tmp/music_exceptions.txt
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
# executor, journal and rollback as [[Media Relayout]] shows, on its own journal.
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-music.jsonl --apply
```
The group is what the tags assert, not where the files sit: **(album-artist tag,
album, year)**. One album arrives as three disc folders and as two copies of one
rip, and one MusicBrainz release group is one album. A compilation is one album
whose tracks name many performers, so it files once under `Various Artists`:
keying on the performer would file one album once per artist. Two different
album-artist tags on one album name are two albums, which is what Jellyfin's two
album items say.
What it refuses:
+- **A sidecar is placed by the audio around it, never by its own name.** A
`.lrc`, `.txt` or picture that shares a track's stem follows that track and
takes the name the track now has, or Jellyfin cannot find it; anything else
goes to the album its own folder was placed in, or to the folder whose discs
sit one level below it. A folder whose audio belongs to two albums parks its
sidecar rather than guessing.
+- **An image album keeps its name.** A folder holding one audio file and a
`.cue` is a whole album in one file; numbering it `01 - …` tells Jellyfin to
index it as one track.
+- **A video container parks**, because Jellyfin will not index it as music and a
rename to `.m4a` would claim it is; **audio no provider row names parks** too,
because unmatched audio is a tagging job, not a renaming job.
ONE writer at a time on this array: two rename passes race, and neither journal
tells. Verify the run against Jellyfin's own tables — see *Proving the move
landed* in [[Media Relayout]].
## See Also
[[Media Relayout]], [[Jellyfin]], [[Media Audiobook Relayout]]
@@ -1,7 +1,7 @@
---
type: reference
title: Media Relayout
description: "Census, plan and apply the movie and TV libraries into Jellyfin format: gates, evidence chain, refusals, rollback."
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
tags: [dankserver, media-server, jellyfin, plex, procedure]
timestamp: 2026-10-30T00:00:00Z
---
@@ -12,18 +12,24 @@ The movie library on [[Entropy Volume]] arrived as release folders —
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
the release. [[Jellyfin]] matches a movie by folder and file name, so the
release noise cost posters, extras and versions. These scripts turn that
library into Jellyfin's own shape — the movies, the TV packs, and the TV that
turns up in the movie share — and nothing else in the repo touches a media file.
library into the shape its media server reads — the movies, the TV packs, the TV
that turns up in the movie share, the music library, and the audiobooks, whose
only reader is Plex — and nothing else in the repo touches a media file.
| Script | Touches media? | Output |
|---|---|---|
| `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
| `scripts/media-census.py` | reads headers and the servers' own DBs, never writes | `enum`, `probed`, and one authority TSV per library |
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
| `scripts/media-shows.py` | no — TSV out | a plan that re-files the TV library itself |
| `scripts/media-music.py` | no — TSV out | a plan that re-files the music library by its tags |
| `scripts/media-audiobooks.py` | no — TSV out | a plan that re-files the audiobooks by the book Plex matched |
| `scripts/media-verify.py` | no — reads the servers' DBs read-only | the verdict that every journaled move landed |
The seams are `tests/media_relayout.bats` and `tests/media_shows.bats`; the target contract is quoted from
The seams are `tests/media_relayout.bats`, `tests/media_shows.bats`,
`tests/media_music.bats`, `tests/media_audiobooks.bats` and `tests/media_verify.bats`;
the movie and TV target contracts are quoted from
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
in `media-plan.py`'s docstring.
@@ -41,8 +47,10 @@ $C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
```
`plex` reads the Plex metadata cache (`--db` defaults to the path the
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
`mode=ro&immutable=1`: a live server is never locked or written.
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with plain
`mode=ro`: a live server is never locked or written, and plain ro is also what
makes a scan's uncheckpointed rows visible. The same rule as
`scripts/media-verify.py` — see *Proving the move landed*.
Then plan — this reads only the three TSVs, so it costs seconds and can run
anywhere, including a laptop:
@@ -73,41 +81,55 @@ An apply also leaves the release folders it emptied standing where they were —
hundreds of them at the library root — which is what `sweep` removes, dry-run
first, from the same journal.
After it lands, both servers need a scan or the change is invisible.
After it lands, both servers need a scan or the change is invisible. [[Plex]] and
[[Jellyfin]] own the exact requests; two laws hold at this step: `204` is
acceptance and anything else queued nothing — re-issue the identical request
rather than diagnosing — and ONE accepted refresh is the whole procedure, because
a second while a scan runs moves `Scan Media Library` to `Cancelling` and every
later `204` is accepted and then discarded.
### Proving the move landed
A scan answers `204` whether or not it finds anything, so proof is the journal
replayed against both servers' own tables: every `move` row has a row at its
`dst`, and **zero** rows still name its `src`. `scripts/media-verify.py` is that
replay: it opens each database read-only (there is no `sqlite3` binary on these
hosts or inside either container) and exits non-zero while a row is stale.
Both libraries live in WAL mode, so the tool opens them plain `mode=ro`:
`immutable=1` promises sqlite the file never changes, which skips the
write-ahead log — a scanner's own uncheckpointed rows then read as missing, and
a read beside a running scan can fail as "database disk image is malformed".
`immutable` stays only the fallback for a database whose `-shm` cannot be read.
```bash
# Run this from the host shell through `podman exec`, never from a neighbour
# container: Plex's HTTPD closes without a single byte on any request that
# arrives from another container's netns (10.89.x.x reads as "empty reply", not
# as a 401). The token rides stdin, never argv, where any process on the host
# could read it out of /proc/<pid>/cmdline.
umask 077
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
podman exec -i plex sh -c 'read -r T
for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
-H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
done' </tmp/px.tok
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
```
[[Jellyfin]]'s scan key lives in `~/.config/secrets/jellyfin` (mode 600; the
same value is `JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a
mode-600 curl config, never a header argument, and it must use the
`MediaBrowser` form — Jellyfin 12.4 refuses a bare `X-Emby-Token` for any key:
"No row at the destination" is not one verdict but four. **never indexed**: the
server indexes no file of that extension (its own extension census says so) — a
sidecar, not a lost film. **gone**: the file sits at neither src nor dst, because
a later pass or the server's own extension sweep renamed it. **phantom**: a row
still naming src while the file has left src. **REAL**: no row at dst *and* the
file still at src — the only class that is this relayout's own damage. Phantoms
are normal for both servers (Plex carries thousands that predate any relayout);
re-scan to clear them and never read them as loss.
```bash
umask 077
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
"$(cat ~/.config/secrets/jellyfin)"
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
echo 'request = "POST"'; } >/tmp/jf.hdr
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
shred -u /tmp/jf.hdr
```
`media-verify.py` prints this split itself — `on disk at dst, no row names it`,
`REAL — file still at src`, `gone — at neither path` — so nobody hand-stats a
thousand paths to find out whether a film is lost or merely unindexed.
`204` is acceptance; the scan then runs on the server's own schedule.
The owning server decides whether a pass can convict anything: Plex's roots are
its `sections` rows joined to their location rows (`Movies`, `TVShows`, `Music`,
`Audiobooks`, `LiveRecording`) and Jellyfin holds no audiobook library at all, so
the audiobook journal verifies with `--db-kind plex` or it proves nothing.
Three doors that look right and are not: `GET /search?query=` matches metadata
titles, never file paths; `?path=` on `/library/sections/N/all` makes Plex's
HTTPD close without a single byte; and the same HTTPD drops requests that carry
no `Accept` and client-identifier header (see the scan block above for the form
that works).
## TV in the movie library
@@ -138,16 +160,15 @@ from the box-set folder. What it refuses it prints and leaves alone:
- a destination two files both claim: the executor refuses the whole plan
before its first rename, and only Frank decides which copy is the keeper.
- an episode the destination library already holds at the same byte count. The
movie share carried a second copy of the whole Monk box set — 124 episodes
both sides, the same episode list, every byte count identical, 31.5 GiB each
way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
such a series in place and says so; a duplicate is deleted or kept by Frank,
never copied into one library by a tool.
movie share once carried a second copy of a whole box set — same episode list,
every byte count identical. The mover leaves such a series in place and says
so; a duplicate is deleted or kept by Frank, never copied into one library by
a tool.
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
Below the threshold the mover prints and leaves, so a lone downloaded episode of
a real show stays in the movie library on purpose — six did on 2026-10-30.
a real show stays in the movie library on purpose.
Filing one by hand is the same executor on a hand-written plan and the same
journal; the trap is the series year. Read it from the file's own name or the
folder it owns, never from the title whose bonus folder it happens to be parked
@@ -158,12 +179,17 @@ show. A name whose marker is `Part 2` is a film and stays put.
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
already matched and numbered most of it. `scripts/media-shows.py` re-files that
library into the same shape, and it never guesses: every series name and every
episode number it writes traces to an authority, and what has none is parked for
a human to read.
library into the shape Jellyfin reads, with **tvdb as the name authority**:
<Series> (<year>) [tvdbid-NNNN]/Season NN/
<Series> (<year>) - SxxEyy - <Episode Title>.ext
It never guesses: every series name, episode number and title it writes traces to
an authority, and what has none is parked for a human to read.
```bash
# 1. The authority: Jellyfin's own matched rows (the DB opens read-only).
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
# and Tvdb ids and the title the row holds (the DB opens read-only).
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
--out /tmp/tv_titles.tsv
# 2. What is on disk (stat only), and the runtimes the film arm needs.
@@ -171,16 +197,33 @@ python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
--out /tmp/tv_enum.tsv
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
# The probe is the slow step: ffprobe reads both ends of every file.
# 3. The plan. --tmdb needs TMDB_API_KEY; without it a series no provider dated
# is simply named without a year, which Jellyfin accepts.
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
# the last one — a duration belongs to the file, not to its name: re-key the
# previous census through that pass's own journal (`src` -> `dst`) instead of
# seeking the array again. It is also the only column the film arm reads, and
# only for a file with no marker and no provider row (4,794 of 58,551 today,
# all of them covered by a re-key that costs seconds, not an hour).
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
# which travels as the `api_key` query parameter. Only a v4 request token is
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
# which the summary line flags as NOT provider-complete. --tvdb reads its
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
# cache file, so the plan is reproducible offline afterwards: run it once to
# fill the cache, and quote the second run.
# Paths are only strings to a dry run, so this step can run on the workstation
# against scp'd TSVs — which is where the two provider secrets live.
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
--out /tmp/tv_plan.tsv --exceptions /tmp/tv_exceptions.txt
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
# executor, journal and rollback as the movie relayout above.
scripts/media-apply.py apply --plan /tmp/tv_plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
--park /mnt/entropy/media-park/TVShows \
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
# applied pass is its unwind record, so a second pass that reuses the name
# destroys the only map back.
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
```
Evidence, strongest first — the order the planner believes things:
@@ -188,14 +231,25 @@ Evidence, strongest first — the order the planner believes things:
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`);
- the **`Season NN` folder it sits in and the number its name leads with inside
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
- the **provider row**: identity (the series name and its year), and the
numbering of a file whose own name says nothing;
- the **provider row**: which episode the file IS, and the numbering of a file
whose own name says nothing;
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
episode title, because a name the server read off a folder is not a show name.
The folder CARRIES the id: `[tvdbid-NNNN]` is what Jellyfin's Tvdb agent and
Plex's scanner both read, so a show whose title is ambiguous resolves by number
instead of by a human guessing which folder is which. The id rides the folder
only — a file name gets 180 characters and the episode title needs the room.
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
series' canonical extra folder;
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies`
only when a movie provider names it — otherwise it parks.
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
when the movie provider's title is a run of words the FILE itself states and its
year agrees with the file's: the query is only what `title_of` salvaged from a
release name, and the first hit for a short one is a near miss — a 1963 Bond film
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
parks.
Two refusals carry the design, and a dry run on the real library found both:
Four refusals carry the design: what the planner will NOT do is the point of it.
- **Folders resolve a series; they never invent one.** A folder's words may
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
@@ -205,14 +259,84 @@ Two refusals carry the design, and a dry run on the real library found both:
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
season 16 episode 20, and one Gundam row says season 2004. The folder, then
the file's own leading number, win; a row that still insists on season 1989
parks its file instead of creating `Season 1989`.
the file's own leading number, win.
- **A season that is a YEAR is the date matcher, not a season.** Jellyfin files
specials by airdate, so a row can arrive saying `season 2002`; that is not a
season and it never becomes a folder. The one authority on such a row is tvdb's
episode list read backwards: the title the row carries names the (season,
episode) tvdb gives it. A title tvdb gives to two episodes names nothing, and
the file parks with the reason it always carried. `Season 00` IS a legal
season — specials live there — so a row that really says 0 files there instead
of parking.
- **A season tvdb's own list contradicts gets no tvdb title.** An anime pack
numbered by its release group books `S05E01` in a season tvdb does not hold, or
books more episodes there than tvdb lists: the slot its number points at is a
different episode, and dressing it in that episode's title is a confident lie.
The file still renames to the standard shape; the title does not ride, and the
series is listed under *seasons whose numbering tvdb contradicts* in the
exceptions. A season holding FEWER files than tvdb lists stays plausible — a
partial pack is the normal shape. Neither reading holds for a list tvdb
cut off at its own page limit: the default episode query answers at most
500 rows and states no total, and One Piece, Crayon Shin-chan and
Detective Conan each answer exactly 500 while holding over a thousand —
a page bounds nothing above itself, so it contradicts nothing.
The provider's name is copied verbatim, warts included: Jellyfin carries a Series
named `Black Clover 001` (92 episodes, no year) because it matched a pack off a
filename. Renaming a show is Jellyfin's work — a folder rename never touches its
provider ids — so the mover keeps the name it is handed and strips only the pack
index a release leads with (`4a. Seed DESTINY` → `Seed DESTINY`).
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
after the folder it scanned, so a library filed by release folders ends up with
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
copying those names verbatim launders folder junk into the contract. The planner
resolves each series to its tvdb id and writes tvdb's name and year; the merged
identity is also what turns a box set split across five release folders into ONE
folder, and what makes two files claiming one episode MEET in the duplicate rule.
A series no id resolves keeps the provider's words and is listed under *no tvdb
identity* in the exceptions — that list, not a rename, is the review.
**The review pile lives outside the library.** Files parked inside
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
show's rows then looked like evidence. So a row whose series is the hidden
top-level folder the file itself sits in is dropped — and the folder is matched
with its leading underscore stripped, because the server *displays* `_unsorted` as
`unsorted`: compare the spelled names and the pile survives the relayout as a
visible series folder that real files get renamed into. `--park` outside the root
makes the next plan re-file the pile instead of skipping it. Inside a hidden
top-level folder a file's own words may MATCH a registered show, never invent one.
**The operator's id map is the door for a name tvdb cannot be searched by.**
`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one
line per pack folder, `folder <TAB> tvdb id <TAB> note`. The file exists because
tvdb titles a record in the show's own language and the English name rides in
*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no
spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the
2023 live action, not to the 1999 record (81797) the pack here is. The map
outranks a provider id — a box set's provider id is the server's own guess at a
folder name — and it merges the release folders that are one show. Everything it
touched is written to the exceptions: every folder it named, every row that
matched no folder, every disagreement with a provider id, every id tvdb never
answered. A line whose id is not a number stops the plan, because a typo renames
a whole pack to the wrong show. Only folders the search CANNOT reach belong in
it; a row for a folder the search would resolve hides a live lookup behind a
number nobody re-checks.
`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a
plan's `mkdir` rows, asks tvdb about every destination folder carrying no
`[tvdbid-`, and writes one row per folder with the verdict that says how much was
guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after
release junk is stripped from the folder's words), `NEAR` (tvdb answered
something else — the best hit is printed for a human and nothing applies it),
`NONE`. Acceptance never moved across the passes that built this: a hit counts
only when tvdb's name equals the query with every non-alphanumeric dropped from
both sides, and the record's year still has to fall inside any year span the
folder names. That gate is what caught three confident picks — Fairy Tail's 2024
sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black
Clover` (378762) offered for Black Clover. It is why the map is approved by hand
and the sheet is never fed to the planner.
## The music and audiobook arms
Each is a page of its own, because each reads a different authority and refuses
for a different reason: [[Media Music Relayout]] files by Jellyfin's track tags,
[[Media Audiobook Relayout]] by Plex's chapter rows. Both share this page's
executor, the one-journal-per-arm law, the sweep and the rollback.
## Laws
@@ -253,19 +377,6 @@ index a release leads with (`4a. Seed DESTINY` → `Seed DESTINY`).
free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
`/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
rolls back exactly like the move that made it.
- **A Jellyfin 401 is the header, not the key.** 12.4 authenticates only
`Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for
every key, valid or not. This page carried "no credential on this box can
trigger a scan, the two Seerr keys answer 401" for weeks because of that
header — the keys were innocent. Before believing a 401, prove the key against
the server's own table: `select count(*) from ApiKeys where AccessToken = ?`
on `jellyfin.db` opened `mode=ro&immutable=1`.
- **A scan that must see a new folder has to be recursive.** `POST
/Library/Refresh` is the incremental pass, and inotify does not cross a
rootless podman bind, so folders created on the host stay invisible to it. The
move is `POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id
from `GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and
that same record's `RefreshStatus` is the progress read.
- **A file whose name carries no words is bonus material, not a folder.**
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
is not a movie. It becomes a review line telling a human to name the file.
@@ -294,16 +405,15 @@ index a release leads with (`4a. Seed DESTINY` → `Seed DESTINY`).
recreates the folder, so the sweep is undone by the same command as the moves.
Proof carries downward: inside a folder the journal proves emptied, every
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
move source itself, so 146 of them held their emptied parents on disk after
the first sweep.
move source itself, so it can hold an emptied parent until the next sweep.
A failed `rmdir` is not a removal: only a folder that is really gone leaves
scope, so the next sweep finishes what a permission problem stopped.
- **An episode-shaped name marks a SERIES, not a film — and the series is the
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
`Monk`. Keyed on the whole title instead, every episode became a group of one
and never reached the five-episode size that calls something a series, so a
box set was filed as 124 movies in 124 folders. The threshold still earns its
keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
`Monk`. Keyed on the whole title instead, every episode became a group of one,
never reaching the five-episode size that calls something a series — a whole
box set filed as one folder per episode. The threshold still earns its keep:
a lone film named `Part 2` stays a film. Episodes of a series are parked
in `_TV needs a TV library - <series>` for a human to move to a TV source —
the movie tool never files TV.
- **One journal is one undo.** The TV move runs on its own journal, and with
@@ -314,14 +424,29 @@ index a release leads with (`4a. Seed DESTINY` → `Seed DESTINY`).
- **Sidecar metadata does not follow a rename, because it lives on another
filesystem.** The media servers write `.nfo` and artwork through the
[[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
array path. After any sweep or TV move, run `scripts/media-view.sh reap`, then
the same command with `--apply`: the sidecars left at the old coordinates become
ghost folders — artwork and an `.nfo` with no video — that both servers browse
and report as content.
array path, so where that view is mounted a sweep or a move must be followed by
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
that both servers browse and report as content. Whether this host mounts the
view at all is stated on that page; `media-apply.py` stays silent when it does not.
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
whiteout in the upper layer, which hides the array's copy without removing it;
the file returns when the view is unmounted. Deletions in this pipeline happen
on the raw path only, which is what the journal can undo.
- **The music album key is case-folded; the folder name is not.** `Master Of
Puppets` and `Master of Puppets` are one album to the matcher, so the spelling
that reaches the folder is the one most tracks carry. Taking it from whichever
row sorted first made the folder name depend on row order, and re-planning an
already-filed library renamed albums that were already in place.
- **An occupied path is never a destination**, not even when its occupant is
scheduled to leave: the executor refuses an occupied target, so a plan that
aims at one fails after the renames that were supposed to clear it. A file
standing where the contract puts it is a keep, and a file already carrying a
collision suffix keeps that suffix — otherwise the plan is a shuffle, not a
fixed point, and a re-run moves albums that are already right.
- **`media-census.py music` reads the `BaseItemProviders` table, not the item's
JSON.** This Jellyfin build carries an empty `ProviderIds` inside
`BaseItems.Data` for albums, so a census that parses the JSON finds no
MusicBrainz or Spotify id on any album and reads as "the library is untagged".
## Rollback
@@ -331,6 +456,14 @@ any folder a sweep removed, restores every source path, removes only the folders
that run created *and that are now empty*, and refuses everything if any moved
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
**Roll a chain back newest journal first, or the tool refuses.** Each pass
re-files what an earlier one moved, so a journal is a clean inverse only until
something downstream touches its files; `ls -l /mnt/entropy/media-relayout*.jsonl`
is the order. Refusing is the tool working: on 2026-10-31 the movie journal could
not restore files the later TV passes had re-filed, and the stray journal could
not restore files Jellyfin's extension sweep had renamed. Roll back what still
inverts and leave the rest as the current state — never force it.
## See Also
[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
+45 -2
View File
@@ -27,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — every library path in the PMS database stayed valid, zero re-scan |
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
@@ -40,7 +40,8 @@ public traffic can be fronted by Caddy like every other vhost.
The migration moved zero bytes: the container bind-mounts the native data
root where it already lived, and the media archive re-mounts at
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
(array, then the overlay view), so every absolute library path in the
(the array today; the overlay view in git has never been deployed), so every
absolute library path in the
PMS database keeps resolving. The one metadata change: rootless podman
cannot map the real host uid 997, so the fleet model applies — container
root runs as the `dank` user, and the deployer adopts the data root with
@@ -86,3 +87,45 @@ root runs as the `dank` user, and the deployer adopts the data root with
while the HTTP layer finishes loading; the first boot of a years-old
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
600 s before failing; later boots are much faster (warm page cache).
## Triggering a scan
A rename on disk is invisible until the section is rescanned, and a moved file is
a `Couldn't find the file to stream` 404 until that scan passes it. Run this from
the host shell through `podman exec`, never from a neighbour container: PMS's
HTTPD closes without a single byte on any request that arrives from another
container's netns (10.89.x.x reads as "empty reply", not as a 401). The token
rides stdin, never argv, where any process on the host could read it out of
`/proc/<pid>/cmdline`; it lives in `Preferences.xml` inside the RW data bind.
```bash
umask 077
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
cat >/tmp/pxscan.sh <<'EOS'
read -r T
cfg=$(mktemp)
mk() { { printf 'header = "X-Plex-Token: %s"\nheader = "Accept: application/json"\n' "$T"
printf 'url = "%s"\n' "$1"; } > "$cfg"; }
mk "http://127.0.0.1:32400/library/sections"
for k in $(curl -s -K "$cfg" | grep -o '"key":"[0-9]*"' | cut -d\" -f4); do
mk "http://127.0.0.1:32400/library/sections/$k/refresh"
printf 'request = "POST"\n' >> "$cfg"
echo "plex section $k: $(curl -s -o /dev/null -w '%{http_code}' -K "$cfg")"
done
shred -u "$cfg"
EOS
podman cp /tmp/pxscan.sh plex:/tmp/pxscan.sh
podman exec -i plex sh /tmp/pxscan.sh </tmp/px.tok
shred -u /tmp/px.tok
```
Every section answers `204` — or `200` with a Directory body when
`Accept: application/json` is sent. Inside the quoted `printf` format a
backslash-newline is LITERAL, not a continuation, so `mk` writes two `printf`s:
one glued line and one stray backslash is a config file curl rejects, and the
failure prints as `option -K: is unknown`, never as a bad config line.
Proof that a rename landed is not a scan response: `scripts/media-verify.py`
replays the run's journal against the database — the only read that
distinguishes "not scanned yet" from "gone"; see [[Media Relayout]].
+4 -2
View File
@@ -48,11 +48,13 @@ is a locator: one row, one clause, no numbers.
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. |
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Entropy Volume]] | Dankserver's /mnt/entropy volume: the LUKS/array chain, the crypttab + fstab boot path, and the containers bound to it |
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
| [[Media Relayout]] | Census, plan and apply the movie and TV libraries into Jellyfin format: gates, evidence chain, refusals, rollback. |
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
+483
View File
@@ -0,0 +1,483 @@
#!/usr/bin/env bats
# Seam: the audiobook-library relayout — scripts/media-census.py audiobooks (the
# export of the rows Plex matched) and scripts/media-audiobooks.py (the plan).
#
# The audiobook tree is the one library on this box that Jellyfin does not read
# at all and Plex reads completely: every chapter here is a Plex Track row under
# an Album row (the BOOK) under one Artist row (the NARRATOR). That is the
# authority this seam files by — never the collection folder a release pack made.
#
# Two laws here are the opposite of the music tool's, and both come from what the
# rows actually contain. Files are NEVER renumbered: 7,556 of 10,485 books carry
# at least one chapter whose index Plex does not know, so a `00 -` invented from
# a missing index would order a book wrong, and playback order comes from the
# embedded track tag either way — the name buys nothing and can only lie. And the
# folder artist is the NARRATOR even though a reader thinks in authors: every
# book has exactly one artist row, while the author survives only in 10% of the
# album titles (`Kryptos by Graham McNeill`) and in no column Plex stores, so one
# rule beats a shelf that is half authors and half narrators.
#
# Every test states which authority a file has and asserts where the PLAN files
# it, never how the planner decided.
load helpers
PLAN="$REPO_ROOT/scripts/media-audiobooks.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# row FILE NARRATOR BOOK YEAR TRACK CHAPTER
# One row of the authority export scripts/media-census.py audiobooks writes.
row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
"$1" "$2" "$3" "${4:-}" "${5:-}" "$6" >> "$ROWS"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
dst() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
}
# kind SRC — the kind the plan gave SRC's move row.
kind() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $4}' "$OUT"
}
setup() {
AB="$BATS_TEST_TMPDIR/Audiobooks"
PARK="$AB/_unsorted"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
# The header is part of the authority export, so the fixture carries it.
printf 'file\tnarrator\tbook\tyear\ttrack\tchapter\n' > "$ROWS"
# One book of one author, split across the two part folders a pack shipped it
# in, with a cover above part 1 and a playlist beside the chapters.
A="$AB/Stephen King/It (1986)"
at "$A [Part 1]/01 - Chapter 1.mp3" 9000
at "$A [Part 1]/02 - Chapter 2.mp3" 9000
at "$A [Part 1]/cover.jpg" 20
at "$A [Part 1]/It.m3u" 30
at "$A [Part 2]/03 - Chapter 3.mp3" 9000
row "$A [Part 1]/01 - Chapter 1.mp3" "George Guidall" "It" 1986 1 "Chapter 1"
row "$A [Part 1]/02 - Chapter 2.mp3" "George Guidall" "It" 1986 2 "Chapter 2"
row "$A [Part 2]/03 - Chapter 3.mp3" "George Guidall" "It" 1986 3 "Chapter 3"
# A book whose album row carries no year at all.
B="$AB/Collected/Project Gold"
at "$B/01.ogg" 9000
row "$B/01.ogg" "Operator X" "Project Gold" "" 1 "Opening"
# One book title, two narrators: two different books.
C="$AB/Shared"
at "$C/Dune side a.m4b" 9000
at "$C/Dune side b.m4b" 9000
at "$C/Dune 2020.m4b" 9000
row "$C/Dune side a.m4b" "Scott Brick" "Dune" 1965 1 "Prologue"
row "$C/Dune side b.m4b" "Scott Brick" "Dune" 1965 2 "Dune"
row "$C/Dune 2020.m4b" "Malcolm Hillgartner" "Dune" 2020 1 "Book One"
# Two part folders of one book whose chapters share a file name.
D="$AB/Pulp"
at "$D/Book A/vol1.mp3" 9000
at "$D/Book A vol2/vol1.mp3" 9000
row "$D/Book A/vol1.mp3" "Nora Pace" "Hard Food" 2001 1 "Vol 1"
row "$D/Book A vol2/vol1.mp3" "Nora Pace" "Hard Food" 2001 2 "Vol 1"
# The same narrator spelled two ways across one book's chapters.
E="$AB/Spell"
at "$E/kharn 1.mp3" 9000
at "$E/kharn 2.mp3" 9000
at "$E/kharn 3.mp3" 9000
row "$E/kharn 1.mp3" "David Timson" "Kharn" 2013 1 "One"
row "$E/kharn 2.mp3" "David timson" "Kharn" 2013 2 "Two"
row "$E/kharn 3.mp3" "David Timson" "Kharn" 2013 3 "Three"
# A folder holding two different books: its loose note has no owner.
F="$AB/Mixed"
at "$F/book one chapter.mp3" 9000
at "$F/book two chapter.mp3" 9000
at "$F/notes.txt" 10
row "$F/book one chapter.mp3" "Ann Reader" "Book One" 2001 1 "One"
row "$F/book two chapter.mp3" "Bob Reader" "Book Two" 2002 1 "Two"
# A file already parked: the pile is for a human, so a row that would file it
# must not pull it back into the tree.
at "$PARK/Old/nobody.mp3" 9000
row "$PARK/Old/nobody.mp3" "Ann Reader" "Book One" 2001 1 "One"
# The odd cases: no row, a video container, a name carrying a character Plex
# says breaks playback, rows with a blank book and a blank narrator, and a
# matched chapter whose file has no extension.
G="$AB/Odd"
at "$G/loose.mp3" 9000
at "$G/concert.mp4" 9000
at "$G/Bad: Name/01 chapter.mp3" 9000
row "$G/Bad: Name/01 chapter.mp3" "Some Reader" "Trail: Marked" 1999 1 "Marked"
at "$G/blank book.mp3" 9000
row "$G/blank book.mp3" "Some Reader" "" 2000 1 "No Book"
at "$G/blank narrator.mp3" 9000
row "$G/blank narrator.mp3" "" "No Artist" 2000 1 "No Artist"
at "$G/NoExt/part01" 9000
row "$G/NoExt/part01" "Archivist" "Tapes" 1971 1 "Tape One"
# Tag-scan placeholders. Two unrelated books whose album rows both read
# `[Unknown Album]`: the name is shared, so filing on it would merge books
# that have nothing to do with each other. A rip timestamp can be glued to
# the placeholder, and a multi-cast read says `Various Artists` — which IS a
# narrator.
H="$AB/Packs"
at "$H/Stross/Ship/c1.mp3" 9000
at "$H/Leckie/State/c1.mp3" 9000
at "$H/Rip/tape.mp3" 9000
at "$H/NoWho/Real Book.mp3" 9000
at "$H/Multicast/Genesis.mp3" 9000
row "$H/Stross/Ship/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
row "$H/Leckie/State/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
row "$H/Rip/tape.mp3" "Ann Reader" "Unknown album (03/03/2015 21:39:55)" "" 1 "One"
row "$H/NoWho/Real Book.mp3" "[Unknown Artist]" "A Named Book" 2001 1 "One"
row "$H/Multicast/Genesis.mp3" "Various Artists" "The Bible" "" 1 "Genesis"
at "$H/Numbered/01 - Grimoire.mp3" 9000
at "$H/Digits/1984.mp3" 9000
row "$H/Numbered/01 - Grimoire.mp3" "0.4" "Grimoire of the Lamb" "" 1 "One"
row "$H/Digits/1984.mp3" "Ann Reader" "1984" 1949 1 "One"
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
# path, no header.
python3 - "$AB" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
}
plan() {
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS" \
--out "$OUT" --exceptions "$EXC"
if [ "$status" -ne 0 ]; then
echo "planner said: $output"
return 1
fi
}
# db_with_part DB FILE NARRATOR BOOK YEAR TRACK CHAPTER — one Plex Track row
# under an Album under an Artist, in the three tables the export joins.
db_with_part() {
local db="$1" file="$2" narrator="$3" book="$4" year="$5" track="$6" chapter="$7"
local del="${8:-0}"
python3 - "$db" "$file" "$narrator" "$book" "$year" "$track" "$chapter" "$del" <<'PY'
import sqlite3, sys
db, file, narrator, book, year, track, chapter, deleted = sys.argv[1:9]
c = sqlite3.connect(db)
c.execute("create table if not exists media_parts (id integer primary key,"
" media_item_id, file, deleted_at)")
c.execute("create table if not exists media_items (id integer primary key,"
" metadata_item_id, deleted_at)")
c.execute("""create table if not exists metadata_items (id integer primary key,
parent_id, metadata_type, title, year, "index", deleted_at)""")
now = None if deleted == "0" else "2026-01-01"
ar = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (NULL, 8, ?, NULL, NULL, NULL)",
(narrator,)).lastrowid
al = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (?, 9, ?, ?, NULL, NULL)",
(ar, book, int(year) if year else None)).lastrowid
mi = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (?, 10, ?, NULL, ?, NULL)",
(al, chapter, int(track) if track else -1)).lastrowid
it = c.execute("insert into media_items (metadata_item_id, deleted_at) VALUES (?, NULL)",
(mi,)).lastrowid
c.execute("insert into media_parts (media_item_id, file, deleted_at) VALUES (?,?,?)",
(it, file, now))
c.commit()
PY
}
# ------------------------------------------------------------------ the export
@test "the census exports the narrator, book and chapter Plex matched for a part" {
DB="$BATS_TEST_TMPDIR/plex.db"
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01.mp3" \
"George Guidall" "It" 1986 1 "Chapter 1"
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file narrator book year track chapter" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01\.mp3 George Guidall It 1986 1 Chapter 1$"
}
@test "the census exports nothing outside the root, and nothing Plex deleted" {
DB="$BATS_TEST_TMPDIR/plex.db"
db_with_part "$DB" "/mnt/other/Library/01.mp3" "Someone" "Elsewhere" 1990 1 "One"
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Gone/01.mp3" "Someone" \
"Deleted" 1990 1 "One" 1
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
[ "$(wc -l < "$BATS_TEST_TMPDIR/exp.tsv")" -eq 1 ] # header only
}
# wal_fresh DB ROOT OUT — a row Plex matched during a scan and committed, but
# has not checkpointed: the normal state of a library being walked. The holder
# connection stops the WAL folding into the main file when the writer closes,
# and the census runs as a real subprocess against those same files.
wal_fresh() {
local db="$1" root="$2" out="$3"
python3 - "$db" "$root" "$out" "$CENSUS" <<'PY'
import sqlite3, subprocess, sys
db, root, out, census = sys.argv[1:5]
w = sqlite3.connect(db)
w.execute("pragma journal_mode=wal")
w.execute("pragma wal_autocheckpoint=0")
w.commit()
w.execute("pragma wal_checkpoint(truncate)") # the rows so far are on disk
holder = sqlite3.connect(db) # last close would checkpoint
ar = w.execute('insert into metadata_items (parent_id, metadata_type, title,'
' year, "index", deleted_at) VALUES (NULL, 8, ?, NULL, NULL, NULL)',
("Fresh Narrator",)).lastrowid
al = w.execute('insert into metadata_items (parent_id, metadata_type, title,'
' year, "index", deleted_at) VALUES (?, 9, ?, ?, NULL, NULL)',
(ar, "Fresh Book", 2026)).lastrowid
mi = w.execute('insert into metadata_items (parent_id, metadata_type, title,'
' year, "index", deleted_at) VALUES (?, 10, ?, NULL, ?, NULL)',
(al, "Fresh Chapter", 1)).lastrowid
it = w.execute("insert into media_items (metadata_item_id, deleted_at)"
" VALUES (?, NULL)", (mi,)).lastrowid
w.execute("insert into media_parts (media_item_id, file, deleted_at)"
" VALUES (?, ?, NULL)", (it, root + "/Fresh/01.mp3"))
w.commit()
sys.exit(subprocess.run([sys.executable, census, "audiobooks", "--root", root,
"--db", db, "--out", out]).returncode)
PY
}
@test "the census exports a row the scanner committed but has not checkpointed" {
DB="$BATS_TEST_TMPDIR/plex.db"
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Old/01.mp3" \
"Old Narrator" "Old Book" 1990 1 "Old Chapter"
run wal_fresh "$DB" "$BATS_TEST_TMPDIR/Audiobooks" "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
[ "$(wc -l < "$BATS_TEST_TMPDIR/exp.tsv")" -eq 3 ] # header + both rows
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Audiobooks/Fresh/01\.mp3 Fresh Narrator Fresh Book 2026 1 Fresh Chapter$"
}
# -------------------------------------------------------------------- the plan
@test "a matched chapter files under its narrator and its book, name untouched" {
plan
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
}
@test "one book arrives in one folder however many part folders it shipped in" {
plan
[ "$(dst "$A [Part 2]/03 - Chapter 3.mp3")" = \
"$AB/George Guidall/It (1986)/03 - Chapter 3.mp3" ]
[ "$(dst "$A [Part 1]/02 - Chapter 2.mp3")" = \
"$AB/George Guidall/It (1986)/02 - Chapter 2.mp3" ]
}
@test "a chapter is never renumbered, whether or not Plex knows its index" {
# The law this guards: playback order comes from the embedded track tag, and
# three quarters of the books have chapters with no index at all, so a
# number written into a name would be invented and would lie.
plan
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
}
@test "a book whose album row has no year gets no parentheses" {
plan
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
}
@test "two chapters claiming one name inside one book are named apart" {
plan
first="$(dst "$D/Book A/vol1.mp3")"
second="$(dst "$D/Book A vol2/vol1.mp3")"
[ "$first" != "$second" ]
[ -n "$first" ] && [ -n "$second" ]
# and the plan still passes the executor's preflight, which refuses a target
# two sources claim.
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
[ "$status" -eq 0 ]
[[ "$output" != *"double-claim"* ]]
}
@test "a cover follows the book whose folder it lies in" {
plan
[ "$(dst "$A [Part 1]/cover.jpg")" = "$AB/George Guidall/It (1986)/cover.jpg" ]
}
@test "a playlist moves with its book and its bytes are never rewritten" {
plan
[ "$(dst "$A [Part 1]/It.m3u")" = "$AB/George Guidall/It (1986)/It.m3u" ]
before=$(md5sum "$A [Part 1]/It.m3u" | cut -d" " -f1)
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" --journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
[ "$status" -eq 0 ]
[ "$(md5sum "$AB/George Guidall/It (1986)/It.m3u" | cut -d" " -f1)" = "$before" ]
# the stored paths inside it still name the old file names, which the tool
# leaves alone rather than editing a file the operator owns.
[ "$(kind "$A [Part 1]/It.m3u")" != "chapter" ]
}
@test "a note in a folder holding two books has no owner and parks" {
plan
[ "$(kind "$F/notes.txt")" = "park" ]
[ "$(dst "$F/notes.txt")" = "$PARK/Mixed/notes.txt" ]
}
@test "an audio file no row names parks with the path it has today" {
plan
[ "$(kind "$G/loose.mp3")" = "park" ]
[ "$(dst "$G/loose.mp3")" = "$PARK/Odd/loose.mp3" ]
}
@test "a video container parks instead of being renamed into audio" {
plan
[ "$(kind "$G/concert.mp4")" = "park" ]
run grep -c "concert\.mp4" "$EXC"
[ "$output" != "0" ]
}
@test "a character Plex says breaks playback never reaches a destination" {
plan
[ "$(dst "$G/Bad: Name/01 chapter.mp3")" = \
"$AB/Some Reader/Trail Marked (1999)/01 chapter.mp3" ]
run cut -f3 "$OUT"
[[ "$output" != *":"* ]]
}
@test "one book title with two narrators stays two books" {
plan
[ "$(dst "$C/Dune side a.m4b")" = "$AB/Scott Brick/Dune (1965)/Dune side a.m4b" ]
[ "$(dst "$C/Dune 2020.m4b")" = \
"$AB/Malcolm Hillgartner/Dune (2020)/Dune 2020.m4b" ]
}
@test "the spelling most chapters carry names the folder, and they share one folder" {
plan
[ "$(dst "$E/kharn 1.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 1.mp3" ]
[ "$(dst "$E/kharn 2.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 2.mp3" ]
[ "$(dst "$E/kharn 3.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 3.mp3" ]
}
@test "a row with no book title parks rather than inventing a folder" {
plan
[ "$(kind "$G/blank book.mp3")" = "park" ]
}
@test "a row with no narrator parks rather than inventing a shelf" {
plan
[ "$(kind "$G/blank narrator.mp3")" = "park" ]
}
@test "books that share a placeholder album name never share a shelf" {
plan
[ "$(dst "$H/Stross/Ship/c1.mp3")" = "$PARK/Packs/Stross/Ship/c1.mp3" ]
[ "$(dst "$H/Leckie/State/c1.mp3")" = "$PARK/Packs/Leckie/State/c1.mp3" ]
}
@test "a placeholder with a rip timestamp glued to it is still a placeholder" {
plan
[ "$(kind "$H/Rip/tape.mp3")" = "park" ]
}
@test "a narrator placeholder parks even when the book is named" {
plan
[ "$(kind "$H/NoWho/Real Book.mp3")" = "park" ]
}
@test "Various Artists is a multi-cast narrator and files its book" {
plan
[ "$(dst "$H/Multicast/Genesis.mp3")" = \
"$AB/Various Artists/The Bible/Genesis.mp3" ]
}
@test "a chapter number mistagged as the narrator parks" {
plan
[ "$(kind "$H/Numbered/01 - Grimoire.mp3")" = "park" ]
}
@test "a book whose title is digits still files, because only a name must have a letter" {
plan
[ "$(dst "$H/Digits/1984.mp3")" = "$AB/Ann Reader/1984 (1949)/1984.mp3" ]
}
@test "the row, not the extension, decides what a chapter is" {
plan
[ "$(kind "$G/NoExt/part01")" = "chapter" ]
[ "$(dst "$G/NoExt/part01")" = "$AB/Archivist/Tapes (1971)/part01" ]
}
@test "the plan replays through the executor with nothing refused" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
[ "$status" -eq 0 ]
[[ "$output" != *"refusing"* ]]
# The refusal this guards is two sources claiming one target: the executor
# stops the whole run on it, so the plan must never hand a path out twice.
[ -z "$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)" ]
}
@test "an occupied destination is refused rather than overwritten" {
plan
mkdir -p "$AB/George Guidall/It (1986)"
at "$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" 1
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
[ "$status" -ne 0 ]
[[ "$output" == *"occupied"* ]]
[ -e "$A [Part 1]/01 - Chapter 1.mp3" ]
}
@test "applying the plan and re-planning moves nothing" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
--journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
[ "$status" -eq 0 ]
python3 - "$AB" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
# The rows still name the paths the chapters were placed at: that is what a
# server re-scan of the moved tree reports.
python3 - "$OUT" "$ROWS" <<'PY'
import csv, sys
out, rows = sys.argv[1], sys.argv[2]
moved = {r[1]: r[2] for r in csv.reader(open(out), delimiter="\t") if r[0] == "move"}
with open(rows) as fh, open(rows + ".2", "w") as fh2:
fh2.write(fh.readline().rstrip("\n") + "\n")
for line in fh:
f = line.split("\t")[0]
fh2.write("\t".join([moved.get(f, f)] + line.rstrip("\n").split("\t")[1:]) + "\n")
PY
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS.2" \
--out "$OUT.2" --exceptions "$EXC.2"
[ "$status" -eq 0 ]
[ "$(awk -F '\t' '$1 == "move"' "$OUT.2" | wc -l)" -eq 0 ]
}
@test "nothing under the park root is planned again" {
plan
run awk -F '\t' '$1 == "move" && $2 ~ /_unsorted/' "$OUT"
[ -z "$output" ]
}
+373
View File
@@ -0,0 +1,373 @@
#!/usr/bin/env bats
# Seam: the music-library relayout — scripts/media-census.py music (the tag and
# provider authority export) and scripts/media-music.py (the plan).
#
# Jellyfin's music rule is the opposite of the movie and TV ones: the embedded
# tags are the metadata and the file name is only a fallback title, so the
# authority here is the Audio row Jellyfin already matched (its album-artist,
# album, year, disc and track columns) and NOT the path a ripper chose. Every
# test states which authority a file has — row only, row with no album item,
# sidecar of a file that has a row, nothing at all — and asserts where the PLAN
# files it, never how the planner decided.
#
# The three outcomes that cost a re-run of a 90k-file plan: a track filed under
# the wrong artist (it splits one album across two artists), a destination that
# two tracks claim (the executor refuses, and the operator loses the run), and a
# lyric file that stops matching the basename of the track it belongs to, which
# silently loses the lyrics. Everything below is written to catch those, plus the
# gate that matters most: the plan must replay through scripts/media-apply.py,
# whose preflight is the fleet's only overwrite guard.
load helpers
PLAN="$REPO_ROOT/scripts/media-music.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# row FILE ARTIST ALBUM YEAR DISC TRACK TITLE [MB] [PERFORMER]
# One row of the authority export scripts/media-census.py music writes.
row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$1" "$2" "$3" "${4:-}" "${5:-}" "${6:-}" "$7" "${8:-}" "${9:-}" >> "$ROWS"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
dst() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
}
setup() {
MUSIC="$BATS_TEST_TMPDIR/Music"
PARK="$MUSIC/_unsorted"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
# A discography pack: the artist's own albums buried under "01. Albums",
# with a lyric, a cover, an EAC log and a playlist beside the tracks.
A="$MUSIC/Artist One - Discography [FLAC]/01. Albums/Album X"
at "$A/01. First.flac" 9000
at "$A/02. Second.flac" 9000
at "$A/01. First.lrc" 10
at "$A/cover.jpg" 20
at "$A/rip.log" 10
at "$A/Album X.m3u" 10
# A multi-disc compilation: one album (the album artist is the compilation
# tag), two discs, and a different performer on each track.
B="$MUSIC/Box (2019)"
at "$B/Disc 1/01 Track A.flac" 9000
at "$B/Disc 2/01 Track B.flac" 9000
# The box cover lies above the two disc folders: the discs below it own it.
at "$B/box.jpg" 10
# One image plus its cuesheet: a whole album in one file.
C="$MUSIC/Image Cue"
at "$C/Image.flac" 9000
at "$C/Image.cue" 10
# The hard cases: no row at all, a video container, two tracks claiming one
# number and one title, a track with no number, and a track whose album
# artist tag is empty so only the performer names it.
D="$MUSIC/Odd"
at "$D/loose.flac" 9000
at "$D/concert.mkv" 9000
at "$D/dup1.flac" 9000
at "$D/dup2.flac" 9000
at "$D/unnumbered.flac" 9000
at "$D/lone.flac" 9000
# A sidecar in a folder that holds two different albums.
E="$MUSIC/Mixed"
at "$E/one.flac" 9000
at "$E/two.flac" 9000
at "$E/notes.pdf" 10
# An album title carrying a character Jellyfin says breaks playback.
F="$MUSIC/Question Mark"
at "$F/01. Bang.flac" 9000
# An album whose tags name no album artist at all, and two different
# performers; and one album title carried by two different album artists.
G="$MUSIC/Tagless"
at "$G/one.flac" 9000
at "$G/two.flac" 9000
H="$MUSIC/Same Title"
at "$H/a.flac" 9000
at "$H/b.flac" 9000
# One album whose tracks disagree on the spelling of its name; the minority
# spelling sorts first, so it must not win the folder.
I="$MUSIC/Puppets"
at "$I/one.flac" 9000
at "$I/two.flac" 9000
at "$I/three.flac" 9000
# The authority export carries a header; the planner reads it by column name.
printf 'file\tartist\talbum\tyear\tdisc\ttrack\ttitle\tmb\tperformer\n' > "$ROWS"
row "$A/01. First.flac" "Artist One" "Album X" 2015 1 1 "First Song"
row "$A/02. Second.flac" "Artist One" "Album X" 2015 1 2 "Second Song: Reprise"
row "$B/Disc 1/01 Track A.flac" "Various Artists" "Best Of" 2019 1 1 "Track A" \
"" "Aaa Singer"
row "$B/Disc 2/01 Track B.flac" "Various Artists" "Best Of" 2019 2 1 "Track B" \
"" "Bbb Singer"
row "$C/Image.flac" "Solo" "Single Album" 2001 1 1 "Single Album"
row "$D/dup1.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
row "$D/dup2.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
row "$D/unnumbered.flac" "No Number" "No Number Album" "" "" "" "Untitled"
row "$D/lone.flac" "" "Session Works" "" "" 1 "Lone" "" "Session Singer"
row "$E/one.flac" "Mix Artist" "First Set" 2000 "" 1 "One"
row "$E/two.flac" "Mix Artist" "Second Set" 2001 "" 1 "Two"
row "$F/01. Bang.flac" "Billie" "WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?" \
2019 "" 1 "!!!!!!"
row "$G/one.flac" "" "Two Seasons" 2005 "" 1 "One" "" "Alpha Band"
row "$G/two.flac" "" "Two Seasons" 2005 "" 2 "Two" "" "Beta Band"
row "$H/a.flac" "Aaa Singer" "Same Title" 2011 "" 1 "A Song"
row "$H/b.flac" "Bbb Singer" "Same Title" 2011 "" 1 "B Song"
row "$I/one.flac" "Metallica" "Master Of Puppets" 2015 "" 1 "One"
row "$I/two.flac" "Metallica" "Master of Puppets" 2015 "" 2 "Two"
row "$I/three.flac" "Metallica" "Master of Puppets" 2015 "" 3 "Three"
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
# path, no header.
python3 - "$MUSIC" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
}
plan() {
run python3 "$PLAN" --root "$MUSIC" --enum "$ENUM" --rows "$ROWS" \
--out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ] || echo "planner said: $output"
}
@test "the census exports the tag and provider row Jellyfin matched for a track" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" <<'PY'
import sqlite3, sys, uuid
c = sqlite3.connect(sys.argv[1])
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, Album,
AlbumArtists, Artists, Data)""")
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
AU = "MediaBrowser.Controller.Entities.Audio.Audio"
AL = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
aid, trk = str(uuid.uuid4()), "/data/Music/Pack/Album X/01. First.flac"
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(aid, None, AL, "Album X", "/data/Music/Pack/Album X", None, None, 2015,
None, "Artist One", None, '{}'))
c.execute("insert into BaseItemProviders values (?,?,?)",
(aid, "MusicBrainzReleaseGroup", "rg-1234"))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), aid, AU, "First Song", trk, 1, 1, 2015,
"album x (the tag)", "Artist One", "Artist One", '{}'))
c.commit()
PY
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file artist album year disc track title mb performer" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/Album X/01\. First\.flac Artist One Album X 2015 1 1 First Song rg-1234 Artist One$"
}
@test "a track whose folder is no album item still exports from its own tags" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" <<'PY'
import sqlite3, sys, uuid
c = sqlite3.connect(sys.argv[1])
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, Album,
AlbumArtists, Artists, Data)""")
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), None, "MediaBrowser.Controller.Entities.Audio.Audio",
"Lone", "/data/Music/Pack/lone.flac", 1, None, None,
"Session Works", None, "Session Singer", '{}'))
c.commit()
PY
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/lone\.flac Session Singer Session Works 1 Lone Session Singer$"
}
@test "a tagged track files under artist, album and year with a numbered title" {
plan
[ "$status" -eq 0 ]
[ "$(dst "$A/01. First.flac")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.flac" ]
}
@test "the tag title wins over the name the ripper gave the file" {
plan
[ "$(dst "$A/02. Second.flac")" = "$MUSIC/Artist One/Album X (2015)/02 - Second Song Reprise.flac" ]
}
@test "a character Jellyfin says breaks playback never reaches a destination" {
plan
[ "$(dst "$F/01. Bang.flac")" = "$MUSIC/Billie/WHEN WE ALL FALL ASLEEP, WHERE DO WE GO (2019)/01 - !!!!!!.flac" ]
run awk -F '\t' '$1 == "move" && $3 ~ /[<>:"|?*]/ {print $3}' "$OUT"
[ -z "$output" ]
}
@test "every disc of a multi-disc album lands in one folder, numbered apart" {
plan
[ "$(dst "$B/Disc 1/01 Track A.flac")" = "$MUSIC/Various Artists/Best Of (2019)/1.01 - Track A.flac" ]
[ "$(dst "$B/Disc 2/01 Track B.flac")" = "$MUSIC/Various Artists/Best Of (2019)/2.01 - Track B.flac" ]
}
@test "an album whose tracks name several artists files under Various Artists" {
plan
run awk -F '\t' '$1 == "move" && $3 ~ /\/Best Of \(2019\)\/.*\.flac$/ {print $3}' "$OUT"
[ "${#lines[@]}" -eq 2 ]
[[ "${lines[0]}" == "$MUSIC/Various Artists/"* ]]
}
@test "an album with no album artist and two performers is a compilation" {
plan
[ "$(dst "$G/one.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/01 - One.flac" ]
[ "$(dst "$G/two.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/02 - Two.flac" ]
}
@test "the same album title under two album artists is two albums" {
plan
[ "$(dst "$H/a.flac")" = "$MUSIC/Aaa Singer/Same Title (2011)/01 - A Song.flac" ]
[ "$(dst "$H/b.flac")" = "$MUSIC/Bbb Singer/Same Title (2011)/01 - B Song.flac" ]
}
@test "the folder carries the album spelling most tracks carry" {
plan
[ "$(dst "$I/one.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/01 - One.flac" ]
[ "$(dst "$I/two.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/02 - Two.flac" ]
[ "$(dst "$I/three.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/03 - Three.flac" ]
}
@test "a lyric file is renamed to match the track it belongs to" {
plan
[ "$(dst "$A/01. First.lrc")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.lrc" ]
}
@test "the album image, the rip log and the playlist follow the album folder" {
plan
[ "$(dst "$A/cover.jpg")" = "$MUSIC/Artist One/Album X (2015)/cover.jpg" ]
[ "$(dst "$A/rip.log")" = "$MUSIC/Artist One/Album X (2015)/rip.log" ]
[ "$(dst "$A/Album X.m3u")" = "$MUSIC/Artist One/Album X (2015)/Album X.m3u" ]
}
@test "a cover above the disc folders of one album follows that album" {
plan
[ "$(dst "$B/box.jpg")" = "$MUSIC/Various Artists/Best Of (2019)/box.jpg" ]
}
@test "an image carrying a whole album keeps its own name beside its cuesheet" {
plan
[ "$(dst "$C/Image.flac")" = "$MUSIC/Solo/Single Album (2001)/Image.flac" ]
[ "$(dst "$C/Image.cue")" = "$MUSIC/Solo/Single Album (2001)/Image.cue" ]
}
@test "an audio file no provider row names is parked, not filed under an artist" {
plan
[[ "$(dst "$D/loose.flac")" == "$PARK/Odd/loose.flac" ]]
}
@test "a video container in the music library is parked, never renamed to an audio one" {
plan
[[ "$(dst "$D/concert.mkv")" == "$PARK/Odd/concert.mkv" ]]
run awk -F '\t' '$1 == "move" && $3 ~ /\.mka$|\.m4a$/ {print $3}' "$OUT"
[ -z "$output" ]
}
@test "two tracks claiming one number and one title get two destinations" {
plan
[ "$(dst "$D/dup1.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same.flac" ]
[ "$(dst "$D/dup2.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same (2).flac" ]
}
@test "an album with no year is named without one, a track with no number likewise" {
plan
[ "$(dst "$D/unnumbered.flac")" = "$MUSIC/No Number/No Number Album/Untitled.flac" ]
}
@test "a track naming only a performer files under that performer" {
plan
[ "$(dst "$D/lone.flac")" = "$MUSIC/Session Singer/Session Works/01 - Lone.flac" ]
}
@test "a sidecar in a folder holding two albums parks instead of guessing" {
plan
[[ "$(dst "$E/notes.pdf")" == "$PARK/Mixed/notes.pdf" ]]
}
@test "the plan creates every folder before the move that needs it" {
plan
python3 - "$OUT" "$MUSIC" <<'PY'
import os, sys
made = set()
for line in open(sys.argv[1]).read().splitlines()[1:]:
action, _src, dest, _kind = line.split("\t")
if action == "mkdir":
made.add(dest)
elif action == "move":
parent = os.path.dirname(dest)
while parent and parent != sys.argv[2]:
assert parent in made, f"{dest} needs {parent}, which is never made"
parent = os.path.dirname(parent)
PY
}
@test "no move lands outside the music library" {
plan
run awk -F '\t' -v l="$MUSIC" \
'$1 == "move" && $3 !~ "^" l "/" {print $3}' "$OUT"
[ -z "$output" ]
}
@test "the executor preflights the plan and no target is claimed twice" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$BATS_TEST_TMPDIR" \
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
[ "$status" -eq 0 ]
dup=$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)
[ -z "$dup" ]
}
@test "re-planning a library the plan already satisfies moves nothing" {
plan
# Applying the plan and re-censusing is what the operator does next: both
# authority files are keyed by path, so both are replayed through the plan
# the way a fresh census of the moved tree would see them. (The enum has no
# header; the rows export does, and its header row is not a path.)
python3 - "$OUT" "$ENUM" "$ROWS" <<'PY'
import os, sys
plan = sys.argv[1]
moves = {}
for line in open(plan).read().splitlines()[1:]:
action, src, dest, _kind = line.split("\t")
if action == "move":
moves[src] = dest
os.makedirs(os.path.dirname(dest), exist_ok=True)
os.rename(src, dest)
for path, col in ((sys.argv[2], 2), (sys.argv[3], 0)):
rows = [line.rstrip("\n").split("\t") for line in open(path)]
with open(path, "w") as fh:
for f in rows:
f[col] = moves.get(f[col], f[col])
fh.write("\t".join(f) + "\n")
PY
plan
run awk -F '\t' '$1 == "move" {print $2 " -> " $3}' "$OUT"
[ -z "$output" ] || { echo "moved again: $output"; false; }
}
+30 -1
View File
@@ -374,7 +374,10 @@ PY
# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
apply_plan() {
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
# `/` is always a mountpoint, so the overlay note behaves as it does on a
# host that runs the metadata view.
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" \
--journal "$JOURNAL" --metadata-view / --apply
}
@test "a run without --apply says what it would do and touches nothing" {
@@ -409,6 +412,32 @@ print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "
[ "$status" -eq 0 ]
}
# The overlay note is only true where the metadata view is really mounted. On a
# host whose media servers bind the array directly (dankserver today: both
# quadlets mount /mnt/entropy, /var/lib/media-meta does not exist) the sidecars
# moved with the media, and "next: run reap" is an order that cannot be carried
# out. --metadata-view is how the caller names that path; the default is the
# host's view path, and the note rides only when it is a mountpoint.
@test "an apply with no mounted metadata view does not order a reap" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"moved"* ]]
[[ "$output" != *"media-view.sh reap"* ]]
}
@test "a rollback with no mounted metadata view does not order a reap" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
run python3 "$APPLY" rollback --journal "$JOURNAL" --root "$LIB" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"rolled back"* ]]
[[ "$output" != *"media-view.sh reap"* ]]
}
@test "a target already holding a different file stops the whole run" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+415 -25
View File
@@ -8,7 +8,18 @@
# file name. So every test here states which authority a file had — name only,
# provider only, both, neither — and asserts where the PLAN files it, never how
# the planner decided. A fixture Jellyfin database stands in for the live one and
# the TMDb cache is an input file, so no test touches the network.
# the name caches are input files, so no test touches the network.
#
# The target contract is tvdb-grade naming:
# <Series> (<year>) [tvdbid-NNNN]/Season NN/<Series> (<year>) - SxxEyy - <Title>.<ext>
# The id rides the FOLDER only: it is what both matchers read the series by, and a
# 180-character ceiling on a file name is not the place to spend it. Season 00
# holds specials; a provider row whose "season" is a YEAR is renumbered by the
# episode title tvdb states for the series' id, never by a guess.
# tvdb is the IDENTITY authority: a provider name a server derived from a folder
# is not a series name, so a polluted name is undone through the series' tvdb id.
# The review pile parks OUTSIDE the library — a parked folder inside it indexes
# as one junk show, and that show's rows are then mistaken for evidence.
#
# The two outcomes that cost a re-run of a 50k-file plan are a file filed as an
# episode that is not one (it steals an episode number from its real owner) and a
@@ -25,11 +36,20 @@ APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] — one row of the authority
# export scripts/media-census.py jellyfin writes.
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] [TMDB] [TITLE] [TVDB]
# — one row of the authority export scripts/media-census.py jellyfin writes.
title_row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\t\n' "$1" "$2" "$3" "${4:-}" "${5:-}" "${6:-}" \
>> "$TITLES"
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" \
"${5:-}" "${6:-}" "${7:-}" "${8:-}" "${9:-}" >> "$TITLES"
}
# file PATH SIZE — `at` plus the row the enum census would have carried had the
# file existed when setup walked the tree. A test that needs a shape the fixture
# does not ship adds it here, or the planner never sees it.
file() {
at "$1" "$2"
local b="${1##*/}"
printf '%s\t.%s\t%s\n' "$2" "${b##*.}" "$1" >> "$ENUM"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
@@ -40,6 +60,7 @@ dst() {
setup() {
LIB="$BATS_TEST_TMPDIR/TVShows"
PARK="$LIB/_unsorted"
OUTSIDE="$BATS_TEST_TMPDIR/park"
MOVIES="$BATS_TEST_TMPDIR/Movies"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
@@ -77,8 +98,15 @@ setup() {
# A season folder whose rows are stale: the database numbers every file of this
# pack season 16 episode 20, four seasons' worth of them.
at "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" 6000
# One episode-shaped file loose at the library root.
at "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" 4000
# A box-set folder whose Series item the server named AFTER THE FOLDER: the
# row's own name is junk, and only a provider id can undo it.
BOX="$LIB/Show Box (2016) Season 1 S01 (1080p WEB-DL)"
at "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" 6000
# The park inside the library: the server indexed the whole pile as ONE show
# named after the folder, so its rows state a series that is not a show.
at "$PARK/Park Show S01E01.mkv" 5000
at "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" 5200
# The enum census lists every file; the planner reads that instead of walking
# the tree, so the fixture writes the same three columns.
@@ -92,17 +120,21 @@ with open(out, "w") as fh:
fh.write(f"{os.stat(p).st_size}\t{os.path.splitext(f)[1]}\t{p}\n")
PY
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\n' > "$TITLES"
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\ttitle\ttvdb\n' > "$TITLES"
printf '{}\n' > "$CACHE"
title_row "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" \
"The Examples" 2010 1 1
"The Examples" 2010 1 1 "" "" "Pilot"
title_row "$ANIME/[grp] Anime Pack - 05.mkv" "Anime Pack" "" 1 5
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2 "" "" "Double Trouble"
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3 "" "" "Loose Ends"
title_row "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" \
"SNL Pack" "" 16 20
title_row "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \
"Show Box (2016) Season 1" 2016 1 1 "" "2334" "Pilot" "78850"
# The row the server wrote for a parked file: the series is the PARK FOLDER.
title_row "$PARK/Park Show S01E01.mkv" "_unsorted" "" 1 1
}
plan() {
@@ -111,6 +143,28 @@ plan() {
--out "$OUT" --exceptions "$EXC"
}
plan_park() { # the review pile outside the library
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
--park "$OUTSIDE" --out "$OUT" --exceptions "$EXC"
}
@test "--tmdb with no key is reported as an unfinished plan, not a finished one" {
# A dead provider is invisible where it happens: every 401 is one line on
# stderr, and a terminal that scrolled past them shows a plan that reads as
# complete while every folder still carries the metadata provider's junk. The
# plan is still written — the answers that DID arrive are worth their rename —
# so the state has to ride in the summary line and in the review file.
# env -u keeps this offline no matter what the developer's shell exports.
run env -u TMDB_API_KEY python3 "$PLAN" --root "$LIB" --enum "$ENUM" \
--titles "$TITLES" --probed "$PROBED" --tmdb-cache "$CACHE" \
--movies "$MOVIES" --tmdb --out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ]
grep -q 'tmdb NOT ASKED (TMDB_API_KEY unset): NOT provider-complete' <<<"$output"
grep -q 'WARNING: tmdb NOT ASKED (TMDB_API_KEY unset) — NOT provider-complete' "$EXC"
grep -qE '^plan: [1-9][0-9]* moves' <<<"$output"
}
@test "the census exports the provider's episode numbering as a TSV" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" "$SEASON" <<'PY'
@@ -125,7 +179,7 @@ sid = str(uuid.uuid4())
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
pre, None, None, 2010, None, 0,
'{"ProviderIds":{"Tmdb":"1234"}}'))
'{"ProviderIds":{"Tmdb":"1234","Tvdb":"78850"}}'))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), sid, "MediaBrowser.Controller.Entities.TV.Episode",
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
@@ -136,34 +190,115 @@ PY
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file series year season episode extra tmdb" ]
[ "${lines[0]}" = "file series year season episode extra tmdb title tvdb" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234$"
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234 Pilot 78850$"
}
@test "an episode its own name states files as Show SxxEyy under Season NN" {
@test "the census reads provider ids from the table newer Jellyfin keeps" {
DB="$BATS_TEST_TMPDIR/jellyfin12.db"
python3 - "$DB" "$SEASON" <<'PY'
import os, sqlite3, sys, uuid
db, season = sys.argv[1], sys.argv[2]
c = sqlite3.connect(db)
pre = "/data/TVShows/" + os.path.basename(season)
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, ExtraType,
IsVirtualItem, Data)""")
# Jellyfin 12 moved the ids out of the Data blob and into their own table.
c.execute("create table BaseItemProviders (ItemId, ProviderId, ProviderValue)")
sid = str(uuid.uuid4())
eid = str(uuid.uuid4())
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
pre, None, None, 2010, None, 0, '{}'))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(eid, sid, "MediaBrowser.Controller.Entities.TV.Episode",
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
None, None, 0, '{}'))
c.execute("insert into BaseItemProviders values (?,?,?)", (sid, "Tvdb", "9999"))
c.commit()
PY
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp12.tsv"
[ "$status" -eq 0 ]
tail -1 "$BATS_TEST_TMPDIR/exp12.tsv" \
| grep -q " Pilot 9999$"
}
@test "an episode its own name states files as the titled standard name" {
plan
[ "$status" -eq 0 ]
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv")" \
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.mkv" ]
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.mkv" ]
}
@test "the tvdb identity names the series folder, id and all, not the folder's echo" {
printf '{"tvdb:series:78850": {"name": "Show Box", "year": "2016"}}\n' > "$CACHE"
plan
want="$LIB/Show Box (2016) [tvdbid-78850]/Season 01"
want+="/Show Box (2016) - S01E01 - Pilot.mkv"
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" = "$want" ]
}
@test "a provider name that already carries the year is not given it twice" {
# TVDb answers the reboot as `Acapulco (2021)`. The folder contract is ONE year,
# and `Acapulco (2021) (2021)` is a folder neither matcher nor human reuses.
printf '{"tvdb:series:78850": {"name": "Show Box (2016)", "year": "2016"}}\n' > "$CACHE"
plan
want="$LIB/Show Box (2016) [tvdbid-78850]/Season 01"
want+="/Show Box (2016) - S01E01 - Pilot.mkv"
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" = "$want" ]
run grep -F '(2016) (2016)' "$OUT"
[ -z "$output" ]
}
@test "a series no authority resolves keeps the provider's name and is reported" {
plan
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
grep -A9 -i "no tvdb" "$EXC" | grep -q "SNL Pack"
}
@test "a row whose series is the park folder is not evidence of a show" {
plan_park
run awk -F '\t' -v p="$PARK/" '$1 == "move" && index($3, p) == 1 {print $3}' "$OUT"
[ -z "$output" ]
run awk -F '\t' '$1 == "move" && $2 ~ /Park Show S01E01/ {print $3}' "$OUT"
[[ "$output" == "$OUTSIDE/"* ]]
}
@test "the pile's sanitized display name is still the pile, not a series" {
# Jellyfin drops the leading underscore when it shows the hidden folder, so the
# row says the series is "unsorted" while the file sits in `_unsorted`. Match
# only the underscored spelling and the plan renames real files into a visible
# folder named after the review pile — which is how 617 of them got there.
title_row "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" "unsorted" "" 1 1 \
"" "" "Twisted Metal"
plan_park
run awk -F '\t' -v l="$LIB" \
'$1 == "move" && index($3, l "/unsorted/") == 1 {print $3}' "$OUT"
[ -z "$output" ]
run awk -F '\t' '$1 == "move" && $2 ~ /Twisted\.Metal\.S01E01/ {print $3}' "$OUT"
[[ "$output" == "$OUTSIDE/"* ]]
}
@test "a subtitle follows its episode and keeps its language flag" {
plan
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.en.srt" ]
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.en.srt" ]
}
@test "a subtitle two folders down inside Subs/<release-name>/ still follows its episode" {
plan
[ "$(dst "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.jpn.srt" ]
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.jpn.srt" ]
}
@test "a subtitle that carries the container's word in its own name still follows the episode" {
plan
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.forced.srt" ]
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.forced.srt" ]
}
@test "a season image keeps its own name; a series nfo keeps the series folder" {
@@ -178,10 +313,10 @@ PY
[ -z "$(awk -F '\t' '$3 ~ /Season .*teaser/ {print $3}' "$OUT")" ]
}
@test "a file only the provider numbers files at the episode it is matched to" {
@test "a file only the provider numbers files at the episode it is matched to, title or no title" {
plan
[ "$(dst "$ANIME/[grp] Anime Pack - 05.mkv")" \
= "$LIB/Anime Pack/Season 01/Anime Pack S01E05.mkv" ]
= "$LIB/Anime Pack/Season 01/Anime Pack - S01E05.mkv" ]
}
@test "a bonus folder inside a release pack belongs to the pack's show, not to its own name" {
@@ -193,19 +328,19 @@ PY
@test "a Season folder outvotes the stale row its files carry" {
plan
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
= "$LIB/SNL Pack/Season 19/SNL Pack S19E01.avi" ]
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
}
@test "one file holding two episodes keeps both numbers in its name" {
plan
[ "$(dst "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv")" \
= "$LIB/Show Two (2016)/Season 01/Show Two S01E02-E03.mkv" ]
= "$LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E02-E03 - Double Trouble.mkv" ]
}
@test "two files claiming one episode: one keeps the name, the other parks" {
plan
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two S01E01.mkv " "$OUT")
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E01 - Two Pilots.mkv " "$OUT")
[ "$n" -eq 1 ]
run dst "$LIB/Dupes/Show Two S01E01.mkv"
[[ "$output" == "$PARK/"* ]]
@@ -240,6 +375,38 @@ json.dump({"movie:some documentary":
[[ "$output" == "$PARK/"* ]]
}
@test "a movie answer the file does not itself name stays in the review pile" {
# TMDb's top hit for a truncated query is a NEAR miss, not an answer: "mobile
# suit gundam 00" comes back as Special Edition I while the file is Special
# Edition III, and a one-word query "russia" comes back as a 1963 Bond film. A
# file leaves the TV library only when it states the title the provider names.
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
python3 -c 'import json, sys
json.dump({"movie:some documentary":
{"name": "Some Documentary Special Part Two", "year": "2015", "id": 55}},
open(sys.argv[1], "w"))' "$CACHE"
plan
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
[[ "$output" == "$PARK/"* ]]
run awk -F '\t' -v m="$MOVIES/" '$1 == "move" && index($3, m) == 1 {print $3}' "$OUT"
[ -z "$output" ]
}
@test "a movie answer from another year is not that film" {
# The year the file states outranks a provider's eagerness: the 2015 special is
# not the 1999 one, however alike their names are.
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
python3 -c 'import json, sys
json.dump({"movie:some documentary":
{"name": "Some Documentary Special", "year": "1999", "id": 55}},
open(sys.argv[1], "w"))' "$CACHE"
plan
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
[[ "$output" == "$PARK/"* ]]
}
@test "the plan creates every folder before the move that needs it" {
plan
python3 - "$OUT" "$LIB" "$MOVIES" "$PARK" <<'PY'
@@ -305,3 +472,226 @@ PY
plan
[ -z "$(awk -F '\t' '$1 == "move" {print $2}' "$OUT")" ]
}
# Specials and the year-shaped season. Jellyfin's contract puts specials in
# Season 00, and its date matcher hands back rows whose "season" is the AIRDATE
# YEAR. Both used to park — 1,412 and 1,033 files of one 52k plan — although one
# is a legal season and the other is renumberable from the title tvdb states.
@test "a provider's season 0 is a special, so it files under Season 00" {
file "$LIB/Show Bumps/Best Of The Examples.mkv" 5000
title_row "$LIB/Show Bumps/Best Of The Examples.mkv" "The Examples" 2010 0 2
plan
[ "$(dst "$LIB/Show Bumps/Best Of The Examples.mkv")" \
= "$LIB/The Examples (2010)/Season 00/The Examples (2010) - S00E02.mkv" ]
}
# These three tests carry a tvdb id of their own. The fixture's box set is 78850 and
# ONE id means ONE show: reusing it here would merge these files into that series and
# the duplicate rule would park them before any numbering question is asked.
@test "a row whose season is a year is renumbered by the title tvdb states" {
printf '{"tvdb:series:4242": %s, "tvdb:episodes:4242": %s}\n' \
'{"name": "The Examples", "year": "2010"}' \
'{"1/7": "Best Of"}' > "$CACHE"
file "$LIB/Show Bumps/Best Of The Examples.mkv" 5000
title_row "$LIB/Show Bumps/Best Of The Examples.mkv" \
"The Examples" 2010 2010 11 "" "" "Best Of" "4242"
plan
want="$LIB/The Examples (2010) [tvdbid-4242]/Season 01"
want+="/The Examples (2010) - S01E07 - Best Of.mkv"
[ "$(dst "$LIB/Show Bumps/Best Of The Examples.mkv")" = "$want" ]
}
@test "a title tvdb gives twice never renumbers an episode" {
printf '{"tvdb:series:4242": %s, "tvdb:episodes:4242": %s}\n' \
'{"name": "The Examples", "year": "2010"}' \
'{"1/7": "Best Of", "2/3": "Best Of"}' > "$CACHE"
file "$LIB/Show Bumps/Best Of The Examples.mkv" 5000
title_row "$LIB/Show Bumps/Best Of The Examples.mkv" \
"The Examples" 2010 2010 11 "" "" "Best Of" "4242"
plan_park
run awk -F '\t' -v p="$OUTSIDE/" '$1 == "move" && index($3, p) == 1 {print $3}' "$OUT"
[[ "$output" == *"Best Of The Examples.mkv"* ]]
}
# Anime numbering is the one place a tvdb title can be confidently wrong: a pack
# numbered by its release group still names S05E01, and tvdb answers that slot with
# a different episode. A season the authority does not hold, or holds fewer of than
# the pack books, is CONTRADICTED: the rename still happens, the title does not.
@test "a season tvdb contradicts keeps the rename but never its titles" {
printf '{"tvdb:series:4242": %s, "tvdb:episodes:4242": %s}\n' \
'{"name": "Anime Gate", "year": "2010"}' \
'{"1/1": "Pilot"}' > "$CACHE"
file "$LIB/Gate Pack/Season 01/one S01E01.mkv" 5000
file "$LIB/Gate Pack/Season 05/two S05E01.mkv" 5000
file "$LIB/Gate Pack/Season 05/three S05E02.mkv" 5000
title_row "$LIB/Gate Pack/Season 01/one S01E01.mkv" \
"Anime Gate" 2010 1 1 "" "" "" "4242"
title_row "$LIB/Gate Pack/Season 05/two S05E01.mkv" \
"Anime Gate" 2010 5 1 "" "" "" "4242"
title_row "$LIB/Gate Pack/Season 05/three S05E02.mkv" \
"Anime Gate" 2010 5 2 "" "" "" "4242"
plan
want="$LIB/Anime Gate (2010) [tvdbid-4242]/Season 01"
want+="/Anime Gate (2010) - S01E01 - Pilot.mkv"
[ "$(dst "$LIB/Gate Pack/Season 01/one S01E01.mkv")" = "$want" ]
want="$LIB/Anime Gate (2010) [tvdbid-4242]/Season 05"
want+="/Anime Gate (2010) - S05E01.mkv"
[ "$(dst "$LIB/Gate Pack/Season 05/two S05E01.mkv")" = "$want" ]
grep -A6 -i "numbering tvdb contradicts" "$EXC" | grep -q "Anime Gate"
}
# Tvdb's default episode query answers at most 500 rows and states no total, so a
# list that long is a PAGE, not a list: One Piece (1999), Crayon Shin-chan (1992)
# and Detective Conan (1996) each answer exactly 500 while holding over a thousand.
# Read as the whole truth, that page contradicts every long-running show's pack
# and strips the tvdb title from all of it.
@test "an episode list the authority paginates never contradicts the pack" {
python3 - "$CACHE" <<'PY'
import json, sys
json.dump({"tvdb:series:4242": {"name": "Long Run", "year": "1999"},
"tvdb:episodes:4242": {f"1/{n}": f"Episode {n}"
for n in range(1, 501)}},
open(sys.argv[1], "w"))
PY
for n in $(seq 1 501); do
ep=$(printf '%02d' "$n")
file "$LIB/Long Pack/Season 01/long $ep.mkv" 1000
title_row "$LIB/Long Pack/Season 01/long $ep.mkv" \
"Long Run" 1999 1 "$n" "" "" "" "4242"
done
plan
want="$LIB/Long Run (1999) [tvdbid-4242]/Season 01"
want+="/Long Run (1999) - S01E01 - Episode 1.mkv"
[ "$(dst "$LIB/Long Pack/Season 01/long 01.mkv")" = "$want" ]
run grep -i "numbering tvdb contradicts" "$EXC"
[ "$status" -ne 0 ]
}
# ------------------------------------------------------------- the operator map
# Most of this library carries no tvdb id anywhere: the authority titles anime by
# its own record — in the show's language — and the English name rides in aliases
# the planner's name search never reads, so a search that demands tvdb's own
# spelling cannot reach it. The operator's map is the other door: one line per
# folder, `folder <TAB> tvdb id <TAB> note`, each id checked by hand against the
# record it names. It outranks a provider id, it merges the release folders that
# are one show, and every folder it touched is named in the exceptions: one wrong
# row there is 1,268 renames to the wrong show, so nothing about it is silent.
plan_map() { # the plan, with the operator's map in hand
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
--tvdb-ids "$1" --out "$OUT" --exceptions "$EXC"
}
@test "a folder the operator mapped to a tvdb id renames to that record" {
printf '{"tvdb:series:331753": %s}\n' \
'{"name": "Black Clover", "year": "2017"}' > "$CACHE"
{
printf '# the note column carries why the name search cannot reach it\n'
printf 'Black Clover 001\t331753\tthe record holds no alias\n'
} > "$BATS_TEST_TMPDIR/ids.tsv"
file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000
title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \
"Black Clover 001" "" 1 1
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
want="$LIB/Black Clover (2017) [tvdbid-331753]/Season 01"
want+="/Black Clover (2017) - S01E01.mkv"
[ "$(dst "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv")" = "$want" ]
}
@test "release folders one id maps are ONE series folder" {
printf '{"tvdb:series:81797": %s}\n' \
'{"name": "One Piece", "year": "1999"}' > "$CACHE"
{
printf '[Anime Time] One Piece (0001-1071)\t81797\tbare "One Piece" is the 2023 live action\n'
printf 'One Piece Ship Collection\t81797\tthe same record, split by release\n'
} > "$BATS_TEST_TMPDIR/ids.tsv"
file "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv" 5000
file "$LIB/One Piece Ship Collection/One.Piece.0002.mkv" 6000
title_row "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv" \
"One Piece" 1999 1 1
title_row "$LIB/One Piece Ship Collection/One.Piece.0002.mkv" \
"One Piece Ship Collection" "" 1 2
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
want="$LIB/One Piece (1999) [tvdbid-81797]/Season 01"
want+="/One Piece (1999) - S01E01.mkv"
[ "$(dst "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv")" \
= "$want" ]
want="$LIB/One Piece (1999) [tvdbid-81797]/Season 01"
want+="/One Piece (1999) - S01E02.mkv"
[ "$(dst "$LIB/One Piece Ship Collection/One.Piece.0002.mkv")" = "$want" ]
}
@test "the map outranks a provider id and says which two disagreed" {
printf '{"tvdb:series:4242": %s, "tvdb:series:78850": %s}\n' \
'{"name": "Real Ship", "year": "1999"}' \
'{"name": "Box Junk", "year": "2016"}' > "$CACHE"
printf '%s\t4242\thand-verified against the record\n' \
"Show Box (2016) Season 1 S01 (1080p WEB-DL)" > "$BATS_TEST_TMPDIR/ids.tsv"
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \
| grep -q "^$LIB/Real Ship (1999) \[tvdbid-4242\]/Season 01/"
run grep -E '78850.*4242|4242.*78850' "$EXC"
[ "$status" -eq 0 ]
}
@test "a map id the authority never answered keeps the provider's name and is named" {
printf 'Black Clover 001\t331753\ttvdb never answered\n' \
> "$BATS_TEST_TMPDIR/ids.tsv"
file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000
title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \
"Black Clover 001" "" 1 1
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
dst "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \
| grep -q "^$LIB/Black Clover 001 \[tvdbid-331753\]/Season 01/"
run grep -q "331753" "$EXC"
[ "$status" -eq 0 ]
}
@test "a map row for a folder the array no longer holds is reported" {
printf 'Gone Pack\t4242\tmoved away\n' > "$BATS_TEST_TMPDIR/ids.tsv"
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
run grep "Gone Pack" "$EXC"
[ "$status" -eq 0 ]
}
@test "a map line whose id is not a number stops the plan" {
printf 'Black Clover 001\tnotanid\ttypo\n' > "$BATS_TEST_TMPDIR/ids.tsv"
file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000
title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \
"Black Clover 001" "" 1 1
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
[ "$status" -ne 0 ]
[[ "$output" == *"notanid"* ]]
}
# The map the fleet ships is scripts/media-shows-tvdb-ids.tsv, and the planner
# reads it by default: an unusable line there stops every plan the operator runs
# from the checkout, so the seam validates the real file, not a fixture copy.
@test "the shipped operator map parses and never stops a plan" {
MAPS="$REPO_ROOT/scripts/media-shows-tvdb-ids.tsv"
run awk -F '\t' '!/^#/ && NF {
if (NF < 2 || $2 !~ /^[0-9]+$/) { print "unusable row: " $0; bad = 1 }
} END { exit bad }' "$MAPS"
[ "$status" -eq 0 ]
dup=$(awk -F '\t' '!/^#/ && NF {print $1}' "$MAPS" | sort | uniq -d)
[ -z "$dup" ]
plan_map "$MAPS"
[ "$status" -eq 0 ]
grep -qi "operator's map" "$EXC"
}
@test "a map that names one folder twice stops the plan" {
printf '{"tvdb:series:78850": %s}\n' \
'{"name": "Naruto", "year": "2002"}' > "$CACHE"
{
printf 'Naruto Complete Series\t78850\tthe dub pack\n'
printf 'Naruto Complete Series\t331753\ta paste of another row\n'
} > "$BATS_TEST_TMPDIR/ids.tsv"
file "$LIB/Naruto Complete Series/Season 01/naruto 01.mkv" 5000
title_row "$LIB/Naruto Complete Series/Season 01/naruto 01.mkv" \
"Naruto Complete Series" "" 1 1
plan_map "$BATS_TEST_TMPDIR/ids.tsv"
[ "$status" -eq 2 ]
[[ "$output" == *"named twice"* ]]
}
+282
View File
@@ -0,0 +1,282 @@
#!/usr/bin/env bats
# Seam: scripts/media-verify.py — the replay of a relayout journal against a
# media server's OWN database, which is the documented proof that a rename
# landed (a scan answers 204 whether or not it finds anything).
#
# Every test states what the database holds and asserts what the command prints
# and returns; nothing here reaches into the script's internals. The fixtures are
# minimal sqlite files with the columns the two servers really use, so the seam
# fails if the tool ever reads the wrong table, forgets Plex's `deleted_at`
# filter, or forgets that Jellyfin stores the container-side path (/data/…) while
# Plex stores the host path verbatim.
load helpers
VERIFY="$REPO_ROOT/scripts/media-verify.py"
# jf_mk <db> — the Jellyfin shape: one BaseItems table, Path is the column.
jf_mk() {
python3 - "$1" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("create table BaseItems (Id INTEGER PRIMARY KEY, Path TEXT, Type TEXT,"
" ParentId INTEGER, Name TEXT)")
con.commit()
PY
}
# jf_row <db> <path> — one item the server believes it has.
jf_row() {
python3 - "$1" "$2" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("insert into BaseItems (Path, Type) values (?, 'x')", (sys.argv[2],))
con.commit()
PY
}
# px_mk <db> — the Plex shape: media_parts joined up to metadata_items.
px_mk() {
python3 - "$1" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("create table metadata_items (id INTEGER PRIMARY KEY, title TEXT,"
" deleted_at TEXT)")
con.execute("create table media_items (id INTEGER PRIMARY KEY, metadata_item_id"
" INTEGER, container TEXT)")
con.execute("create table media_parts (id INTEGER PRIMARY KEY, media_item_id"
" INTEGER, file TEXT, deleted_at TEXT)")
con.commit()
PY
}
# px_row <db> <path> [DELETED_AT] — one part; a non-empty DELETED_AT is a row a
# deleted scan left behind, which must never count as proof.
px_row() {
python3 - "$1" "$2" "${3:-}" <<'PY'
import sqlite3, sys
db, path, gone = sys.argv[1], sys.argv[2], sys.argv[3] or None
con = sqlite3.connect(db)
cur = con.execute("insert into metadata_items (title, deleted_at) values ('t', ?)",
(gone,))
mi = cur.lastrowid
cur = con.execute("insert into media_items (metadata_item_id, container)"
" values (?, 'mkv')", (mi,))
con.execute("insert into media_parts (media_item_id, file, deleted_at)"
" values (?, ?, NULL)", (cur.lastrowid, path))
con.commit()
PY
}
# move_row <src> <dst> — one move in the journal.
move_row() {
printf '{"op": "move", "src": "%s", "dst": "%s"}\n' "$1" "$2" >> "$JOURNAL"
}
setup() {
SHARE="$BATS_TEST_TMPDIR/mnt/entropy"
JFDB="$BATS_TEST_TMPDIR/jellyfin.db"
PXDB="$BATS_TEST_TMPDIR/plex.db"
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
: > "$JOURNAL"
jf_mk "$JFDB"
px_mk "$PXDB"
}
@test "every move that the database holds at its destination is reported landed" {
move_row "$SHARE/Movies/Old Name (1999).mkv" "$SHARE/Movies/Old Name (1999)/Old Name (1999).mkv"
move_row "$SHARE/Movies/Two.mkv" "$SHARE/Movies/Two/Two.mkv"
# Jellyfin stores the container-side path: /data stands for the share.
jf_row "$JFDB" "/data/Movies/Old Name (1999)/Old Name (1999).mkv"
jf_row "$JFDB" "/data/Movies/Two/Two.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[0]}" = "jellyfin: 2 moves checked" ]
[ "${lines[1]}" = "landed: 2" ]
[ "${lines[2]}" = "still at src: 0" ]
[ "${lines[3]}" = "missing from the DB: 0" ]
}
@test "a database that has not rescanned yet is refused, naming the stale rows" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
jf_row "$JFDB" "/data/Movies/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[1]}" = "landed: 0" ]
[ "${lines[2]}" = "still at src: 1" ]
[[ "$output" == *"$SHARE/Movies/Old.mkv"* ]]
}
@test "a moved file no row of the database names is reported missing" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 1" ]
[[ "$output" == *"$SHARE/Movies/Old/Old.mkv"* ]]
}
@test "Plex stores the host path verbatim, so a row at the destination lands" {
move_row "$SHARE/TVShows/Show/Season 01/Show S01E01.mkv" \
"$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
px_row "$PXDB" "$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
--share "$SHARE"
[ "$status" -eq 0 ]
[ "${lines[0]}" = "plex: 1 moves checked" ]
[ "${lines[1]}" = "landed: 1" ]
}
@test "a Plex row a deleted scan left behind is not proof a move landed" {
move_row "$SHARE/TVShows/Old S01E01.mkv" "$SHARE/TVShows/Show/Old S01E01.mkv"
px_row "$PXDB" "$SHARE/TVShows/Show/Old S01E01.mkv" "2026-10-30 00:00:00"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
--share "$SHARE"
[ "$status" -eq 1 ]
[ "${lines[1]}" = "landed: 0" ]
[ "${lines[3]}" = "missing from the DB: 1" ]
}
@test "a database without the columns that kind needs is an error, not a zero" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$PXDB" \
--share "$SHARE" --container /data
[ "$status" -eq 2 ]
[[ "$output" == *"$PXDB"* ]]
[[ "$output" == *"BaseItems"* ]]
}
@test "only move rows are counted, so folder operations cannot inflate the tally" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
jf_row "$JFDB" "/data/Movies/Old/Old.mkv"
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
printf '{"op": "rmdir", "path": "%s/Movies/Empty"}\n' "$SHARE" >> "$JOURNAL"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[0]}" = "jellyfin: 1 moves checked" ]
}
@test "a journal with no moves is clean, not a failure" {
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[1]}" = "landed: 0" ]
}
@test "the offender list stops at --show so a stale library stays readable" {
for n in 1 2 3 4; do
move_row "$SHARE/Movies/F$n.mkv" "$SHARE/Movies/F$n/F$n.mkv"
done
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data --show 2
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 4" ]
[[ "$output" == *"$SHARE/Movies/F1/F1.mkv"* ]]
[[ "$output" != *"$SHARE/Movies/F4/F4.mkv"* ]]
}
@test "the missing rows are counted by extension, so a sidecar is not mistaken for a lost film" {
move_row "$SHARE/Music/Album/Mandatory Fun.log" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.log"
move_row "$SHARE/Music/Album/Film.mkv" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Film.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 2" ]
[[ "$output" == *"missing by extension: .log 1, .mkv 1"* ]]
}
@test "--ignore-ext keeps a file the server never indexes out of the verdict" {
move_row "$SHARE/Music/Album/Song.flac" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
jf_row "$JFDB" "/data/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
move_row "$SHARE/Music/Album/Mandatory Fun.cue" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.cue"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data --ignore-ext log,cue
[ "$status" -eq 0 ]
[ "${lines[1]}" = "landed: 1" ]
[ "${lines[3]}" = "missing from the DB: 0" ]
[[ "$output" == *"not indexed (extension ignored): 1"* ]]
}
# wal_case <db> — a Jellyfin database whose NEW row lives only in the
# write-ahead log: the server has committed the rename but not checkpointed it,
# which is the normal state of a library a scan is writing. The holder
# connection keeps the WAL from folding into the main file, and the verifier
# runs as a real subprocess so it faces the same files Plex and Jellyfin leave.
wal_case() {
python3 - "$JFDB" "$JOURNAL" "$SHARE" "$VERIFY" <<'PY'
import sqlite3, subprocess, sys
db, journal, share, verify = sys.argv[1:5]
w = sqlite3.connect(db)
w.execute("pragma journal_mode=wal")
w.execute("pragma wal_autocheckpoint=0")
w.execute("insert into BaseItems (Path, Type) values ('/data/Movies/Old.mkv', 'x')")
w.commit()
w.execute("pragma wal_checkpoint(truncate)")
holder = sqlite3.connect(db) # last-connection close would checkpoint
w.execute("update BaseItems set Path='/data/Movies/Old/Old.mkv'")
w.commit()
r = subprocess.run([sys.executable, verify, "--journal", journal,
"--db-kind", "jellyfin", "--db", db,
"--share", share, "--container", "/data"],
capture_output=True, text=True)
print(r.stdout, end="")
sys.exit(r.returncode)
PY
}
@test "a row the server committed but has not checkpointed is proof, not a miss" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run wal_case
[ "$status" -eq 0 ]
[ "${lines[1]}" = "landed: 1" ]
[ "${lines[3]}" = "missing from the DB: 0" ]
}
# The DB alone cannot tell a lost file from a sidecar: both are "no row at the
# destination". The tool therefore stats the two paths and separates the classes
# the runbook names, so an operator never reads a `.pdf` as a lost film or,
# worse, reads a real loss as a sidecar.
@test "a file the database does not name that still sits at src is the pass's own damage" {
mkdir -p "$SHARE/Movies"
echo film > "$SHARE/Movies/Old.mkv"
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 1" ]
[ "${lines[4]}" = " on disk at dst, no row names it: 0" ]
[ "${lines[5]}" = " REAL — file still at src: 1" ]
[ "${lines[6]}" = " gone — at neither path: 0" ]
[[ "$output" == *"real: $SHARE/Movies/Old.mkv"* ]]
}
@test "a file at neither path is gone, and a file at dst the server ignores is not a loss" {
move_row "$SHARE/Movies/Gone.mkv" "$SHARE/Movies/Gone/Gone.mkv"
mkdir -p "$SHARE/Movies/Sidecar"
echo cover > "$SHARE/Movies/Sidecar/cover.jpg"
move_row "$SHARE/Movies/Album/cover.jpg" "$SHARE/Movies/Sidecar/cover.jpg"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 2" ]
[ "${lines[4]}" = " on disk at dst, no row names it: 1" ]
[ "${lines[5]}" = " REAL — file still at src: 0" ]
[ "${lines[6]}" = " gone — at neither path: 1" ]
[[ "$output" == *"gone: $SHARE/Movies/Gone/Gone.mkv"* ]]
}