feat(media): measure a series per episode, and let the tag outrank a borrowed id

media-tvdupes.py censuses a TV library by episode key and never by folder. 10 of
the 25 shows that looked duplicated in the 2026-11-02 census held complementary
halves (Game of Thrones Complete carries no S01 or S08, Modern Family S05 is the
only copy of season 5), so a folder-level verdict would have staged some 340 GB
of unique episodes to recover 54 GiB of true copies.

Show identity comes from the folder's [tvdbid-N] tag, from the tvshow.nfo the
metadata view exposes, or from an operator --ids map. A derived id that
contradicts a tag is dropped and reported, never unioned: Jellyfin's provider
table carries a 2026 special's id on the folder tagged [tvdbid-73838], and
unioning the two makes one show of a 2000 series and a 2026 special whose bigger
files then dominate the original's pilot. Files named by an id key only when the
server's own episode table (the --witness) names them, so a bare number is never
mistaken for an episode.

Seam: tests/media_tvdupes.bats — 12 assertions on which files land in the plan
and which the tool refuses to touch, on a synthetic library.
This commit is contained in:
Frank committed 2026-11-02 00:44:46 -08:00
1 parent a25f139281
commit 9f9043c037
5 files changed
+790 -5

No files matched your search

@@ -1,9 +1,9 @@
---
type: reference
title: Media Duplicate Prune
description: "Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge."
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
tags: [dankserver, media-server, storage, procedure]
timestamp: 2026-11-01T01:00:00Z
timestamp: 2026-11-02T01:00:00Z
---
# Media Duplicate Prune
@@ -13,6 +13,10 @@ downloaded again, or the same film grabbed a second time at another quality —
[[Entropy Volume]] is permanently full, so both are worth finding. They need
different rules, and neither is decided by name similarity alone.
A series duplicates differently again: one show spread over several folders, so
neither the file nor the folder is the unit — see *A series is measured per
episode*.
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
@@ -24,6 +28,7 @@ and films nothing served. It is gone; its contents went to `NeedsProcessing`.
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |
## The screen
@@ -51,6 +56,84 @@ folder; then the largest. For the same-work class the largest is the answer, and
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
outlive `Film.mkv` for no reason but a space sorting before a dot.
## A series is measured per episode
`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
executor's own format; it never moves anything itself.
```bash
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
--hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
--meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
--ids /mnt/entropy/tv-tvdb-ids.tsv \
--out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
```
- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
`BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
stripped to the bare top-level folder name — 616 folders, none carrying two
ids, written to `/mnt/entropy/tv-tvdb-ids.tsv`. Two folders with disjoint ids
are two shows however alike they read — *Avatar: The Last Airbender* (1999)
beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
had also titled *Millennium* — and a pair whose identity cannot be proven is
reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
fill an untagged folder and are **dropped** when they contradict its own
`[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
[tvdbid-73838]`; unioned, those folders are one show and the special's bigger
files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
staged file rests on a borrowed id.
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
the pass would have staged about 340 GB of unique episodes; judged per episode it
staged 54 GiB of true copies.
- **A file is droppable only when every episode it carries is held elsewhere in the
same show at >= its size** (Frank's rule, applied per episode). A byte-identical
pair ties and the tie breaks by path so exactly one copy stages; a file holding
one unique episode stays whatever else it carries.
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
poster or a whole season, so key-less names are never staged and are counted in
the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
Where a packager named files by id, Jellyfin's own table is the witness: a
`path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
`pragma table_info` before trusting the query, because an empty witness silently
falls back to name parsing.
Seven clips in the library are still named only by number, and where Jellyfin
was asked its own `Name` for them is the number too, so no database holds a
better title — that residue is naming, not a task.
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
when both halves are held elsewhere, and it wins the survivor seat for both when
it is the largest carrier: the byte law compares files, and a name both servers
parse as a range stays indexed as two episodes.
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
copies were bare rips, and counting a hold's non-media files is the census that
tells you which case you are in.
- **The audit gates the plan file.** Every survivor is re-statted and every
destination re-checked before `--out` is written; on failure the report prints
`AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
- **Two folders that look duplicated and share zero episode keys are complementary,
not duplicated.** That is how the two pairs the census still refuses to merge were
settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
episode keys: 0` in both cases, so a tag would have freed nothing and cost a
rename with library consequences. Re-running the whole census with the id map over
all 753 top-level folders (606 already tagged) staged **0 files**: the library is
duplicate-free to the limit of provable identity, and what remains is naming.
## Guards
Each of these is a file that would have been destroyed without it:
@@ -97,6 +180,14 @@ keeper is missing — either one drops the whole group rather than half-applying
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
- The 2026-11-02 TV pass holds 220 files (54 GiB) in
`/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
same day's TVShows detritus pass holds 109 files (552 KiB) in
`/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
`tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges
pending Frank's word, beside the 2026-11-01 hold above. All four journals and
plans live at the `entropy` root, not `/tmp`.
- **Re-verify each survivor by size, not by the recorded path, before purging.**
Detection artifacts and plans key on paths, and a folder renamed by an
interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look
@@ -117,7 +208,9 @@ stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack o
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
held path.
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
@@ -128,6 +221,10 @@ subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
extension the census does not recognise is a reason to stop, not to sweep.
A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
file is a manual match — a human decision about what the folder is — and detritus
rules do not overrule one.
`scripts/media-verify.py` reads differently for a prune than for a relayout:
`landed: 0` is the **correct** end state (no server may index the hold), so the
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
@@ -138,7 +138,7 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
no `Accept` and client-identifier header (see the scan block above for the form
that works).
A duplicate prune is the same census and the same proof, with its own keep rules: [[Media Duplicate Prune]].
A duplicate prune is the same census and the same proof, with its own keep rules — films by title and duration, a series per episode: [[Media Duplicate Prune]].
## TV in the movie library