feat(media): measure a series per episode, and let the tag outrank a borrowed id
media-tvdupes.py censuses a TV library by episode key and never by folder. 10 of the 25 shows that looked duplicated in the 2026-11-02 census held complementary halves (Game of Thrones Complete carries no S01 or S08, Modern Family S05 is the only copy of season 5), so a folder-level verdict would have staged some 340 GB of unique episodes to recover 54 GiB of true copies. Show identity comes from the folder's [tvdbid-N] tag, from the tvshow.nfo the metadata view exposes, or from an operator --ids map. A derived id that contradicts a tag is dropped and reported, never unioned: Jellyfin's provider table carries a 2026 special's id on the folder tagged [tvdbid-73838], and unioning the two makes one show of a 2000 series and a 2026 special whose bigger files then dominate the original's pilot. Files named by an id key only when the server's own episode table (the --witness) names them, so a bare number is never mistaken for an episode. Seam: tests/media_tvdupes.bats — 12 assertions on which files land in the plan and which the tool refuses to touch, on a synthetic library.
This commit is contained in:
1 parent
a25f139281
commit
9f9043c037
5 files changed
+790
-5
No files matched your search
@@ -1,9 +1,9 @@
|
||||
---
|
||||
type: reference
|
||||
title: Media Duplicate Prune
|
||||
description: "Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge."
|
||||
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
|
||||
tags: [dankserver, media-server, storage, procedure]
|
||||
timestamp: 2026-11-01T01:00:00Z
|
||||
timestamp: 2026-11-02T01:00:00Z
|
||||
---
|
||||
|
||||
# Media Duplicate Prune
|
||||
@@ -13,6 +13,10 @@ downloaded again, or the same film grabbed a second time at another quality —
|
||||
[[Entropy Volume]] is permanently full, so both are worth finding. They need
|
||||
different rules, and neither is decided by name similarity alone.
|
||||
|
||||
A series duplicates differently again: one show spread over several folders, so
|
||||
neither the file nor the folder is the unit — see *A series is measured per
|
||||
episode*.
|
||||
|
||||
`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
|
||||
and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||
|
||||
@@ -24,6 +28,7 @@ and films nothing served. It is gone; its contents went to `NeedsProcessing`.
|
||||
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
|
||||
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
|
||||
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
|
||||
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |
|
||||
|
||||
## The screen
|
||||
|
||||
@@ -51,6 +56,84 @@ folder; then the largest. For the same-work class the largest is the answer, and
|
||||
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
|
||||
outlive `Film.mkv` for no reason but a space sorting before a dot.
|
||||
|
||||
## A series is measured per episode
|
||||
|
||||
`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
|
||||
executor's own format; it never moves anything itself.
|
||||
|
||||
```bash
|
||||
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
|
||||
--hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
|
||||
--meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
|
||||
--ids /mnt/entropy/tv-tvdb-ids.tsv \
|
||||
--out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json
|
||||
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
|
||||
```
|
||||
|
||||
- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
|
||||
name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
|
||||
and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
|
||||
match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
|
||||
`BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
|
||||
stripped to the bare top-level folder name — 616 folders, none carrying two
|
||||
ids, written to `/mnt/entropy/tv-tvdb-ids.tsv`. Two folders with disjoint ids
|
||||
are two shows however alike they read — *Avatar: The Last Airbender* (1999)
|
||||
beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
|
||||
had also titled *Millennium* — and a pair whose identity cannot be proven is
|
||||
reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
|
||||
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
|
||||
fill an untagged folder and are **dropped** when they contradict its own
|
||||
`[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
|
||||
provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
|
||||
[tvdbid-73838]`; unioned, those folders are one show and the special's bigger
|
||||
files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
|
||||
the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
|
||||
staged file rests on a borrowed id.
|
||||
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
|
||||
2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
|
||||
Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
|
||||
season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
|
||||
the pass would have staged about 340 GB of unique episodes; judged per episode it
|
||||
staged 54 GiB of true copies.
|
||||
- **A file is droppable only when every episode it carries is held elsewhere in the
|
||||
same show at >= its size** (Frank's rule, applied per episode). A byte-identical
|
||||
pair ties and the tie breaks by path so exactly one copy stages; a file holding
|
||||
one unique episode stays whatever else it carries.
|
||||
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
|
||||
poster or a whole season, so key-less names are never staged and are counted in
|
||||
the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
|
||||
Where a packager named files by id, Jellyfin's own table is the witness: a
|
||||
`path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
|
||||
and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
|
||||
rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
|
||||
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
|
||||
`pragma table_info` before trusting the query, because an empty witness silently
|
||||
falls back to name parsing.
|
||||
Seven clips in the library are still named only by number, and where Jellyfin
|
||||
was asked its own `Name` for them is the number too, so no database holds a
|
||||
better title — that residue is naming, not a task.
|
||||
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
|
||||
when both halves are held elsewhere, and it wins the survivor seat for both when
|
||||
it is the largest carrier: the byte law compares files, and a name both servers
|
||||
parse as a range stays indexed as two episodes.
|
||||
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
|
||||
artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
|
||||
survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
|
||||
copies were bare rips, and counting a hold's non-media files is the census that
|
||||
tells you which case you are in.
|
||||
- **The audit gates the plan file.** Every survivor is re-statted and every
|
||||
destination re-checked before `--out` is written; on failure the report prints
|
||||
`AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
|
||||
- **Two folders that look duplicated and share zero episode keys are complementary,
|
||||
not duplicated.** That is how the two pairs the census still refuses to merge were
|
||||
settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
|
||||
beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
|
||||
Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
|
||||
episode keys: 0` in both cases, so a tag would have freed nothing and cost a
|
||||
rename with library consequences. Re-running the whole census with the id map over
|
||||
all 753 top-level folders (606 already tagged) staged **0 files**: the library is
|
||||
duplicate-free to the limit of provable identity, and what remains is naming.
|
||||
|
||||
## Guards
|
||||
|
||||
Each of these is a file that would have been destroyed without it:
|
||||
@@ -97,6 +180,14 @@ keeper is missing — either one drops the whole group rather than half-applying
|
||||
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
|
||||
`dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
|
||||
- The 2026-11-02 TV pass holds 220 files (54 GiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
|
||||
journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
|
||||
same day's TVShows detritus pass holds 109 files (552 KiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
|
||||
`tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges
|
||||
pending Frank's word, beside the 2026-11-01 hold above. All four journals and
|
||||
plans live at the `entropy` root, not `/tmp`.
|
||||
- **Re-verify each survivor by size, not by the recorded path, before purging.**
|
||||
Detection artifacts and plans key on paths, and a folder renamed by an
|
||||
interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look
|
||||
@@ -117,7 +208,9 @@ stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack o
|
||||
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
|
||||
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
|
||||
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
|
||||
held path.
|
||||
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
|
||||
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
|
||||
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.
|
||||
|
||||
The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
|
||||
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
|
||||
@@ -128,6 +221,10 @@ subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
|
||||
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
|
||||
extension the census does not recognise is a reason to stop, not to sweep.
|
||||
|
||||
A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
|
||||
file is a manual match — a human decision about what the folder is — and detritus
|
||||
rules do not overrule one.
|
||||
|
||||
`scripts/media-verify.py` reads differently for a prune than for a relayout:
|
||||
`landed: 0` is the **correct** end state (no server may index the hold), so the
|
||||
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
|
||||
|
||||
@@ -138,7 +138,7 @@ HTTPD close without a single byte; and the same HTTPD drops requests that carry
|
||||
no `Accept` and client-identifier header (see the scan block above for the form
|
||||
that works).
|
||||
|
||||
A duplicate prune is the same census and the same proof, with its own keep rules: [[Media Duplicate Prune]].
|
||||
A duplicate prune is the same census and the same proof, with its own keep rules — films by title and duration, a series per episode: [[Media Duplicate Prune]].
|
||||
|
||||
## TV in the movie library
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@ is a locator: one row, one clause, no numbers.
|
||||
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
|
||||
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
|
||||
| [[Media Audiobook Relayout]] | Re-filing /mnt/entropy/Audiobooks by Plex's chapter rows: narrator shelves, no renumbering, the refusals. |
|
||||
| [[Media Duplicate Prune]] | Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge. |
|
||||
| [[Media Duplicate Prune]] | Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge. |
|
||||
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
|
||||
| [[Media Music Relayout]] | Re-filing /mnt/entropy/Music by Jellyfin's tag rows: the census, the album key, the refusals. |
|
||||
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
|
||||
|
||||
Reference in new issue
Block a user