serverAppConfigs/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Media Duplicate Prune	Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge.	dankserver media-server storage procedure	2026-11-02T01:00:00Z

Media Duplicate Prune

The movie library holds the same film twice in two different ways — the same file downloaded again, or the same film grabbed a second time at another quality — and Entropy Volume is permanently full, so both are worth finding. They need different rules, and neither is decided by name similarity alone.

A series duplicates differently again: one show spread over several folders, so neither the file nor the folder is the unit — see A series is measured per episode.

/mnt/entropy/media-park was a third kind: a parking lot that held whole shows and films nothing served. It is gone; its contents went to NeedsProcessing.

What counts

Class	Test	Action
Byte-identical	equal size, equal content hash	keep one copy, hold the rest
Same work, worse encode	same title, same year, every runtime within 5 %	keep the largest — Frank's rule
Different cut	same title and year, runtimes more than 5 % apart	never touched: an extended and a theatrical version are different products
Same words, different film	title matches, year or duration does not	never grouped — this is the false-merge class that would destroy a unique file
Series split across folders	same show by tvdb id, then the episode (S,E) — never the folder name	keep the largest carrier of each episode (scripts/media-tvdupes.py)

The screen

The array cannot afford a full-content read, so identity is established in steps and each step only narrows what the next one touches:

1. scripts/media-census.py enumerate — size is free, and identical files share it.
2. Head+tail blake2b, 4 MiB at each end, on the size collisions only.
3. Before anything moves, confirm with a sparse-window hash: 32 MiB at eight evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for the better part of an hour; the sparse window cost 50 s, a file that merely shares its head and tail cannot survive it, and the destination is a recoverable hold rather than a deletion.
4. Durations come from scripts/media-census.py probe — the relayout's own census. Do not hand-roll a second probe: it re-seeks the array for an answer the repo tool already produces, and a probe is the one census step that costs hours instead of seconds.

Which copy survives

In order: not one inside an extras, featurettes, samples or trailers folder (an extra is not a copy of the film, and extra names collide across films); then the shallowest path; then the file whose name equals its Title (year) folder; then the largest. For the same-work class the largest is the answer, and the canonical-name rule only breaks a tie — without it a Film - 1080p.mkv can outlive Film.mkv for no reason but a space sorting before a dot.

A series is measured per episode

scripts/media-tvdupes.py censuses a TV library and writes a staging plan in the executor's own format; it never moves anything itself.

python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
    --hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
    --meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
    --ids /mnt/entropy/tv-tvdb-ids.tsv \
    --out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune

• Identity before geometry. A folder's tvdb id is read from its [tvdbid-N] name tag, from the tvshow.nfo the Media Metadata View exposes (--meta), and from a folder<TAB>id map (--ids). Build that map from Jellyfin's own match, not a lookup: BaseItems (Type like '%TV.Series') joined to BaseItemProviders where ProviderId = 'Tvdb', the /data/TVShows/ prefix stripped to the bare top-level folder name — 616 folders, none carrying two ids, written to /mnt/entropy/tv-tvdb-ids.tsv. Two folders with disjoint ids are two shows however alike they read — Avatar: The Last Airbender (1999) beside the 2024 remake, Millennium (1996) beside a German thriller Jellyfin had also titled Millennium — and a pair whose identity cannot be proven is reported UNRESOLVED and left alone. The fix is the tag, never a guess.
• A tag outranks every derived id. The --ids map and the tvshow.nfo layer fill an untagged folder and are dropped when they contradict its own [tvdbid-N], which the report prints as DISAGREE. Measured the same day: the provider table carries a 2026 special's id on Malcolm in the Middle (2000) [tvdbid-73838]; unioned, those folders are one show and the special's bigger files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for the hold as "worse copies". The pass that staged 54 GiB ran with no --ids, so no staged file rests on a borrowed id.
• A folder-level verdict destroys content. Of 25 structural candidates in the 2026-11-02 census, 10 were complementary halves: Game of Thrones: The Complete Series holds no S01 and no S08, Modern Family S05 is the only copy of season 5, Loiter Squad Uncensored the only copy of season 1. Judged per folder the pass would have staged about 340 GB of unique episodes; judged per episode it staged 54 GiB of true copies.
• A file is droppable only when every episode it carries is held elsewhere in the same show at >= its size (Frank's rule, applied per episode). A byte-identical pair ties and the tie breaks by path so exactly one copy stages; a file holding one unique episode stays whatever else it carries.
• A bare number is never an episode. 256383.mp4 may be an episode id, a poster or a whole season, so key-less names are never staged and are counted in the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material). Where a packager named files by id, Jellyfin's own table is the witness: a path<TAB>season<TAB>episode TSV from BaseItems where Type like '%TV.Episode' and both index numbers are set, read mode=ro&immutable=1, its /data prefix rewritten to the raw array path. Jellyfin 12 keeps Path, ParentIndexNumber and IndexNumber on BaseItems and stores the full CLR type name — confirm with pragma table_info before trusting the query, because an empty witness silently falls back to name parsing. Seven clips in the library are still named only by number, and where Jellyfin was asked its own Name for them is the number too, so no database holds a better title — that residue is naming, not a task.
• A pack states a range and competes on all of it. S01E01-02 is droppable only when both halves are held elsewhere, and it wins the survivor seat for both when it is the largest carrier: the byte law compares files, and a name both servers parse as a range stays indexed as two episodes.
• A sidecar that shares the doomed copy's stem goes with it — subtitles, .nfo, artwork keyed on the stem — so a staged duplicate leaves no orphan beside the survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated copies were bare rips, and counting a hold's non-media files is the census that tells you which case you are in.
• The audit gates the plan file. Every survivor is re-statted and every destination re-checked before --out is written; on failure the report prints AUDIT FAILED and writes no plan, so there is nothing to hand the executor.
• Two folders that look duplicated and share zero episode keys are complementary, not duplicated. That is how the two pairs the census still refuses to merge were settled without touching them: Star Wars - The Clone Wars (12 files, 1.3 GiB) beside Star Wars The Clone Wars (121 files, 214 GiB), and the Batman: The Animated Series leftovers (3 files) beside the tagged folder (85) — shared episode keys: 0 in both cases, so a tag would have freed nothing and cost a rename with library consequences. Re-running the whole census with the id map over all 753 top-level folders (606 already tagged) staged 0 files: the library is duplicate-free to the limit of provable identity, and what remains is naming.

Guards

Each of these is a file that would have been destroyed without it:

• .!qB is qBittorrent state, not a duplicate. It can be byte-identical to the finished file while qB still counts the download incomplete; moving it breaks the re-check and re-downloads the film. Report it, leave it. Both leftovers in Movies are 13 months stale, so they are Frank's purge decision, not a prune decision.
• Require a year, and keep digits in the title. Stripping digits merged Akira — Teaser 1 with Teaser 2; without a year, two different extras that happened to share a name merged.
• Both members must be feature length, and a name carrying disc, cd or part is excluded: half a multi-disc release is not a copy of the other half.
• Media containers only. Grouping every extension buries the report in subtitle and cover pairs — and a duration guard that appears to fire on 720 groups is counting subtitles, not protecting films.

The hold

Nothing is deleted. Every doomed copy moves into a dated hold with the same executor the relayout uses, so any file comes back:

scripts/media-apply.py apply   --plan /mnt/entropy/dup-prune-similar-plan.tsv \
    --root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl          # dry
scripts/media-apply.py apply   --plan … --root /mnt/entropy --journal … --apply
scripts/media-apply.py verify  --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
scripts/media-apply.py sweep   --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply

The plan rows are the tool's own format (action src dst kind, mkdir rows for every hold parent), and the planner re-checks the two things that go stale between analysis and apply: a file that has moved since the census, and a group whose keeper is missing — either one drops the whole group rather than half-applying it.

• A hold on the same volume frees nothing. The purge is a separate, approved act and the only thing that moves the free-space number; df does not budge when 1 TiB moves sideways.
• sweep removes the release folders the pass emptied. Empty folders dated a year ago are torrent detritus (Subtitle,info, Screens, Sample) from a previous era, not this pass — read the mtimes before blaming the prune.
• The 2026-11-01 pass held 226 files (~1.1 TiB) in /mnt/entropy/.AGENTTRASH/20261101-duplicate-prune, journals dup-prune-exact.jsonl and dup-prune-similar.jsonl. Purge pending Frank's word.
• The 2026-11-02 TV pass holds 220 files (54 GiB) in /mnt/entropy/.AGENTTRASH/20261102-tv-duplicates — plan tv-dupes-plan.tsv, journal tv-dupes.jsonl, 220 moves / 16 hold folders / 7 husks swept — and the same day's TVShows detritus pass holds 109 files (552 KiB) in /mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus with plan tvshows-detritus-plan.tsv and journal tvshows-detritus.jsonl. Both purges pending Frank's word, beside the 2026-11-01 hold above. All four journals and plans live at the entropy root, not /tmp.
• Re-verify each survivor by size, not by the recorded path, before purging. Detection artifacts and plans key on paths, and a folder renamed by an interactive session (Entropy Volume, Write doors) makes its keeper look deleted while it sits untouched under the new name. The 2026-11-01 audit hit three such groups: every keeper was present, same bytes, different folder. A purge decided from the plan's paths alone would have reported survivors gone.

After the pass

Both servers still hold rows for files that are now in the hold, and neither notices: the media bind is the Media Metadata View, and array changes do not cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to Idle, then Plex's Movies section alone, one accepted refresh each; the requests are owned by Jellyfin and Plex.

The Plex leg may be deferred on purpose: stop and mask Plex (systemctl --user stop plex && systemctl --user mask plex) so its metadata fetches cannot stack on a live Jellyfin scan, and because a masked unit stays stopped through a reboot — unmask before starting it. The 2026-11-01 morning ran it once Jellyfin reached Idle: one accepted refresh on the Movies section cleared every row that named a held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one accepted refresh, then media-verify.py against that journal reads landed: 0, still at src: 0, REAL: 0 on both servers — the correct end state for a prune.

The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in /mnt/entropy/.AGENTTRASH/20261101-detritus-sweep, plan detritus-sweep-plan.tsv, journal detritus-sweep.jsonl, 542 husks removed by sweep. It moved files only from folders whose whole subtree held no media, so it creates no stale rows and needs no scan. The guards that hold a folder back are the finding: external subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — .sup PGS, .thd/.eac3/.dts audio, .cci captions, 25 GiB — are not trash, and an extension the census does not recognise is a reason to stop, not to sweep.

A folder holding a .plexmatch is held back even when its subtree looks empty: that file is a manual match — a human decision about what the folder is — and detritus rules do not overrule one.

scripts/media-verify.py reads differently for a prune than for a relayout: landed: 0 is the correct end state (no server may index the hold), so the verdict is still at src: 0 with REAL: 0, and the rows that simply vanished from the database are the pass succeeding.

One residue is normal, so do not chase it: a section pass rewrites the feature rows first, and rows for clips in a film's other/ or extras/ subfolder can still name a held path after the section's refreshing flag is back to 0 — and a second section pass does not clear them either, so do not run one hoping for 0. Beside REAL: 0 that is a dead disc-menu clip — not a false prune, and no reason to touch the cache, the database, or the hold.

Failure signatures

Symptom	Meaning
A size group whose members split under the sparse hash	The head+tail screen collided. Those files stay exactly where they are and get reported.
The planner holds far fewer files than the analyzer counted	Two implementations of "largest wins". The analyzer must emit keep/hold marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing.
df unchanged after a TiB of moves	Expected: the hold is on the same filesystem. Only the purge frees space.
A film that "disappeared" after a prune	It is in the hold, named by the journal: media-apply.py rollback --journal … --apply.

Links

• Media Relayout — the census tools, the executor, the journal format.
• Media Metadata View — why neither server sees the change without a scan.
• Jellyfin · Plex — the scan requests, one server at a time.
• Entropy Volume — the array that is always full.