--- type: reference title: Media Duplicate Prune description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge." tags: [dankserver, media-server, storage, procedure] timestamp: 2026-11-02T01:00:00Z --- # Media Duplicate Prune The movie library holds the same film twice in two different ways — the same file downloaded again, or the same film grabbed a second time at another quality — and [[Entropy Volume]] is permanently full, so both are worth finding. They need different rules, and neither is decided by name similarity alone. A series duplicates differently again: one show spread over several folders, so neither the file nor the folder is the unit — see *A series is measured per episode*. `/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows and films nothing served. It is gone; its contents went to `NeedsProcessing`. ## What counts | Class | Test | Action | |---|---|---| | Byte-identical | equal size, equal content hash | keep one copy, hold the rest | | Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule | | Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products | | Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file | | Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) | ## The screen The array cannot afford a full-content read, so identity is established in steps and each step only narrows what the next one touches: 1. `scripts/media-census.py enumerate` — size is free, and identical files share it. 2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only. 3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for the better part of an hour; the sparse window cost 50 s, a file that merely shares its head and tail cannot survive it, and the destination is a recoverable hold rather than a deletion. 4. Durations come from `scripts/media-census.py probe` — the relayout's own census. Do not hand-roll a second probe: it re-seeks the array for an answer the repo tool already produces, and a probe is the one census step that costs hours instead of seconds. ## Which copy survives In order: not one inside an `extras`, `featurettes`, `samples` or `trailers` folder (an extra is not a copy of the film, and extra names collide across films); then the shallowest path; then the file whose name equals its `Title (year)` folder; then the largest. For the same-work class the largest is the answer, and the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can outlive `Film.mkv` for no reason but a space sorting before a dot. ## A series is measured per episode `scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the executor's own format; it never moves anything itself. ```bash python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \ --hold /mnt/entropy/.AGENTTRASH/-tv-duplicates \ --meta /var/lib/media-meta/view/TVShows --witness witness.tsv \ --ids /mnt/entropy/tv-tvdb-ids.tsv \ --out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json # then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune ``` - **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]` name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`), and from a `folderid` map (`--ids`). Build that map from Jellyfin's own match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to `BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix stripped to the bare top-level folder name — 616 folders, none carrying two ids, written to `/mnt/entropy/tv-tvdb-ids.tsv`. Two folders with disjoint ids are two shows however alike they read — *Avatar: The Last Airbender* (1999) beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin had also titled *Millennium* — and a pair whose identity cannot be proven is reported `UNRESOLVED` and left alone. The fix is the tag, never a guess. - **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer fill an untagged folder and are **dropped** when they contradict its own `[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the provider table carries a 2026 special's id on `Malcolm in the Middle (2000) [tvdbid-73838]`; unioned, those folders are one show and the special's bigger files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no staged file rests on a borrowed id. - **A folder-level verdict destroys content.** Of 25 structural candidates in the 2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder the pass would have staged about 340 GB of unique episodes; judged per episode it staged 54 GiB of true copies. - **A file is droppable only when every episode it carries is held elsewhere in the same show at >= its size** (Frank's rule, applied per episode). A byte-identical pair ties and the tie breaks by path so exactly one copy stages; a file holding one unique episode stays whatever else it carries. - **A bare number is never an episode.** `256383.mp4` may be an episode id, a poster or a whole season, so key-less names are never staged and are counted in the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material). Where a packager named files by id, Jellyfin's own table is the witness: a `pathseasonepisode` TSV from `BaseItems` where `Type like '%TV.Episode'` and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and `IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with `pragma table_info` before trusting the query, because an empty witness silently falls back to name parsing. Seven clips in the library are still named only by number, and where Jellyfin was asked its own `Name` for them is the number too, so no database holds a better title — that residue is naming, not a task. - **A pack states a range and competes on all of it.** `S01E01-02` is droppable only when both halves are held elsewhere, and it wins the survivor seat for both when it is the largest carrier: the byte law compares files, and a name both servers parse as a range stays indexed as two episodes. - **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`, artwork keyed on the stem — so a staged duplicate leaves no orphan beside the survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated copies were bare rips, and counting a hold's non-media files is the census that tells you which case you are in. - **The audit gates the plan file.** Every survivor is re-statted and every destination re-checked before `--out` is written; on failure the report prints `AUDIT FAILED` and writes no plan, so there is nothing to hand the executor. - **Two folders that look duplicated and share zero episode keys are complementary, not duplicated.** That is how the two pairs the census still refuses to merge were settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB) beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared episode keys: 0` in both cases, so a tag would have freed nothing and cost a rename with library consequences. Re-running the whole census with the id map over all 753 top-level folders (606 already tagged) staged **0 files**: the library is duplicate-free to the limit of provable identity, and what remains is naming. ## Guards Each of these is a file that would have been destroyed without it: - **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to the finished file while qB still counts the download incomplete; moving it breaks the re-check and re-downloads the film. Report it, leave it. Both leftovers in `Movies` are 13 months stale, so they are Frank's purge decision, not a prune decision. - **Require a year, and keep digits in the title.** Stripping digits merged *Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that happened to share a name merged. - Both members must be feature length, and a name carrying `disc`, `cd` or `part` is excluded: half a multi-disc release is not a copy of the other half. - Media containers only. Grouping every extension buries the report in subtitle and cover pairs — and a duration guard that appears to fire on 720 groups is counting subtitles, not protecting films. ## The hold Nothing is deleted. Every doomed copy moves into a dated hold with the same executor the relayout uses, so any file comes back: ```bash scripts/media-apply.py apply --plan /mnt/entropy/dup-prune-similar-plan.tsv \ --root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl # dry scripts/media-apply.py apply --plan … --root /mnt/entropy --journal … --apply scripts/media-apply.py verify --plan … --journal /mnt/entropy/dup-prune-similar.jsonl scripts/media-apply.py sweep --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply ``` The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for every hold parent), and the planner re-checks the two things that go stale between analysis and apply: a file that has moved since the census, and a group whose keeper is missing — either one drops the whole group rather than half-applying it. - **A hold on the same volume frees nothing.** The purge is a separate, approved act and the only thing that moves the free-space number; `df` does not budge when 1 TiB moves sideways. - `sweep` removes the release folders the pass emptied. Empty folders dated a year ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous era, not this pass — read the mtimes before blaming the prune. - The 2026-11-01 pass held 226 files (~1.1 TiB) in `/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word. - The 2026-11-02 TV pass holds 220 files (54 GiB) in `/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`, journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the same day's TVShows detritus pass holds 109 files (552 KiB) in `/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan `tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges pending Frank's word, beside the 2026-11-01 hold above. All four journals and plans live at the `entropy` root, not `/tmp`. - **Re-verify each survivor by size, not by the recorded path, before purging.** Detection artifacts and plans key on paths, and a folder renamed by an interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look deleted while it sits untouched under the new name. The 2026-11-01 audit hit three such groups: every keeper was present, same bytes, different folder. A purge decided from the plan's paths alone would have reported survivors gone. ## After the pass Both servers still hold rows for files that are now in the hold, and neither notices: the media bind is the [[Media Metadata View]], and array changes do not cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to `Idle`, then Plex's Movies section alone, one accepted refresh each; the requests are owned by [[Jellyfin]] and [[Plex]]. The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a live Jellyfin scan, and because a masked unit stays stopped through a reboot — `unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached `Idle`: one accepted `refresh` on the Movies section cleared every row that named a held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one accepted refresh, then `media-verify.py` against that journal reads `landed: 0`, `still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune. The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in `/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`, journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only from folders whose whole subtree held no media, so it creates no stale rows and needs no scan. The guards that hold a folder back are the finding: external subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS, `.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an extension the census does not recognise is a reason to stop, not to sweep. A folder holding a `.plexmatch` is held back even when its subtree looks empty: that file is a manual match — a human decision about what the folder is — and detritus rules do not overrule one. `scripts/media-verify.py` reads differently for a prune than for a relayout: `landed: 0` is the **correct** end state (no server may index the hold), so the verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished from the database are the pass succeeding. One residue is normal, so do not chase it: a section pass rewrites the feature rows first, and rows for clips in a film's `other/` or `extras/` subfolder can still name a held path after the section's `refreshing` flag is back to `0` — and a second section pass does not clear them either, so do not run one hoping for `0`. Beside `REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch the cache, the database, or the hold. ## Failure signatures | Symptom | Meaning | |---|---| | A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. | | The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. | | `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. | | A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. | ## Links - [[Media Relayout]] — the census tools, the executor, the journal format. - [[Media Metadata View]] — why neither server sees the change without a scan. - [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time. - [[Entropy Volume]] — the array that is always full.