serverAppConfigs/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md

---
type: reference
title: Media Duplicate Prune
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
tags: [dankserver, media-server, storage, procedure]
timestamp: 2026-11-02T01:00:00Z
---

# Media Duplicate Prune

The movie library holds the same film twice in two different ways — the same file
downloaded again, or the same film grabbed a second time at another quality — and
[[Entropy Volume]] is permanently full, so both are worth finding. They need
different rules, and neither is decided by name similarity alone.

A series duplicates differently again: one show spread over several folders, so
neither the file nor the folder is the unit — see *A series is measured per
episode*.

`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
and films nothing served. It is gone; its contents went to `NeedsProcessing`.

## What counts

| Class | Test | Action |
|---|---|---|
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |

## The screen

The array cannot afford a full-content read, so identity is established in steps
and each step only narrows what the next one touches:

1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
   evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
   the better part of an hour; the sparse window cost 50 s, a file that merely
   shares its head and tail cannot survive it, and the destination is a
   recoverable hold rather than a deletion.
4. Durations come from `scripts/media-census.py probe` — the relayout's own
   census. Do not hand-roll a second probe: it re-seeks the array for an answer
   the repo tool already produces, and a probe is the one census step that costs
   hours instead of seconds.

## Which copy survives

In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
folder (an extra is not a copy of the film, and extra names collide across films);
then the shallowest path; then the file whose name equals its `Title (year)`
folder; then the largest. For the same-work class the largest is the answer, and
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
outlive `Film.mkv` for no reason but a space sorting before a dot.

## A series is measured per episode

`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
executor's own format; it never moves anything itself.

```bash
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
    --hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
    --meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
    --ids /mnt/entropy/tv-tvdb-ids.tsv \
    --out /mnt/entropy/tv-dupes-plan.tsv --json /tmp/tv-groups.json
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
```

- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
  name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
  and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
  match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
  `BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
  stripped to the bare top-level folder name — 616 folders, none carrying two
  ids, written to `/mnt/entropy/tv-tvdb-ids.tsv`. Two folders with disjoint ids
  are two shows however alike they read — *Avatar: The Last Airbender* (1999)
  beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
  had also titled *Millennium* — and a pair whose identity cannot be proven is
  reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
  fill an untagged folder and are **dropped** when they contradict its own
  `[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
  provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
  [tvdbid-73838]`; unioned, those folders are one show and the special's bigger
  files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
  the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
  staged file rests on a borrowed id.
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
  2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
  Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
  season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
  the pass would have staged about 340 GB of unique episodes; judged per episode it
  staged 54 GiB of true copies.
- **A file is droppable only when every episode it carries is held elsewhere in the
  same show at >= its size** (Frank's rule, applied per episode). A byte-identical
  pair ties and the tie breaks by path so exactly one copy stages; a file holding
  one unique episode stays whatever else it carries.
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
  poster or a whole season, so key-less names are never staged and are counted in
  the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
 Where a packager named files by id, Jellyfin's own table is the witness: a
 `path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
 and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
 rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
`pragma table_info` before trusting the query, because an empty witness silently
falls back to name parsing.
  Seven clips in the library are still named only by number, and where Jellyfin
  was asked its own `Name` for them is the number too, so no database holds a
  better title — that residue is naming, not a task.
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
  when both halves are held elsewhere, and it wins the survivor seat for both when
  it is the largest carrier: the byte law compares files, and a name both servers
  parse as a range stays indexed as two episodes.
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
  artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
  survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
  copies were bare rips, and counting a hold's non-media files is the census that
  tells you which case you are in.
- **The audit gates the plan file.** Every survivor is re-statted and every
  destination re-checked before `--out` is written; on failure the report prints
  `AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
- **Two folders that look duplicated and share zero episode keys are complementary,
  not duplicated.** That is how the two pairs the census still refuses to merge were
  settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
  beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
  Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
  episode keys: 0` in both cases, so a tag would have freed nothing and cost a
  rename with library consequences. Re-running the whole census with the id map over
  all 753 top-level folders (606 already tagged) staged **0 files**: the library is
  duplicate-free to the limit of provable identity, and what remains is naming.

## Guards

Each of these is a file that would have been destroyed without it:

- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
  the finished file while qB still counts the download incomplete; moving it
  breaks the re-check and re-downloads the film. Report it, leave it. Both
  leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
  not a prune decision.
- **Require a year, and keep digits in the title.** Stripping digits merged
  *Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
  happened to share a name merged.
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
  is excluded: half a multi-disc release is not a copy of the other half.
- Media containers only. Grouping every extension buries the report in subtitle
  and cover pairs — and a duration guard that appears to fire on 720 groups is
  counting subtitles, not protecting films.

## The hold

Nothing is deleted. Every doomed copy moves into a dated hold with the same
executor the relayout uses, so any file comes back:

```bash
scripts/media-apply.py apply   --plan /mnt/entropy/dup-prune-similar-plan.tsv \
    --root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl          # dry
scripts/media-apply.py apply   --plan … --root /mnt/entropy --journal … --apply
scripts/media-apply.py verify  --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
scripts/media-apply.py sweep   --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
```

The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
every hold parent), and the planner re-checks the two things that go stale between
analysis and apply: a file that has moved since the census, and a group whose
keeper is missing — either one drops the whole group rather than half-applying it.

- **A hold on the same volume frees nothing.** The purge is a separate, approved
  act and the only thing that moves the free-space number; `df` does not budge
  when 1 TiB moves sideways.
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
  ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
  era, not this pass — read the mtimes before blaming the prune.
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
  `/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
  `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.
- The 2026-11-02 TV pass holds 220 files (54 GiB) in
  `/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
  journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
  same day's TVShows detritus pass holds 109 files (552 KiB) in
  `/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
  `tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. Both purges
  pending Frank's word, beside the 2026-11-01 hold above. All four journals and
  plans live at the `entropy` root, not `/tmp`.
- **Re-verify each survivor by size, not by the recorded path, before purging.**
  Detection artifacts and plans key on paths, and a folder renamed by an
  interactive session ([[Entropy Volume]], *Write doors*) makes its keeper look
  deleted while it sits untouched under the new name. The 2026-11-01 audit hit
  three such groups: every keeper was present, same bytes, different folder. A
  purge decided from the plan's paths alone would have reported survivors gone.

## After the pass

Both servers still hold rows for files that are now in the hold, and neither
notices: the media bind is the [[Media Metadata View]], and array changes do not
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
are owned by [[Jellyfin]] and [[Plex]].

The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user
stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.

The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
from folders whose whole subtree held no media, so it creates no stale rows and
needs no scan. The guards that hold a folder back are the finding: external
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
extension the census does not recognise is a reason to stop, not to sweep.

A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
file is a manual match — a human decision about what the folder is — and detritus
rules do not overrule one.

`scripts/media-verify.py` reads differently for a prune than for a relayout:
`landed: 0` is the **correct** end state (no server may index the hold), so the
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
from the database are the pass succeeding.

One residue is normal, so do not chase it: a section pass rewrites the feature rows
first, and rows for clips in a film's `other/` or `extras/` subfolder can still name
a held path after the section's `refreshing` flag is back to `0` — and a second
section pass does not clear them either, so do not run one hoping for `0`. Beside
`REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch
the cache, the database, or the hold.

## Failure signatures

| Symptom | Meaning |
|---|---|
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |

## Links

- [[Media Relayout]] — the census tools, the executor, the journal format.
- [[Media Metadata View]] — why neither server sees the change without a scan.
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
- [[Entropy Volume]] — the array that is always full.