serverAppConfigs/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md

---
type: reference
title: Media Duplicate Prune
description: "Find and hold duplicate films and series on the array: the screen, per-episode rule, guards, purge."
tags: [dankserver, media-server, storage, procedure]
timestamp: 2026-11-02T01:00:00Z
---

# Media Duplicate Prune

The movie library holds the same film twice in two different ways — the same file
downloaded again, or the same film grabbed a second time at another quality — and
[[Entropy Volume]] is permanently full, so both are worth finding. They need
different rules, and neither is decided by name similarity alone.

A series duplicates differently again: one show spread over several folders, so
neither the file nor the folder is the unit — see *A series is measured per
episode*.

`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
and films nothing served. It is gone; its contents went to `NeedsProcessing`.

## What counts

| Class | Test | Action |
|---|---|---|
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |
| Series split across folders | same show by tvdb id, then the episode `(S,E)` — never the folder name | keep the largest carrier of each episode (`scripts/media-tvdupes.py`) |

## The screen

The array cannot afford a full-content read, so identity is established in steps
and each step only narrows what the next one touches:

1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
   evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
   the better part of an hour; the sparse window cost 50 s, a file that merely
   shares its head and tail cannot survive it, and the destination is a
   recoverable hold rather than a deletion.
4. Durations come from `scripts/media-census.py probe` — the relayout's own
   census. Do not hand-roll a second probe: it re-seeks the array for an answer
   the repo tool already produces, and a probe is the one census step that costs
   hours instead of seconds.

## Which copy survives

In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
folder (an extra is not a copy of the film, and extra names collide across films);
then the shallowest path; then the file whose name equals its `Title (year)`
folder; then the largest. For the same-work class the largest is the answer, and
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
outlive `Film.mkv` for no reason but a space sorting before a dot.

## A series is measured per episode

`scripts/media-tvdupes.py` censuses a TV library and writes a staging plan in the
executor's own format; it never moves anything itself.

```bash
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
    --hold /mnt/entropy/.AGENTTRASH/<date>-tv-duplicates \
    --meta /var/lib/media-meta/view/TVShows --witness witness.tsv \
    --ids /mnt/entropy/.media-campaign/tv-tvdb-ids.tsv \
    --out /mnt/entropy/.media-campaign/tv-dupes-plan.tsv --json /tmp/tv-groups.json
# then media-apply.py apply / verify / sweep / rollback, exactly as for a film prune
```

- **Identity before geometry.** A folder's tvdb id is read from its `[tvdbid-N]`
  name tag, from the `tvshow.nfo` the [[Media Metadata View]] exposes (`--meta`),
  and from a `folder<TAB>id` map (`--ids`). Build that map from Jellyfin's own
  match, not a lookup: `BaseItems` (`Type like '%TV.Series'`) joined to
  `BaseItemProviders` where `ProviderId = 'Tvdb'`, the `/data/TVShows/` prefix
  stripped to the bare top-level folder name — 616 folders, none carrying two
  ids, written to `/mnt/entropy/.media-campaign/tv-tvdb-ids.tsv`. Two folders with disjoint ids
  are two shows however alike they read — *Avatar: The Last Airbender* (1999)
  beside the 2024 remake, *Millennium* (1996) beside a German thriller Jellyfin
  had also titled *Millennium* — and a pair whose identity cannot be proven is
  reported `UNRESOLVED` and left alone. The fix is the tag, never a guess.
- **A tag outranks every derived id.** The `--ids` map and the `tvshow.nfo` layer
  fill an untagged folder and are **dropped** when they contradict its own
  `[tvdbid-N]`, which the report prints as `DISAGREE`. Measured the same day: the
  provider table carries a 2026 special's id on `Malcolm in the Middle (2000)
  [tvdbid-73838]`; unioned, those folders are one show and the special's bigger
  files dominate the original's S01E01–E04 — 3.2 GiB of the 2000 series bound for
  the hold as "worse copies". The pass that staged 54 GiB ran with no `--ids`, so no
  staged file rests on a borrowed id.
- **A folder-level verdict destroys content.** Of 25 structural candidates in the
  2026-11-02 census, **10 were complementary halves**: *Game of Thrones: The
  Complete Series* holds no S01 and no S08, `Modern Family S05` is the only copy of
  season 5, *Loiter Squad Uncensored* the only copy of season 1. Judged per folder
  the pass would have staged about 340 GB of unique episodes; judged per episode it
  staged 54 GiB of true copies.
- **A file is droppable only when every episode it carries is held elsewhere in the
  same show at >= its size** (Frank's rule, applied per episode). A byte-identical
  pair ties and the tie breaks by path so exactly one copy stages; a file holding
  one unique episode stays whatever else it carries.
- **A bare number is never an episode.** `256383.mp4` may be an episode id, a
  poster or a whole season, so key-less names are never staged and are counted in
  the report (4,036 after the 2026-11-02 pass, almost all genuine bonus material).
 Where a packager named files by id, Jellyfin's own table is the witness: a
 `path<TAB>season<TAB>episode` TSV from `BaseItems` where `Type like '%TV.Episode'`
 and both index numbers are set, read `mode=ro&immutable=1`, its `/data` prefix
 rewritten to the raw array path. Jellyfin 12 keeps `Path`, `ParentIndexNumber` and
`IndexNumber` on `BaseItems` and stores the full CLR type name — confirm with
`pragma table_info` before trusting the query, because an empty witness silently
falls back to name parsing.
  Seven clips in the library are still named only by number, and where Jellyfin
  was asked its own `Name` for them is the number too, so no database holds a
  better title — that residue is naming, not a task.
- **A pack states a range and competes on all of it.** `S01E01-02` is droppable only
  when both halves are held elsewhere, and it wins the survivor seat for both when
  it is the largest carrier: the byte law compares files, and a name both servers
  parse as a range stays indexed as two episodes.
- **A sidecar that shares the doomed copy's stem goes with it** — subtitles, `.nfo`,
  artwork keyed on the stem — so a staged duplicate leaves no orphan beside the
  survivor. The 2026-11-02 hold is 220 media files and nothing else: the dominated
  copies were bare rips, and counting a hold's non-media files is the census that
  tells you which case you are in.
- **The audit gates the plan file.** Every survivor is re-statted and every
  destination re-checked before `--out` is written; on failure the report prints
  `AUDIT FAILED` and writes no plan, so there is nothing to hand the executor.
- **Two folders that look duplicated and share zero episode keys are complementary,
  not duplicated.** That is how the two pairs the census still refuses to merge were
  settled without touching them: *Star Wars - The Clone Wars* (12 files, 1.3 GiB)
  beside *Star Wars The Clone Wars* (121 files, 214 GiB), and the *Batman: The
  Animated Series* leftovers (3 files) beside the tagged folder (85) — `shared
  episode keys: 0` in both cases, so a tag would have freed nothing and cost a
  rename with library consequences. Re-running the whole census with the id map over
  all 753 top-level folders (606 already tagged) staged **0 files**: the library is
  duplicate-free to the limit of provable identity, and what remains is naming.

## Guards

Each of these is a file that would have been destroyed without it:

- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
  the finished file while qB still counts the download incomplete; moving it
  breaks the re-check and re-downloads the film. Report it, leave it. Both
  leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
  not a prune decision.
- **Require a year, and keep digits in the title.** Stripping digits merged
  *Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
  happened to share a name merged.
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
  is excluded: half a multi-disc release is not a copy of the other half.
- Media containers only. Grouping every extension buries the report in subtitle
  and cover pairs — and a duration guard that appears to fire on 720 groups is
  counting subtitles, not protecting films.
- **A short file named after a film is promotion, not a movie.** Thirty-two files
  in `Movies` — 31 clips and the `.nfo` filed beside one of them — carried real
  titles (`The Descent (2005) - SD.mkv`) and were 8–20-second h264 video at
  640×360, 1024×576 or 1280×720, 49 MiB all together —
  25 of them in a `_release-group watermarks/` folder and the rest at the library
  root or inside one film's folder. No duplicate screen ever catches them: nothing
  shares their size, and the feature-length guard above makes them invisible to the
  similar-title pass too. They were held as `.AGENTTRASH/20261104-promo-clips`
  (journal `/mnt/entropy/.media-campaign/promo-hold.jsonl`, which survives) and purged
  the same evening, once both databases read `REAL: 0` against that journal.
  The census that finds the next batch is a duration floor, not a hash: a video under
  a minute on a name that reads like a feature is advertising.

## The hold

Nothing is deleted. Every doomed copy moves into a dated hold with the same
executor the relayout uses, so any file comes back:

```bash
scripts/media-apply.py apply   --plan /mnt/entropy/.media-campaign/dup-prune-similar-plan.tsv \
    --root /mnt/entropy --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl          # dry
scripts/media-apply.py apply   --plan … --root /mnt/entropy --journal … --apply
scripts/media-apply.py verify  --plan … --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl
scripts/media-apply.py sweep   --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
scripts/media-apply.py rollback --journal /mnt/entropy/.media-campaign/dup-prune-similar.jsonl --root /mnt/entropy --apply
```

The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
every hold parent), and the planner re-checks the two things that go stale between
analysis and apply: a file that has moved since the census, and a group whose
keeper is missing — either one drops the whole group rather than half-applying it.

- **A hold on the same volume frees nothing.** The purge is a separate, approved
  act and the only thing that moves the free-space number; `df` does not budge
  when 1 TiB moves sideways.
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
  ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
  era, not this pass — read the mtimes before blaming the prune.
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
  `/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
  `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`.
- The 2026-11-02 TV pass held 220 files (54 GiB) in
  `/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates` — plan `tv-dupes-plan.tsv`,
  journal `tv-dupes.jsonl`, 220 moves / 16 hold folders / 7 husks swept — and the
  same day's TVShows detritus pass staged 109 files (552 KiB) in
  `/mnt/entropy/.AGENTTRASH/20261102-tvshows-detritus` with plan
  `tvshows-detritus-plan.tsv` and journal `tvshows-detritus.jsonl`. All four
  journals and plans live at the `entropy` root, not `/tmp`.
- **Purged 2026-11-04, on Frank's word, after the gate below:** the duplicate
  prune, the TV duplicate pass, the Ghibli duplicates (6 files — both `Nausicaä`
  copies among them) and the 2026-11-04 promo-clip hold left the array trash,
  ~1.2 TB freed (the array went 218 GB → 1.4 TB available). The journals survive as
  the only record of where a purged file came from; rollback after a purge is
  impossible, which is why the gate runs first.

### The purge gate: a carrier for every held file

`REAL: 0` in both databases says only that no row points at a moved path; it says
nothing about whether the film survived. So a purge waits on a stronger proof:
**every held file has a same-work carrier in the live library at >= its size,
re-derived at purge time**, never copied from the plan. The 2026-11-04 purge re-found
all 452 held files this way; the first pass of the gate could not place 142 of them,
and every one of those was a matcher gap, not a lost film. What the re-derivation has
to absorb:

- **The layout moves under the plan.** The relayout renames folders to
  `Title (YYYY)` and `[tvdbid-N]/Season N` and renames the keeper files too, so the
  paths *and* the sizes a plan recorded no longer describe the library. Keys are work
  identity — normalized title+year for a film, `(S,E)` for a TV pack — never a path.
- **Fold accents, then compare.** The library holds `Nausicaä`; a plan, a
  `find -ipath "*Nausicaa*"`, and an ASCII substring search all miss it. NFKD-fold and
  drop combining marks, or a real keeper reads absent for two rounds and the file
  beside it starts to look like an orphan.
- **The carrier is often another container, and usually larger.** A held `.mp4` can be
  kept as the sibling `.mkv`; keep-larger means the keeper is frequently far bigger (a
  377 MB held episode matched a 501 MB keeper; a 1.6 GiB film stands beside a 34 GiB
  BDRemux of itself in another folder). "Same folder, same bytes" reports survivors as
  lost — it is the wrong test.
- **An exact twin can sit in a sibling folder**, so "no same-size file in the folder"
  needs an array-wide name search before it means anything.

### Running the purge

`scripts/purge-agenttrash.sh` is the only door — never `rm`. It acts on exactly one
object: a root's own `.AGENTTRASH`. A partial purge is therefore **staged, not
filtered**: move the approved buckets into a scratch root
(`.PURGE-<date>/.AGENTTRASH/`) and point the purger at that root, and park anything
not approved at the array root as `.HOLD-<date>-*`, out of its reach. Dry-run first —
the plan must print exactly one `plan empty` row, and it is the same row `--execute`
acts on — and send `--log` into `.media-campaign/`, because `/tmp` is not a record.
Then read `df`: a purge is the only media act that moves the free-space number.

## After the pass

Both servers still hold rows for files that are now in the hold, and neither
notices: the media bind is the [[Media Metadata View]], and array changes do not
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
are owned by [[Jellyfin]] and [[Plex]].

The Plex leg may be deferred on purpose: stop **and mask** Plex (`systemctl --user
stop plex && systemctl --user mask plex`) so its metadata fetches cannot stack on a
live Jellyfin scan, and because a masked unit stays stopped through a reboot —
`unmask` before starting it. The 2026-11-01 morning ran it once Jellyfin reached
`Idle`: one accepted `refresh` on the Movies section cleared every row that named a
held path. The 2026-11-02 TV pass did the same on the TV Shows section (key 3): one
accepted refresh, then `media-verify.py` against that journal reads `landed: 0`,
`still at src: 0`, `REAL: 0` on **both** servers — the correct end state for a prune.

The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
from folders whose whole subtree held no media, so it creates no stale rows and
needs no scan. The guards that hold a folder back are the finding: external
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
extension the census does not recognise is a reason to stop, not to sweep.

A folder holding a `.plexmatch` is held back even when its subtree looks empty: that
file is a manual match — a human decision about what the folder is — and detritus
rules do not overrule one.

`scripts/media-verify.py` reads differently for a prune than for a relayout:
`landed: 0` is the **correct** end state (no server may index the hold), so the
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
from the database are the pass succeeding.

One residue is normal, so do not chase it: a section pass rewrites the feature rows
first, and rows for clips in a film's `other/` or `extras/` subfolder can still name
a held path after the section's `refreshing` flag is back to `0` — and a second
section pass does not clear them either, so do not run one hoping for `0`. Beside
`REAL: 0` that is a dead disc-menu clip — not a false prune, and no reason to touch
the cache, the database, or the hold.

## Failure signatures

| Symptom | Meaning |
|---|---|
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |

## Links

- [[Media Relayout]] — the census tools, the executor, the journal format.
- [[Media Metadata View]] — why neither server sees the change without a scan.
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
- [[Entropy Volume]] — the array that is always full.