serverAppConfigs/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md

---
type: reference
title: Media Duplicate Prune
description: "Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge."
tags: [dankserver, media-server, storage, procedure]
timestamp: 2026-11-01T01:00:00Z
---

# Media Duplicate Prune

The movie library holds the same film twice in two different ways — the same file
downloaded again, or the same film grabbed a second time at another quality — and
[[Entropy Volume]] is permanently full, so both are worth finding. They need
different rules, and neither is decided by name similarity alone.

`/mnt/entropy/media-park` was a third kind: a parking lot that held whole shows
and films nothing served. It is gone; its contents went to `NeedsProcessing`.

## What counts

| Class | Test | Action |
|---|---|---|
| Byte-identical | equal size, equal content hash | keep one copy, hold the rest |
| Same work, worse encode | same title, same year, every runtime within 5 % | keep the largest — Frank's rule |
| Different cut | same title and year, runtimes more than 5 % apart | **never touched**: an extended and a theatrical version are different products |
| Same words, different film | title matches, year or duration does not | never grouped — this is the false-merge class that would destroy a unique file |

## The screen

The array cannot afford a full-content read, so identity is established in steps
and each step only narrows what the next one touches:

1. `scripts/media-census.py enumerate` — size is free, and identical files share it.
2. Head+tail `blake2b`, 4 MiB at each end, on the size collisions only.
3. **Before anything moves**, confirm with a sparse-window hash: 32 MiB at eight
   evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for
   the better part of an hour; the sparse window cost 50 s, a file that merely
   shares its head and tail cannot survive it, and the destination is a
   recoverable hold rather than a deletion.
4. Durations come from `scripts/media-census.py probe` — the relayout's own
   census. Do not hand-roll a second probe: it re-seeks the array for an answer
   the repo tool already produces, and a probe is the one census step that costs
   hours instead of seconds.

## Which copy survives

In order: not one inside an `extras`, `featurettes`, `samples` or `trailers`
folder (an extra is not a copy of the film, and extra names collide across films);
then the shallowest path; then the file whose name equals its `Title (year)`
folder; then the largest. For the same-work class the largest is the answer, and
the canonical-name rule only breaks a tie — without it a `Film - 1080p.mkv` can
outlive `Film.mkv` for no reason but a space sorting before a dot.

## Guards

Each of these is a file that would have been destroyed without it:

- **`.!qB` is qBittorrent state, not a duplicate.** It can be byte-identical to
  the finished file while qB still counts the download incomplete; moving it
  breaks the re-check and re-downloads the film. Report it, leave it. Both
  leftovers in `Movies` are 13 months stale, so they are Frank's purge decision,
  not a prune decision.
- **Require a year, and keep digits in the title.** Stripping digits merged
  *Akira — Teaser 1* with *Teaser 2*; without a year, two different extras that
  happened to share a name merged.
- Both members must be feature length, and a name carrying `disc`, `cd` or `part`
  is excluded: half a multi-disc release is not a copy of the other half.
- Media containers only. Grouping every extension buries the report in subtitle
  and cover pairs — and a duration guard that appears to fire on 720 groups is
  counting subtitles, not protecting films.

## The hold

Nothing is deleted. Every doomed copy moves into a dated hold with the same
executor the relayout uses, so any file comes back:

```bash
scripts/media-apply.py apply   --plan /mnt/entropy/dup-prune-similar-plan.tsv \
    --root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl          # dry
scripts/media-apply.py apply   --plan … --root /mnt/entropy --journal … --apply
scripts/media-apply.py verify  --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
scripts/media-apply.py sweep   --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
```

The plan rows are the tool's own format (`action src dst kind`, `mkdir` rows for
every hold parent), and the planner re-checks the two things that go stale between
analysis and apply: a file that has moved since the census, and a group whose
keeper is missing — either one drops the whole group rather than half-applying it.

- **A hold on the same volume frees nothing.** The purge is a separate, approved
  act and the only thing that moves the free-space number; `df` does not budge
  when 1 TiB moves sideways.
- `sweep` removes the release folders the pass emptied. Empty folders dated a year
  ago are torrent detritus (`Subtitle,info`, `Screens`, `Sample`) from a previous
  era, not this pass — read the mtimes before blaming the prune.
- The 2026-11-01 pass held 226 files (~1.1 TiB) in
  `/mnt/entropy/.AGENTTRASH/20261101-duplicate-prune`, journals
  `dup-prune-exact.jsonl` and `dup-prune-similar.jsonl`. Purge pending Frank's word.

## After the pass

Both servers still hold rows for files that are now in the hold, and neither
notices: the media bind is the [[Media Metadata View]], and array changes do not
cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to
`Idle`, then Plex's Movies section alone, one accepted refresh each; the requests
are owned by [[Jellyfin]] and [[Plex]].

As of 2026-11-01 the Plex leg is deliberately deferred: Plex is stopped and
**masked** (`systemctl --user stop plex && systemctl --user mask plex`) so its
metadata fetches cannot stack on Jellyfin's live scan, and a masked unit stays
stopped through a reboot — `unmask` before starting it. Its Movies section (key 1)
holds ~187 rows naming held paths until that one refresh is accepted.

The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in
`/mnt/entropy/.AGENTTRASH/20261101-detritus-sweep`, plan `detritus-sweep-plan.tsv`,
journal `detritus-sweep.jsonl`, 542 husks removed by `sweep`. It moved files only
from folders whose whole subtree held no media, so it creates no stale rows and
needs no scan. The guards that hold a folder back are the finding: external
subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — `.sup` PGS,
`.thd`/`.eac3`/`.dts` audio, `.cci` captions, 25 GiB — are not trash, and an
extension the census does not recognise is a reason to stop, not to sweep.

`scripts/media-verify.py` reads differently for a prune than for a relayout:
`landed: 0` is the **correct** end state (no server may index the hold), so the
verdict is `still at src: 0` with `REAL: 0`, and the rows that simply vanished
from the database are the pass succeeding.

## Failure signatures

| Symptom | Meaning |
|---|---|
| A size group whose members split under the sparse hash | The head+tail screen collided. Those files stay exactly where they are and get reported. |
| The planner holds far fewer files than the analyzer counted | Two implementations of "largest wins". The analyzer must emit `keep`/`hold` marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing. |
| `df` unchanged after a TiB of moves | Expected: the hold is on the same filesystem. Only the purge frees space. |
| A film that "disappeared" after a prune | It is in the hold, named by the journal: `media-apply.py rollback --journal … --apply`. |

## Links

- [[Media Relayout]] — the census tools, the executor, the journal format.
- [[Media Metadata View]] — why neither server sees the change without a scan.
- [[Jellyfin]] · [[Plex]] — the scan requests, one server at a time.
- [[Entropy Volume]] — the array that is always full.