serverAppConfigs/serverAppConfigs-docs/Dankserver/Media Duplicate Prune.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Media Duplicate Prune	Find and hold duplicate films on the entropy array: the hash ladder, the keep rules, the guards, the purge.	dankserver media-server storage procedure	2026-11-01T01:00:00Z

Media Duplicate Prune

The movie library holds the same film twice in two different ways — the same file downloaded again, or the same film grabbed a second time at another quality — and Entropy Volume is permanently full, so both are worth finding. They need different rules, and neither is decided by name similarity alone.

/mnt/entropy/media-park was a third kind: a parking lot that held whole shows and films nothing served. It is gone; its contents went to NeedsProcessing.

What counts

Class	Test	Action
Byte-identical	equal size, equal content hash	keep one copy, hold the rest
Same work, worse encode	same title, same year, every runtime within 5 %	keep the largest — Frank's rule
Different cut	same title and year, runtimes more than 5 % apart	never touched: an extended and a theatrical version are different products
Same words, different film	title matches, year or duration does not	never grouped — this is the false-merge class that would destroy a unique file

The screen

The array cannot afford a full-content read, so identity is established in steps and each step only narrows what the next one touches:

1. scripts/media-census.py enumerate — size is free, and identical files share it.
2. Head+tail blake2b, 4 MiB at each end, on the size collisions only.
3. Before anything moves, confirm with a sparse-window hash: 32 MiB at eight evenly spaced offsets. Re-hashing a remux-heavy set in full read 150 GiB for the better part of an hour; the sparse window cost 50 s, a file that merely shares its head and tail cannot survive it, and the destination is a recoverable hold rather than a deletion.
4. Durations come from scripts/media-census.py probe — the relayout's own census. Do not hand-roll a second probe: it re-seeks the array for an answer the repo tool already produces, and a probe is the one census step that costs hours instead of seconds.

Which copy survives

In order: not one inside an extras, featurettes, samples or trailers folder (an extra is not a copy of the film, and extra names collide across films); then the shallowest path; then the file whose name equals its Title (year) folder; then the largest. For the same-work class the largest is the answer, and the canonical-name rule only breaks a tie — without it a Film - 1080p.mkv can outlive Film.mkv for no reason but a space sorting before a dot.

Guards

Each of these is a file that would have been destroyed without it:

• .!qB is qBittorrent state, not a duplicate. It can be byte-identical to the finished file while qB still counts the download incomplete; moving it breaks the re-check and re-downloads the film. Report it, leave it. Both leftovers in Movies are 13 months stale, so they are Frank's purge decision, not a prune decision.
• Require a year, and keep digits in the title. Stripping digits merged Akira — Teaser 1 with Teaser 2; without a year, two different extras that happened to share a name merged.
• Both members must be feature length, and a name carrying disc, cd or part is excluded: half a multi-disc release is not a copy of the other half.
• Media containers only. Grouping every extension buries the report in subtitle and cover pairs — and a duration guard that appears to fire on 720 groups is counting subtitles, not protecting films.

The hold

Nothing is deleted. Every doomed copy moves into a dated hold with the same executor the relayout uses, so any file comes back:

scripts/media-apply.py apply   --plan /mnt/entropy/dup-prune-similar-plan.tsv \
    --root /mnt/entropy --journal /mnt/entropy/dup-prune-similar.jsonl          # dry
scripts/media-apply.py apply   --plan … --root /mnt/entropy --journal … --apply
scripts/media-apply.py verify  --plan … --journal /mnt/entropy/dup-prune-similar.jsonl
scripts/media-apply.py sweep   --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply
scripts/media-apply.py rollback --journal /mnt/entropy/dup-prune-similar.jsonl --root /mnt/entropy --apply

The plan rows are the tool's own format (action src dst kind, mkdir rows for every hold parent), and the planner re-checks the two things that go stale between analysis and apply: a file that has moved since the census, and a group whose keeper is missing — either one drops the whole group rather than half-applying it.

• A hold on the same volume frees nothing. The purge is a separate, approved act and the only thing that moves the free-space number; df does not budge when 1 TiB moves sideways.
• sweep removes the release folders the pass emptied. Empty folders dated a year ago are torrent detritus (Subtitle,info, Screens, Sample) from a previous era, not this pass — read the mtimes before blaming the prune.
• The 2026-11-01 pass held 226 files (~1.1 TiB) in /mnt/entropy/.AGENTTRASH/20261101-duplicate-prune, journals dup-prune-exact.jsonl and dup-prune-similar.jsonl. Purge pending Frank's word.
• Re-verify each survivor by size, not by the recorded path, before purging. Detection artifacts and plans key on paths, and a folder renamed by an interactive session (Entropy Volume, Write doors) makes its keeper look deleted while it sits untouched under the new name. The 2026-11-01 audit hit three such groups: every keeper was present, same bytes, different folder. A purge decided from the plan's paths alone would have reported survivors gone.

After the pass

Both servers still hold rows for files that are now in the hold, and neither notices: the media bind is the Media Metadata View, and array changes do not cross it as inotify events. Scan in the order the rescan law sets — Jellyfin to Idle, then Plex's Movies section alone, one accepted refresh each; the requests are owned by Jellyfin and Plex.

The Plex leg may be deferred on purpose: stop and mask Plex (systemctl --user stop plex && systemctl --user mask plex) so its metadata fetches cannot stack on a live Jellyfin scan, and because a masked unit stays stopped through a reboot — unmask before starting it. The 2026-11-01 morning ran it once Jellyfin reached Idle: one accepted refresh on the Movies section cleared every row that named a held path.

The same evening's detritus pass is a second, separate hold: 423 files (12 MiB) in /mnt/entropy/.AGENTTRASH/20261101-detritus-sweep, plan detritus-sweep-plan.tsv, journal detritus-sweep.jsonl, 542 husks removed by sweep. It moved files only from folders whose whole subtree held no media, so it creates no stale rows and needs no scan. The guards that hold a folder back are the finding: external subtitle folders (428 of them, 6 GiB) and ripped Blu-ray streams — .sup PGS, .thd/.eac3/.dts audio, .cci captions, 25 GiB — are not trash, and an extension the census does not recognise is a reason to stop, not to sweep.

scripts/media-verify.py reads differently for a prune than for a relayout: landed: 0 is the correct end state (no server may index the hold), so the verdict is still at src: 0 with REAL: 0, and the rows that simply vanished from the database are the pass succeeding.

One residue is normal, so do not chase it: a section pass rewrites the feature rows first, and rows for clips in a film's other/ or extras/ subfolder can still name a held path after the section's refreshing flag is back to 0 — and a second section pass does not clear them either, so do not run one hoping for 0. Beside REAL: 0 that is a dead disc-menu clip — not a false prune, and no reason to touch the cache, the database, or the hold.

Failure signatures

Symptom	Meaning
A size group whose members split under the sparse hash	The head+tail screen collided. Those files stay exactly where they are and get reported.
The planner holds far fewer files than the analyzer counted	Two implementations of "largest wins". The analyzer must emit keep/hold marks and the planner only transcribes them — a planner that re-derives the winner from the hold list alone drops the keeper out of the group and prunes almost nothing.
df unchanged after a TiB of moves	Expected: the hold is on the same filesystem. Only the purge frees space.
A film that "disappeared" after a prune	It is in the hold, named by the journal: media-apply.py rollback --journal … --apply.

Links

• Media Relayout — the census tools, the executor, the journal format.
• Media Metadata View — why neither server sees the change without a scan.
• Jellyfin · Plex — the scan requests, one server at a time.
• Entropy Volume — the array that is always full.