---
name: media-library-dup-similarity-census
description: "Census a large media library for byte duplicates and same-film/different-encode copies with codecs, over SSH, without disturbing a live inference service."
---

# Media library duplicate + similarity census

Read-only pipeline for `find duplicates and similars in /mnt/... with sizes and codecs`
on a multi-TiB spindle library. Everything `nice -n 19`, one owner per stage, results as TSV.

## Stages (never skip the cheap ones)

1. **Enumerate** → TSV `size \t ext \t path`. Skip `.AGENTTRASH`, `@eaDir`, `.thumbnails`.
   Report totals per extension + check `find -type f -links +2` (hardlinks are a separate
   duplicate class).
2. **Size groups** → candidates for byte-identity. Seconds.
3. **Partial hash**: blake2b of first 4 MiB + last 4 MiB. Kills size collisions cheaply.
4. **Full blake2b** on the survivors only. Confirm, don't assume — report how many
   partial-groups split on the full hash (0 = the head+tail screen was exact).
5. **ffprobe census** (codec, WxH, pix_fmt, duration, audio codec list) for every video
   container. Parallel 6 workers; expect ~11-21 files/s on a spindle array with contention.
6. **Title/year/duration clustering** for "same film, different encode".

## Similarity engine: the rules that survive contact with real data

- **Runtime vetoes names.** Films titled `The Menu`, `Trailer`, `Trailer Park Boys`,
  `ET — The Extra-Terrestrial` run 100+ min → not bonus material. Any file over ~45 min
  is a film whatever it is called or wherever it sits.
- **Long-shaped bonus words** (commentary, documentary, interview) believed only under
  ~25 min; a feature-length commentary cut is a film.
- **Bonus folder evidence is path-wide below the film root** — `Film/Extras/x.mp4`,
  `Film/Featurettes/Deleted Scenes/x.mkv`, `Film/04b. Film Extras/Logo.mp4` are all bonus.
  The ROOT folder is exempt: it names the film. Match whole words only, so
  `Extraordinary` / `The Extras` are not evidence.
- **A file named for the folder's film name is that film**, not its bonus — otherwise a
  whole movie filed in another film's `Extras/` disappears from the film census.
- **Cluster by duration inside a title bucket**, not by title alone: 108 min vs 121 min
  is a different cut, and franchise prefixes ("jump street") collide without a duration check.
- **Exclude release-group watermark clips** (`ETRG.mp4`, `RARBG.com.mp4`, `TGx.mp4`,
  <20 MiB): they pair with themselves across unrelated films.
- **Exclude samples/trailers/menus/galleries** — generic names create cross-film pairs.
- BDMV trees: one pseudo-entry per tree, described by its longest stream (`BDMV/index.bdmv`
  ends in `.bdmv`, so a `VIDEO_SUFFIX` check catches it); size = whole tree.
- AppleDouble `._*` files are not media.

## Operational traps

- `python3 enum.py` shadows the stdlib `enum` module — never name a script a stdlib name.
- Detached remote jobs: `setsid nice -n 19 ... > log 2>&1 < /dev/null &`. Killing the SSH
  client does NOT kill the remote child — two hashers raced on one output file and truncated
  it to 0 bytes. `pgrep -af` on the remote before relaunching, and kill the orphan by pid.
- A background bash job with a default deadline can die mid-read; for long reads use
  `timeout: 0` or run detached and poll the log.
- Run probes and hashers **sequentially**, not in parallel — they contend on one array and
  each starves the other.
- Ship scripts with `cat local | ssh host 'cat > /tmp/x.py'`; a filename collision with the
  remote's cwd is invisible until it crashes.
- `/tmp/mc/` on Dankserver can vanish mid-session (systemd tmpfiles); pull TSVs local early.
- Quote remote awk/heredocs carefully — nested quoting fails before it ever runs.

## Reporting

Give: per-extension library totals; exact-dup table (size, codec, path); top same-film groups
by reclaim bytes with per-file size + codec + runtime; a reclaim table that states whether the
classes overlap; and the noise classes you excluded (extras, watermarks, `.!qB` partials,
TV episodes filed as movies). Audit your own duplicate tier: flag groups whose year/base-name/
runtime disagree and read every flagged group by hand before reporting the total.