68 lines
4.1 KiB
Markdown
68 lines
4.1 KiB
Markdown
---
|
|
name: media-library-dup-similarity-census
|
|
description: "Census a large media library for byte duplicates and same-film/different-encode copies with codecs, over SSH, without disturbing a live inference service."
|
|
---
|
|
|
|
# Media library duplicate + similarity census
|
|
|
|
Read-only pipeline for `find duplicates and similars in /mnt/... with sizes and codecs`
|
|
on a multi-TiB spindle library. Everything `nice -n 19`, one owner per stage, results as TSV.
|
|
|
|
## Stages (never skip the cheap ones)
|
|
|
|
1. **Enumerate** → TSV `size \t ext \t path`. Skip `.AGENTTRASH`, `@eaDir`, `.thumbnails`.
|
|
Report totals per extension + check `find -type f -links +2` (hardlinks are a separate
|
|
duplicate class).
|
|
2. **Size groups** → candidates for byte-identity. Seconds.
|
|
3. **Partial hash**: blake2b of first 4 MiB + last 4 MiB. Kills size collisions cheaply.
|
|
4. **Full blake2b** on the survivors only. Confirm, don't assume — report how many
|
|
partial-groups split on the full hash (0 = the head+tail screen was exact).
|
|
5. **ffprobe census** (codec, WxH, pix_fmt, duration, audio codec list) for every video
|
|
container. Parallel 6 workers; expect ~11-21 files/s on a spindle array with contention.
|
|
6. **Title/year/duration clustering** for "same film, different encode".
|
|
|
|
## Similarity engine: the rules that survive contact with real data
|
|
|
|
- **Runtime vetoes names.** Films titled `The Menu`, `Trailer`, `Trailer Park Boys`,
|
|
`ET — The Extra-Terrestrial` run 100+ min → not bonus material. Any file over ~45 min
|
|
is a film whatever it is called or wherever it sits.
|
|
- **Long-shaped bonus words** (commentary, documentary, interview) believed only under
|
|
~25 min; a feature-length commentary cut is a film.
|
|
- **Bonus folder evidence is path-wide below the film root** — `Film/Extras/x.mp4`,
|
|
`Film/Featurettes/Deleted Scenes/x.mkv`, `Film/04b. Film Extras/Logo.mp4` are all bonus.
|
|
The ROOT folder is exempt: it names the film. Match whole words only, so
|
|
`Extraordinary` / `The Extras` are not evidence.
|
|
- **A file named for the folder's film name is that film**, not its bonus — otherwise a
|
|
whole movie filed in another film's `Extras/` disappears from the film census.
|
|
- **Cluster by duration inside a title bucket**, not by title alone: 108 min vs 121 min
|
|
is a different cut, and franchise prefixes ("jump street") collide without a duration check.
|
|
- **Exclude release-group watermark clips** (`ETRG.mp4`, `RARBG.com.mp4`, `TGx.mp4`,
|
|
<20 MiB): they pair with themselves across unrelated films.
|
|
- **Exclude samples/trailers/menus/galleries** — generic names create cross-film pairs.
|
|
- BDMV trees: one pseudo-entry per tree, described by its longest stream (`BDMV/index.bdmv`
|
|
ends in `.bdmv`, so a `VIDEO_SUFFIX` check catches it); size = whole tree.
|
|
- AppleDouble `._*` files are not media.
|
|
|
|
## Operational traps
|
|
|
|
- `python3 enum.py` shadows the stdlib `enum` module — never name a script a stdlib name.
|
|
- Detached remote jobs: `setsid nice -n 19 ... > log 2>&1 < /dev/null &`. Killing the SSH
|
|
client does NOT kill the remote child — two hashers raced on one output file and truncated
|
|
it to 0 bytes. `pgrep -af` on the remote before relaunching, and kill the orphan by pid.
|
|
- A background bash job with a default deadline can die mid-read; for long reads use
|
|
`timeout: 0` or run detached and poll the log.
|
|
- Run probes and hashers **sequentially**, not in parallel — they contend on one array and
|
|
each starves the other.
|
|
- Ship scripts with `cat local | ssh host 'cat > /tmp/x.py'`; a filename collision with the
|
|
remote's cwd is invisible until it crashes.
|
|
- `/tmp/mc/` on Dankserver can vanish mid-session (systemd tmpfiles); pull TSVs local early.
|
|
- Quote remote awk/heredocs carefully — nested quoting fails before it ever runs.
|
|
|
|
## Reporting
|
|
|
|
Give: per-extension library totals; exact-dup table (size, codec, path); top same-film groups
|
|
by reclaim bytes with per-file size + codec + runtime; a reclaim table that states whether the
|
|
classes overlap; and the noise classes you excluded (extras, watermarks, `.!qB` partials,
|
|
TV episodes filed as movies). Audit your own duplicate tier: flag groups whose year/base-name/
|
|
runtime disagree and read every flagged group by hand before reporting the total.
|