command-center/agent/managed-skills/media-library-dup-similarity-census/SKILL.md

name, description

name	description
media-library-dup-similarity-census	Census a large media library for byte duplicates and same-film/different-encode copies with codecs, over SSH, without disturbing a live inference service.

Media library duplicate + similarity census

Read-only pipeline for find duplicates and similars in /mnt/... with sizes and codecs on a multi-TiB spindle library. Everything nice -n 19, one owner per stage, results as TSV.

Stages (never skip the cheap ones)

1. Enumerate → TSV size \t ext \t path. Skip .AGENTTRASH, @eaDir, .thumbnails. Report totals per extension + check find -type f -links +2 (hardlinks are a separate duplicate class).
2. Size groups → candidates for byte-identity. Seconds.
3. Partial hash: blake2b of first 4 MiB + last 4 MiB. Kills size collisions cheaply.
4. Full blake2b on the survivors only. Confirm, don't assume — report how many partial-groups split on the full hash (0 = the head+tail screen was exact).
5. ffprobe census (codec, WxH, pix_fmt, duration, audio codec list) for every video container. Parallel 6 workers; expect ~11-21 files/s on a spindle array with contention.
6. Title/year/duration clustering for "same film, different encode".

Similarity engine: the rules that survive contact with real data

• Runtime vetoes names. Films titled The Menu, Trailer, Trailer Park Boys, ET — The Extra-Terrestrial run 100+ min → not bonus material. Any file over ~45 min is a film whatever it is called or wherever it sits.
• Long-shaped bonus words (commentary, documentary, interview) believed only under ~25 min; a feature-length commentary cut is a film.
• Bonus folder evidence is path-wide below the film root — Film/Extras/x.mp4, Film/Featurettes/Deleted Scenes/x.mkv, Film/04b. Film Extras/Logo.mp4 are all bonus. The ROOT folder is exempt: it names the film. Match whole words only, so Extraordinary / The Extras are not evidence.
• A file named for the folder's film name is that film, not its bonus — otherwise a whole movie filed in another film's Extras/ disappears from the film census.
• Cluster by duration inside a title bucket, not by title alone: 108 min vs 121 min is a different cut, and franchise prefixes ("jump street") collide without a duration check.
• Exclude release-group watermark clips (ETRG.mp4, RARBG.com.mp4, TGx.mp4, <20 MiB): they pair with themselves across unrelated films.
• Exclude samples/trailers/menus/galleries — generic names create cross-film pairs.
• BDMV trees: one pseudo-entry per tree, described by its longest stream (BDMV/index.bdmv ends in .bdmv, so a VIDEO_SUFFIX check catches it); size = whole tree.
• AppleDouble ._* files are not media.

Operational traps

• python3 enum.py shadows the stdlib enum module — never name a script a stdlib name.
• Detached remote jobs: setsid nice -n 19 ... > log 2>&1 < /dev/null &. Killing the SSH client does NOT kill the remote child — two hashers raced on one output file and truncated it to 0 bytes. pgrep -af on the remote before relaunching, and kill the orphan by pid.
• A background bash job with a default deadline can die mid-read; for long reads use timeout: 0 or run detached and poll the log.
• Run probes and hashers sequentially, not in parallel — they contend on one array and each starves the other.
• Ship scripts with cat local | ssh host 'cat > /tmp/x.py'; a filename collision with the remote's cwd is invisible until it crashes.
• /tmp/mc/ on Dankserver can vanish mid-session (systemd tmpfiles); pull TSVs local early.
• Quote remote awk/heredocs carefully — nested quoting fails before it ever runs.

Reporting

Give: per-extension library totals; exact-dup table (size, codec, path); top same-film groups by reclaim bytes with per-file size + codec + runtime; a reclaim table that states whether the classes overlap; and the noise classes you excluded (extras, watermarks, .!qB partials, TV episodes filed as movies). Audit your own duplicate tier: flag groups whose year/base-name/ runtime disagree and read every flagged group by hand before reporting the total.