Compare commits

...
Author SHA1 Message Date
Frank e83162a2ac feat(media): a TV mover that will not file a show the library already has
Two fixes found by one live sweep, shipped together because the second is
what makes the first finishable.

1. scripts/media-tv.py — the other half of media-plan.py's refusal to file
   TV. Groups episode-shaped files by the series name before the marker
   (media-titles.ep_of, imported not restated), writes a plan in the
   executor's own format: <library>/<Series> (<year>)/Season <NN>/<own
   name>.<ext>. Refuses a group below --min, an episode naming no season,
   and an episode the destination library already holds at the same byte
   count — the movie share carries a second copy of all 124 Monk episodes
   the TV library has had for years; a duplicate is Frank's call, never a
   tool's copy. Applied by media-apply.py with its own journal and
   --root /mnt/entropy.

2. scripts/media-apply.py — gone meant "rmdir intended", so a folder whose
   rmdir failed was blacklisted forever and no later sweep could finish it
   (live movie sweep: 4517 targeted, 146 permanently stranded). gone now
   means "actually gone": a failed intent is retried by the next sweep.

tests: 9 new tests in media_relayout.bats, each red against the previous
revision — 8 TV + 1 sweep retry behind a chmod 500 parent. ./tests/run.sh
633 green. No live move run yet.
2026-10-30 20:30:24 -07:00
dank 5cd9a0c4c2 Merge pull request 'fix(media): the series is the name before the marker; a sweep carries its own empty subfolders' (#95) from fix/media-series-key into main 2026-10-30 20:20:24 -07:00
Frank a803aa046a fix(media): a sweep carries the emptied folder's own empty subfolders
The live sweep removed 2271 husks and left 146 standing, every one a release
folder blocked by a junk subfolder (Screens/, Sample/, a BDMV leaf) that no
file ever moved out of. Eligibility was per-folder, so the blocker was never a
target and its parent could never be removed.

Proof now carries downward: inside a folder the journal shows emptied, every
empty directory qualifies. Nothing under such a folder can hold data, and every
removal is still journalled as rmdir and undone by rollback.

The new test is red against the previous revision, green now. The live sweep
touched no file: 8193 before, 8193 after.
2026-10-30 20:18:54 -07:00
Frank 73f77a13c3 fix(media): a series is the name before the episode marker
Keyed on the whole episode title, every episode of a box set became a group of
one and never reached the five-episode size that calls something a series, so
the live run filed 124 Monk episodes as 124 movie folders, each named after its
own episode. ep_of now returns the series — the name before SxxEyy / 1x04 /
Part N — and the planner groups and names the quarantine pile by that series,
so one show is one pile: 'Monk - S05E03 - Mr. Monk and the Big Game' is series
monk. The threshold keeps its job: a lone film named Part 2 stays a film.

Re-planning the live census with this moves nothing differently except the 124
Monk files, which now join the TV pile: 168 episode files across 3 series.
2026-10-30 20:12:05 -07:00
dank dd76fcf5d4 Merge pull request 'feat(media): sweep removes the folders a completed run emptied' (#94) from feat/media-sweep into main 2026-10-30 20:04:24 -07:00
Frank 1cb28095a7 feat(media): sweep removes the folders a completed run emptied
An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
2026-10-30 20:03:16 -07:00
dank e1760e3316 Merge pull request 'feat(media): the movie-library relayout toolchain — census, plan, executor' (#93) from feat/media-relayout into main 2026-10-30 19:38:03 -07:00
dank 272e06daf6 feat(media): the movie-library relayout toolchain — census, plan, executor
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
2026-10-30 19:33:44 -07:00
dank 4db985b90d Merge pull request 'fix(qwen38): the preflight asks whether the pin is ONE of an image's digests' (#92) from fix/quant-pipeline-hardening into main 2026-10-30 16:30:07 -07:00
dank 8073d0aa9d Merge pull request 'fix(qwen38): one identity preflight owns every fact a calibration window needs' (#91) from fix/quant-pipeline-hardening into main
Merge pull request #91
2026-10-30 16:26:41 -07:00
dank 81e0995f7d Merge pull request 'fix(qwen38): the calibration window must prove the rig owns the socket' (#90) from fix/quant-pipeline-hardening into main 2026-10-30 16:08:29 -07:00
dank 608b0058e3 Merge pull request 'campaign(qwen38): the calibration floor is 12M tokens' (#89) from fix/quant-pipeline-hardening into main 2026-10-30 15:47:25 -07:00
9 changed files with 3215 additions and 0 deletions

No files matched your search

+2
View File
@@ -807,6 +807,7 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -848,5 +849,6 @@ history.
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
+425
View File
@@ -0,0 +1,425 @@
#!/usr/bin/env python3
"""media-apply.py — the only thing that moves movie-library files.
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
an interrupted run has destroyed nothing. Every run is resumable and every run
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
BEFORE the rename it authorises. That ordering is the whole design: a crash
leaves either an intent whose move happened or an intent whose move never ran,
and both are recognised on the next run. A file that moved with no journal line
cannot have been moved by this tool, and saying so is better than guessing.
Four subcommands:
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
no --apply => report only and write NOTHING, not even folders
rollback --journal J [--root ROOT] [--apply]
back to the paths the census saw; folders this run created are
removed only when empty, never anything else
verify --plan PLAN --journal J [--root ROOT]
every planned file is at its target with the journalled byte count
sweep --journal J [--root ROOT] [--apply]
removes the folders THIS RUN emptied — the journal has to show a
file that moved out of one, so a folder someone made by hand can
never be removed by this tool, and rollback puts them back
Refusals (exit 2, before the first rename, with nothing touched): a target
already occupied by a foreign file, a source that has gone, a duplicate target,
a path outside --root, a journal that was already rolled back, a move that would
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
--force relaxes only the guards that are opinion (journal already rolled back,
paths outside --root); it never allows overwriting a file.
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
inside it: a log inside the library is a file Jellyfin will index.
"""
import argparse
import datetime
import json
import os
import sys
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
class Refuse(Exception):
"""A gate the operator has to see; every refusal leaves the library alone."""
def journal_default(root):
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
class Journal:
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
def __init__(self, path, writing):
self.path, self.writing = path, writing
self.records = []
if os.path.exists(path):
for line in open(path):
line = line.strip()
if line:
self.records.append(json.loads(line))
self._fh = None
def already(self, dst, src):
"""An intent for this exact (src, dst) was written by an earlier run."""
return any(r.get("op") == "move" and r.get("dst") == dst
and r.get("src") == src for r in self.records)
@property
def rolled_back(self):
return any(r.get("op") == "rollback" for r in self.records)
def record(self, **fields):
self.records.append(fields)
if not self.writing:
return
if self._fh is None:
self._fh = open(self.path, "a")
self._fh.write(json.dumps(fields) + "\n")
self._fh.flush()
os.fsync(self._fh.fileno())
def close(self):
if self._fh:
self._fh.close()
def read_plan(path, root, force):
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
mkdirs, keeps, moves = [], [], []
with open(path) as fh:
header = fh.readline()
if not header.startswith("action\t"):
raise Refuse(f"{path} is not a plan (first column is "
f"{header.split(chr(9))[0]!r}, expected 'action') —"
" regenerate it with scripts/media-plan.py")
for line in fh:
f = line.rstrip("\n").split("\t")
if f[0] == "mkdir":
mkdirs.append(f[2])
elif f[0] == "keep":
keeps.append(f[1])
elif f[0] == "move":
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
if not force:
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
if p and not (p == root or p.startswith(root + os.sep)):
raise Refuse(f"{p} is outside --root {root}"
" (a plan built for another library?)")
return mkdirs, keeps, moves
def do_apply(args, root, journal, mkdirs, keeps, moves):
"""Preflight everything, then folders, then renames. Never a partial start."""
problems = []
for p in keeps:
if not os.path.exists(p):
problems.append(f"missing: keep row source is gone: {p}")
seen = {}
for src, dst, _kind in moves:
if src == dst:
problems.append(f"no-op: move row whose target is its source: {src}")
if dst in seen:
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
seen[dst] = src
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue # this intent already ran
if not os.path.exists(src):
if os.path.exists(dst):
problems.append(f"missing: {src} is gone and {dst} carries no journal"
" line — it moved outside this tool, refusing to guess")
else:
problems.append(f"missing: {src} is gone and {dst} is not there either")
continue
if os.path.exists(dst):
problems.append(f"occupied: target already holds another file: {dst}")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved")
for p in sorted(problems)[:20]:
print(" " + p)
if len(problems) > 20:
print(f" ... and {len(problems) - 20} more")
return 2
if not args.apply:
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
" — nothing written (pass --apply to run)")
for src, dst, _k in moves[:5]:
print(f" would move {src} -> {dst}")
return 0
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
dirs_new = 0
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
if d in made or os.path.isdir(d):
continue
journal.record(op="mkdir", dst=d)
os.makedirs(d, exist_ok=True)
dirs_new += 1
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
# library this size is not an option anyone would choose.
cross = []
for src, dst, _k in moves:
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue
parent = os.path.dirname(dst)
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
cross.append(dst)
if cross:
print(f"refusing: {len(cross)} moves would cross a filesystem"
f" (first: {cross[0]}) — this tool only renames")
return 2
moved = done = 0
for src, dst, _kind in moves:
if journal.already(dst, src):
if os.path.exists(dst) and not os.path.exists(src):
done += 1
continue
if os.path.exists(src) and not os.path.exists(dst):
os.rename(src, dst) # the intent ran, the move did not
moved += 1
continue
st = os.stat(src)
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
os.rename(src, dst)
moved += 1
print(f"applied: {moved} moved, {done} already done in an earlier run,"
f" {dirs_new} folders created, {len(keeps)} files stayed put")
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
f" --journal {journal.path} --root {root} --apply")
return 0
def do_verify(root, journal, moves, keeps):
"""Every planned file sits at its target carrying the journalled bytes."""
journalled = {(r["src"], r["dst"]): r for r in journal.records
if r.get("op") == "move"}
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
if r.get("op") == "move"]
bad, missing, bytes_total = [], [], 0
for src, dst, _kind in todo:
rec = journalled.get((src, dst))
if not os.path.exists(dst):
missing.append(dst)
continue
size = os.path.getsize(dst)
bytes_total += size
if rec and rec.get("bytes") not in (None, size):
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
if os.path.exists(src) and src != dst:
bad.append(f"{src}: still in place as well as at {dst}")
missing += [p for p in keeps if not os.path.exists(p)]
if bad or missing:
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
for line in (missing + bad)[:20]:
print(" " + line)
return 2
empty = [r["dst"] for r in journal.records
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
and not os.listdir(r["dst"])]
print(f"verified: {len(todo)} files at their targets"
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
if empty:
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
return 0
def sweep_targets(root, journal):
"""The folders this run emptied, deepest first — never one it did not touch.
Eligibility is evidence, not appearance. A folder is a target only when the
journal shows a file moved out of it or out of something beneath it, it holds
nothing but empty directories now, it is not a folder this run created, and no
journalled destination lives under it. A symlink anywhere inside disqualifies
the whole subtree: this tool never follows a link, and deleting the folder
that holds one would delete someone's mount point.
Proof carries downward: once a folder qualifies, every empty directory inside
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
a move source, so without that rule it holds its emptied parent on disk
forever — the parent can never be removed while the child stands.
"""
moves = [r for r in journal.records if r.get("op") == "move"]
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
# An rmdir INTENT that failed left its folder standing, and the next sweep
# must be able to finish it: only a folder that is really gone is out of
# scope. Counting every intent as done made 146 live survivors permanent.
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
and not os.path.exists(r["path"])}
dsts = [r["dst"] for r in moves]
marked = set()
for r in moves:
d = os.path.dirname(r["src"])
while d.startswith(root + os.sep):
marked.add(d)
d = os.path.dirname(d)
def removable(d):
if d in made or d in gone or os.path.islink(d):
return False
if any(p == d or p.startswith(d + os.sep) for p in dsts):
return False
try:
entries = os.listdir(d)
except OSError:
return False
for e in entries:
c = os.path.join(d, e)
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
return False
return True
roots = [d for d in marked if removable(d)]
targets = set(roots)
for d in roots:
for dirpath, dirnames, _ in os.walk(d):
for c in dirnames:
p = os.path.join(dirpath, c)
if removable(p):
targets.add(p)
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
def do_sweep(args, root, journal):
"""Remove the empty husks an apply left standing at the library root."""
moves = [r for r in journal.records if r.get("op") == "move"]
if not moves:
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
" does not guess at emptiness")
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
if not (p == root or p.startswith(root + os.sep))})
if outside and not args.force:
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
f" (first: {outside[0]}) — this journal belongs to another"
" library")
targets = sweep_targets(root, journal)
if not targets:
print("swept: 0 folders — nothing this run emptied is still standing empty")
return 0
if not args.apply:
print(f"dry run: {len(targets)} folders this run emptied would be removed"
" (pass --apply to run)")
for d in targets[:5]:
print(f" would remove {d}")
return 0
removed = 0
for d in targets:
journal.record(op="rmdir", path=d)
try:
os.rmdir(d)
except OSError as e:
print(f"left in place: {d} ({e})")
continue
removed += 1
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
print(f"journal: {journal.path} — rollback recreates them with their files")
return 0
def do_rollback(args, root, journal):
"""Undo every journalled move, newest first, after a full preflight."""
moves = [r for r in journal.records if r.get("op") == "move"]
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
problems = []
for r in reversed(moves):
src, dst = r["src"], r["dst"]
if not args.force and not (src.startswith(root + os.sep)
or dst.startswith(root + os.sep)):
problems.append(f"outside --root: {dst}")
continue
if os.path.exists(dst) and os.path.exists(src):
problems.append(f"occupied: both paths exist, refusing to choose:"
f" {src} | {dst}")
elif os.path.exists(dst):
pass
elif os.path.exists(src):
continue # already undone
else:
problems.append(f"missing: {dst} is gone and {src} never came back")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved back")
for p in problems[:20]:
print(" " + p)
return 2
if not args.apply:
print(f"dry run: would move {len(moves)} files back to their census paths"
f" and recreate {len(set(swept))} swept folders (pass --apply)")
return 0
undone = 0
recreated = 0
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
if not os.path.isdir(d):
os.makedirs(d, exist_ok=True)
recreated += 1
for r in reversed(moves):
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
os.rename(r["dst"], r["src"])
undone += 1
dirs_gone = 0
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
if d == root or not os.path.isdir(d) or os.listdir(d):
continue
try:
os.rmdir(d)
dirs_gone += 1
except OSError as e:
print(f"left in place: {d} ({e})")
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
print(f"rolled back: {undone} files back where the census found them,"
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
return 0
def main():
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
ap.add_argument("--plan")
ap.add_argument("--root", default=DEFAULT_ROOT)
ap.add_argument("--journal")
ap.add_argument("--apply", action="store_true", help="actually change the library")
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
args = ap.parse_args()
root = os.path.abspath(args.root)
if not os.path.isdir(root):
raise Refuse(f"no library at {root}")
journal_path = args.journal or journal_default(root)
journal = Journal(journal_path, args.apply)
try:
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
raise Refuse("this journal has already been rolled back"
" (--force to replay it)")
if args.cmd in ("apply", "verify") and not args.plan:
raise Refuse(f"{args.cmd} needs --plan")
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
if args.plan else ([], [], []))
if args.cmd == "apply":
return do_apply(args, root, journal, mkdirs, keeps, moves)
if args.cmd == "verify":
return do_verify(root, journal, moves, keeps)
if args.cmd == "sweep":
return do_sweep(args, root, journal)
return do_rollback(args, root, journal)
finally:
journal.close()
if __name__ == "__main__":
try:
sys.exit(main())
except Refuse as r:
print(f"refusing: {r}")
sys.exit(2)
+203
View File
@@ -0,0 +1,203 @@
#!/usr/bin/env python3
r"""Read-only census of a movie library — the inputs a relayout plan is built from.
media-census.py enumerate [--root DIR] [--out TSV]
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
Run it on the machine that mounts the library. Nothing here writes inside the
library and nothing is deleted; the only outputs are the three TSVs below, and
scripts/media-plan.py reads them instead of touching media.
enum bytes <TAB> ext <TAB> path every file under the root
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
one row per video-container file; `audio` is "codec:lang+codec:lang",
`subs` the subtitle-stream count, `spare` the audio streams carrying
neither a default nor a forced tag
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
the titles Plex already matched — the authority over any filename guess
`probe` is the slow step (every container is opened for its header, ~10 min for
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
live service must not starve it. `enumerate` is a stat pass, so re-censusing
before a plan is cheap — and a plan is only ever as good as these three files.
"""
import argparse
import csv
import json
import os
import re
import sqlite3
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
# Trash and thumbnail caches are not library content, and this repo's trash policy
# means .AGENTTRASH can sit inside a movie folder at any time.
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
"parent_title", "parent_orig", "parent_year", "parent_edition",
"container", "vcodec", "acodec", "width", "height"]
PLEX_SQL = """
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
coalesce(mi.edition_title,''), coalesce(mi.year,0),
coalesce(par.title,''), coalesce(par.original_title,''),
coalesce(par.year,0), coalesce(par.edition_title,''),
m.container, m.video_codec, m.audio_codec, m.width, m.height
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
left join metadata_items par on par.id = mi.parent_id
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
"""
def enumerate_lib(root, out):
"""One row per real file under root: size, extension, path."""
rows = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames
if not SKIP_DIR.search(os.path.join(dirpath, d))]
for fn in filenames:
p = os.path.join(dirpath, fn)
if SKIP_DIR.search(p):
continue
try:
st = os.lstat(p)
except OSError:
continue
if not os.path.isfile(p) or os.path.islink(p):
continue # links and sockets are not library content
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
with open(out, "w") as fh:
for size, ext, p in rows:
fh.write(f"{size}\t{ext}\t{p}\n")
vid = [r for r in rows if r[1] in PROBE_EXT]
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
def probe_one(path):
"""ffprobe header read -> the probed TSV row (never the whole file)."""
try:
os.nice(19) # a census must never outrank the media server
except OSError:
pass
try:
size = os.stat(path).st_size
except OSError:
return None
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
"-show_format", "-show_streams", path]
try:
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
data = json.loads(done.stdout or "{}")
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
json.JSONDecodeError, OSError):
data = {}
st = data.get("streams", [])
fmt = data.get("format", {})
v = next((s for s in st if s.get("codec_type") == "video"), None)
aud = [s for s in st if s.get("codec_type") == "audio"]
subs = [s for s in st if s.get("codec_type") == "subtitle"]
fps = "-"
if v and v.get("r_frame_rate"):
try:
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
except ValueError:
pass
def num(x):
try:
return round(float(x))
except (TypeError, ValueError):
return 0
return [size, path, num(fmt.get("duration")),
(v or {}).get("codec_name", "-"),
f"{v['width']}x{v['height']}" if v else "-",
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
len(subs),
sum(1 for a in aud if not re.search(
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
fmt.get("format_name", "-") or "-"]
def probe(enum, out, jobs):
"""Probe every video-container row of the enum TSV, in parallel."""
paths = []
with open(enum) as fh:
for line in fh:
f = line.rstrip("\n").split("\t", 2)
if len(f) == 3 and f[1] in PROBE_EXT:
paths.append(f[2])
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
done = unreadable = 0
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
for row in pool.map(probe_one, paths):
if row is None:
continue
if row[3] == "-":
unreadable += 1 # header unreadable: the row still gets written
done += 1
fh.write("\t".join(str(x) for x in row) + "\n")
if done % 250 == 0:
print(f" {done}/{len(paths)}", flush=True)
print(f"probed={done} unreadable={unreadable} -> {out}")
def plex(root, db, out):
"""Plex's matched titles for files under root, from its DB, read-only."""
uri = f"file:{db}?mode=ro&immutable=1"
try:
con = sqlite3.connect(uri, uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t")
w.writerow(PLEX_COLS)
n = 0
for r in con.execute(PLEX_SQL, (root + "/%",)):
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
n += 1
print(f"plex-matched={n} -> {out}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
"help": "library root"}
e = sub.add_parser("enumerate", help="stat every file under the root")
e.add_argument("--root", **root)
e.add_argument("--out", default="/tmp/media-enum.tsv")
p = sub.add_parser("probe", help="ffprobe every video container")
p.add_argument("--enum", default="/tmp/media-enum.tsv")
p.add_argument("--out", default="/tmp/media-probed.tsv")
p.add_argument("--jobs", type=int, default=6)
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
x.add_argument("--root", **root)
x.add_argument("--db", default=PLEX_DB)
x.add_argument("--out", default="/tmp/plex-movies.tsv")
a = ap.parse_args()
if a.cmd == "enumerate":
enumerate_lib(a.root, a.out)
elif a.cmd == "probe":
probe(a.enum, a.out, a.jobs)
else:
plex(a.root, a.db, a.out)
if __name__ == "__main__":
main()
+881
View File
@@ -0,0 +1,881 @@
#!/usr/bin/env python3
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
[--out TSV] [--exceptions TXT]
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
metadata-cache export whose clean titles outrank the filename parser; the title
authority for everything Plex never matched is scripts/media-titles.py beside this
file. No media file is opened, so planning the whole library costs seconds.
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
parents exist before children. EXCEPTIONS is the human review list: every file the
plan could not name a film for, every collision it refused, the READMEs the
executor must write into the piles it created.
Target contract (jellyfin.org/docs/general/server/media/movies):
* one folder per movie, named Title (year)
* the video inside is named exactly like the folder
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
match the folder name character-for-character or Jellyfin splits them apart
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
* <>:"/\|?* break the matcher, so they never survive here
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
"""
import argparse
import collections
import csv
import importlib.util
import os
import re
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
# this file is the title authority. Nothing here opens a media file.
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="library root the plan moves files inside")
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
help="Plex metadata export (clean titles); absent = parser only")
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
return ap.parse_args()
ARGS = parse_args()
ROOT = ARGS.root
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
r"|(additional|extra) languages|languages|resources")
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
r"|(additional|extra) languages|commentar(y|ies)"
r"|audio commentary|audio tracks?")
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
".opus", ".ogg", ".wma"}
NFO_EXT = {".nfo"}
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
"banner": "banner", "logo": "logo", "clearlogo": "logo",
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
"discart": "poster", "disc": "poster"}
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
# library holds six of those films), so leaving it in a name turns ONE movie folder
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
# folders, none of them a movie.
BAD_CHARS = re.compile(r'[<>:"/|?*]')
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
"eng", "english", "italian", "dual", "audio", "forced", "default"}
# Canonical title where the library only ever carries a localised name.
OVERRIDES = {"28 giorni dopo": "28 Days Later",
"28 settimane dopo": "28 Weeks Later"}
rows = []
for line in open(ENUM):
size, ext, path = line.rstrip("\n").split("\t")
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
ext_by = {r["path"]: r["ext"] for r in rows}
size_by = {r["path"]: r["bytes"] for r in rows}
def rel(p):
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
def parts_of(p):
return [q for q in rel(p).split("/") if q]
def is_qb(p):
return ".!q" in p.lower()
def disc_root(p, fold=True):
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
ps = parts_of(p)
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
for i, q in enumerate(ps):
if q.upper() in kinds:
return ROOT + "/" + "/".join(ps[:i])
return None
def row_name(r):
"""The name a disc-tree entry should be identified by: the engine stores the
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
return r.get("name_src") or r["path"]
def sanitize(name):
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return (name[:180] or "untitled")
def ok_year(y):
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
def words_of(text):
"""The matching words of a name — or of a whole path, which is how callers
pass a file together with the folders it sits in.
Split on everything that is not a letter or a digit: eng.norm does not touch
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
film could not be found by the folder that holds it — which is how one box
set's featurettes ended up in another film's folder. Both sides of every
comparison below go through here. A bare number counts only when it could be
a release year, so release junk ("1080", "2160", "2", "7") never poses as
agreement between two names.
"""
out = set()
for w in WORD_SPLIT.split(eng.norm(text)):
if not w or w in eng.STOP:
continue
if w.isdigit() and not 1870 <= int(w) <= 2100:
continue
out.add(w)
return out
# ------------------------------------------------------------------ Plex titles
# Plex has matched a large part of the library's media files to real titles
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
# holds a MATCH, its title wins; the filename parser carries everything else
# (extras, disc trees, TV, and whatever Plex never matched).
def plex_titles():
if not os.path.exists(PLEX):
return {}
out = {}
with open(PLEX, newline="") as fh:
for r in csv.DictReader(fh, delimiter="\t"):
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
# Only a match is an authority, and a match carries a year: an
# unmatched row echoes the filename, which the parser reads better
# than a copied release name. The junk filter stays for the matches
# whose title still carries the release.
if t and ok_year(y) and not re.search(
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
out[r["file"]] = (t, y)
return out
PLEX_T = plex_titles()
# ------------------------------------------------------------- classification
erows = eng.load(PROBE)
for r in erows:
for k, v in OVERRIDES.items():
if k in r["title"]:
r["title"] = v
r["key"] = eng.key_of(r["title"])
r["base"] = eng.base_words(r["title"], r["key"])
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
extras = {r["path"] for r in erows
if r["path"] not in marks
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
matched=r["path"] in PLEX_T)}
eps = collections.defaultdict(list)
for r in erows:
if r["path"] in marks | extras:
continue
ep = eng.ep_of(r["path"])
if ep:
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
# A folder INSIDE a movie's folder is bonus territory even when its own name is
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
# misspelled Extras folder, so the engine's folder rule never fired and the
# featurette became a movie folder of its own — whose words then out-scored the
# real Saboteur and drew that box set's other extras into it. The rule needs no
# metadata: the folder holding the longest film owns the shorter video filed
# under it, provided that shorter file's own name speaks the film's name — a
# whole other film filed there stays a film, exactly as the engine's folder rule
# exempts it. A different year in the file's own name IS that other film
# speaking, so it vetoes the rule.
def longest_of(rows):
"""The longest video in each folder: that folder's feature."""
out = {}
for r in rows:
if r["dur"] > 0:
d = os.path.dirname(r["path"])
if d not in out or r["dur"] > out[d]["dur"]:
out[d] = r
return out
def not_disc(r):
"""A disc tree's innards belong to the tree, never to the folder's film."""
return not any(c in DISC_DIRS for c in r["path"].split("/"))
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
def under_a_feature(path, dur):
"""True when a folder above this file holds a longer film that this file's
own name names."""
d = os.path.dirname(os.path.dirname(path))
while len(d) > len(ROOT):
f = FEATURE_OF.get(d)
if f:
if f["dur"] < dur:
return False
theirs = (words_of(os.path.basename(f["path"]))
| words_of(os.path.basename(d)))
if not words_of(os.path.basename(path)) & theirs:
return False
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
return not (y1 and y2 and y1.group() != y2.group())
d = os.path.dirname(d)
return False
extras |= {r["path"] for r in erows
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
buckets = collections.defaultdict(list)
for r in film_rows:
buckets[eng.key_of(r["title"])].append(r)
def res_label(r):
m = RES_RE.search(r["res"] or "")
if m:
n, kind = m.group(1).lower(), m.group(2).lower()
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
try:
w = int((r["res"] or "0x0").split("x")[0])
except (ValueError, IndexError):
w = 0
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
def version_label(r):
bits = [res_label(r)]
vc = (r["vcodec"] or "").lower()
if vc in ("av1", "libaom", "libsvtav1"):
bits.append("AV1")
elif vc in ("vc1", "wmv3"):
bits.append("VC1")
if r["bytes"] > 40 * 2**30:
bits.append("REMUX")
if "imax" in r["path"].lower():
bits.append("IMAX")
return sanitize(" ".join(bits))
def group_name(items):
titles = collections.Counter(r["title"] for r in items)
ascii_t = [t for t in titles if t.isascii()]
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
years = [r["year"] for r in items if r["year"] != "?"] or \
[r["soft"] for r in items if r["soft"]]
year = collections.Counter(years).most_common(1)[0][0] if years else None
return sanitize(f"{title} ({year})" if year else title)
groups = []
for key, items in buckets.items():
for c in eng.dur_clusters(items):
g = {"rows": c, "base": group_name(c)}
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
if known:
# one film, one name: a Plex match for any copy names every copy
t, yr = collections.Counter(known).most_common(1)[0][0]
g["plex"] = t
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
groups.append(g)
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
for g in groups:
if g["base"] in clash:
# two genuinely different films share title+year: separate folders, and
# say so in the name, because Jellyfin can only match one item per folder
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
g["flag"] = "same title and year as another film"
moves, mkdirs, problems, readmes = [], set(), [], []
# --------------------------------------------------------------- unnamed groups
# A movie folder has to READ as a title. The engine's runtime veto says any file
# over 45 minutes that is not named like an extra is a film whatever it is called
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
# with no identity whose numeric keys went on to capture every subtitle named
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
# real film), so the gate is: a year, or a word that is not release noise. With
# neither, the rows are bonus material until a human names them.
def named(g):
if re.search(r"\(\d{4}\)$", g["base"]):
return True
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
unnamed = [g for g in groups if not named(g)]
groups = [g for g in groups if named(g)]
for g in unnamed:
for r in g["rows"]:
extras.add(r["path"])
problems.append(
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
f"bonus pile, not a folder — give the file a real name and re-plan")
group_of = {}
for g in groups:
g["dir"] = f"{ROOT}/{g['base']}"
for r in g["rows"]:
group_of[r["path"]] = g
dr = disc_root(r["path"])
if dr:
group_of[dr] = g
by_path = {r["path"]: r for r in erows}
# -------------------------------------------------------------- bookkeeping
claimed = {}
existing = set(rows_map := {r["path"] for r in rows})
at_target, booked_src = set(), set()
def book(src, dst, kind):
if src == dst:
at_target.add(src)
booked_src.add(src)
return
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
why = "rebook" if src in booked_src else "collision"
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
if src in booked_src: # one source may only ever move once
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
return
if dst.lower() in claimed:
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
return
claimed[dst.lower()] = src
booked_src.add(src)
comp = dst.split("/")
for i in range(1, len(comp)):
mkdirs.add("/".join(comp[:i]))
moves.append((src, dst, kind))
def uniq_dir(d, stem, ext, hint=""):
"""A destination that no other source already took."""
cand = f"{d}/{stem}{ext}"
if cand.lower() not in claimed:
return cand
n = 2
while True:
cand = f"{d}/{stem} {n}{ext}"
if cand.lower() not in claimed:
return cand
n += 1
def media_dir(p):
dr = disc_root(p)
return os.path.dirname(dr) if dr else os.path.dirname(p)
# ------------------------------------------------------------------ films
# film / disc moves: the largest copy carries the folder's own name, the rest
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
def vext(p):
return ext_by.get(p) or os.path.splitext(p)[1].lower()
disc_children = collections.defaultdict(list)
for q in rows:
dr = disc_root(q["path"])
if dr:
disc_children[dr].append(q["path"])
for g in groups:
primary_taken = False
disc_kinds = set()
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
dr = disc_root(r["path"])
if dr:
inner = os.path.basename(dr).upper()
if inner in disc_kinds:
problems.append(
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
f"share the folder — discs have no multiple versions")
continue
disc_kinds.add(inner)
for src in disc_children[dr]:
tail = os.path.relpath(src, dr)
fold = disc_root(src, fold=False)
if fold != dr and os.path.dirname(src) != dr:
tail = os.path.basename(dr) + "/" + tail
book(src, f"{g['dir']}/{tail}", "disc-file")
continue
label = "" if not primary_taken else f" - {version_label(r)}"
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
n = 2
while dst.lower() in claimed:
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
n += 1
r["dst"] = dst
primary_taken = True
book(r["path"], dst, "film")
g["primary"] = primary_taken
# ------------------------------------------------------------------ extras
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
("interviews", r"interview|cast and crew|session piece|commentary"),
("featurettes", r"featurette|behind the scenes|making of|documentary"
r"|heritage|archival|video archive|music video"),
("samples", r"sample"),
("shorts", r"\bshort\b|short film"),
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
r"|audition|test footage|watermark|workprint|poster"
r"|picture in picture|singalong|karaoke|original score")]
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
r"|\bost\b|music|song", re.I)
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
def extras_kind(path):
low = path.lower()
if THEME_RE.search(low):
return "theme-music"
if BACKDROP_RE.search(low):
return "backdrops"
for name, rx in EXTRA_KIND:
if re.search(rx, low):
return name
return "extras"
def attachable_title(path):
t, _y = eng.title_of(path)
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
for g in groups:
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
top_groups = collections.defaultdict(list)
for g in groups:
for t in g["tops"]:
top_groups[t].append(g)
group_by_key = collections.defaultdict(list)
for g in groups:
for r in g["rows"]:
group_by_key[eng.key_of(r["title"])].append(g)
group_by_dir = {}
for g in groups:
for r in g["rows"]:
group_by_dir.setdefault(media_dir(r["path"]), g)
dr = disc_root(r["path"])
if dr:
group_by_dir.setdefault(os.path.dirname(dr), g)
def group_words(g):
"""The words that identify a film group: its titles and the names of the
folders its files live in."""
if "words" not in g:
w = set()
if g.get("plex"):
w |= words_of(g["plex"])
for r in g["rows"]:
w |= words_of(r["title"])
w |= words_of(os.path.basename(media_dir(r["path"])))
g["words"] = w
return g["words"]
def score_words(text, groups):
"""Ranks film groups by how many of their words a path carries."""
words = words_of(text)
out = [(len(words & group_words(g)), g) for g in groups]
out.sort(key=lambda x: -x[0])
return out
def group_for_orphan(path):
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
several films must be resolved by WORDS, never by position: taking the first
film of the folder attached every loose subtitle in a box set to whichever
movie happened to sort first."""
stem = os.path.splitext(os.path.basename(path))[0]
key = attachable_title(path)
if key and group_by_key.get(key):
cands = group_by_key[key]
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
pool = same_top or cands
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
pool = top_groups.get(gtop, [])
if len(pool) > 1: # a loose collection folder (box set, bonus
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
return g if n >= 2 else None # album) is resolved by words, not position
if len(pool) == 1:
return pool[0]
d = os.path.dirname(path)
while d and d != ROOT:
if d in group_by_dir:
return group_by_dir[d]
d = os.path.dirname(d)
return None
tv_tops = {parts_of(p)[0] for p in tv_set}
def film_folder_for(path):
"""A movie folder for a lone feature that has no group: derived from its own
title when a year proves it, else None (goes to the review pile)."""
t, y = eng.title_of(path)
if not t or y == "?":
return None
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
mkdirs.add(d)
return d
for p in sorted(extras):
if disc_root(p):
continue
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
continue # a special inside a TV set: staged with it
g = group_for_orphan(p)
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
stem = os.path.splitext(os.path.basename(p))[0]
if g is None:
solo = film_folder_for(p)
if solo:
dst = f"{solo}/{sanitize(stem)}{ext}"
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
f"{os.path.basename(solo)}")
else:
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
problems.append(f"NO-FILM extra {rel(p)}")
else:
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
ext, os.path.dirname(p)), "extra")
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
# they belong to the film of the folder they sit in, kept as a named subtree.
for q in rows:
pth = q["path"]
dr = disc_root(pth)
if not dr or pth in by_path or pth in booked_src:
continue
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
if g is None:
continue
inner = os.path.basename(dr)
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
f"(a second master: compare before keeping both)")
for src in disc_children[dr]:
if src not in booked_src:
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
# ----------------------------------------------------------------- sidecars
media_by_dir = collections.defaultdict(list)
for r in erows:
if r["path"] in marks:
continue
dst = r.get("dst")
if dst:
media_by_dir[media_dir(r["path"])].append(r)
def sidecar_target(path, e):
d = os.path.dirname(path)
stem = os.path.splitext(os.path.basename(path))[0]
if d == ROOT:
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
# set of root-level MOVIE files, and scoring a stray name against it
# attached every root-level leftover to whichever root-level film was
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
return None, None
cands = media_by_dir.get(d, [])
if not cands:
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
while nd and nd != ROOT and not media_by_dir.get(nd):
nd = os.path.dirname(nd)
if media_by_dir.get(nd):
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
g = group_for_orphan(path) # box set's own root) carries evidence
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
if g is None or g["dir"] not in dirs:
# ...but only a STRONG word match outranks the folder the file
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
# against a junk group named "3" and escaped its own movie's
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
# and genuinely belongs to the other film of the set.
n = score_words(stem, [g])[0][0] if g else 0
if n >= 2:
return (g, None)
cands = media_by_dir[nd]
if not cands:
g = group_for_orphan(path)
return (g, None) if g else (None, None)
if len(cands) > 1:
w = [t for t in eng.norm(stem).split() if t not in NOISE]
cands = sorted(cands, key=lambda r: (
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
-r["bytes"]))
return (None, cands[0])
for q in rows:
p, e = q["path"], q["ext"]
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
continue
if os.path.basename(p).startswith("._"):
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
# dead files through four movie folders (and they are video extensions in
# the AppleDouble case, so they also skipped every other filter). They
# claim nothing: one pile, and they stay in it — nothing is deleted here.
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
"", os.path.dirname(p)), "junk")
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
continue
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
g = group_for_orphan(p)
stem = os.path.splitext(os.path.basename(p))[0]
if g is not None:
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
else f"{g['dir']}/{g['base']}{e}", "iso")
else:
solo = film_folder_for(p)
if solo:
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
f"folder -> {os.path.basename(solo)}")
else:
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
problems.append(f"NO-FILM disc image {rel(p)}")
continue
if e in VIDEO_EXT:
# A video the engine never probed (unreadable header) and not an AppleDouble
# remnant: the plan does not guess at it, so it stays where it is.
continue
stem = os.path.splitext(os.path.basename(p))[0]
g, media = sidecar_target(p, e)
if media is not None:
base = os.path.splitext(os.path.basename(media["dst"]))[0]
d = os.path.dirname(media["dst"])
elif g is not None:
base, d = g["base"], g["dir"]
else:
# Nothing names this file's film. NEVER scatter it: an archive set or an
# album tree must move as the unit it is, so the whole loose folder goes
# to the review pile and a film folder's own leftovers stay with it.
ps = parts_of(p)
if len(ps) < 2:
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
elif len(top_groups.get(ps[0], [])):
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
else:
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
book(dst=dst, src=p, kind="orphan")
problems.append(f"NO-OWNER sidecar {rel(p)}")
continue
if e in IMG_EXT:
low = eng.norm(stem)
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
kind = "poster" if word else next(
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
if re.search(r"\btheme\b", low):
dst = f"{d}/theme{e}"
else:
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
e, os.path.dirname(p)), "image")
elif e in NFO_EXT:
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
elif e in SUB_EXT | AUDIO_EXT:
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
langs = [t.lower() for t in toks if t.lower() in LANGS]
flags = [t.lower() for t in toks if t.lower() in FLAGS]
keep = [sanitize(t) for t in toks
if t.lower() not in langs + flags + list(NOISE)
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
if e in SUB_EXT:
# subtitles: <movie>/srt[/language]
d = f"{d}/srt"
pd = os.path.basename(os.path.dirname(p))
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}" # a real language folder survives
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
d = f"{d}/theme-music" # bonus audio: one canonical folder
pd = os.path.basename(os.path.dirname(p))
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
d = f"{d}/{sanitize(pd)}"
else:
pd = os.path.basename(os.path.dirname(p))
if not AUDIO_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}"
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
else:
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
os.path.dirname(p)), "misc")
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
"one-folder-per-movie layout; `_release-group watermarks/`, "
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
for key in tv_keys:
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
"these are TV episodes found in the MOVIES library; move the folder "
"to a TV library source, else Jellyfin groups them wrongly as one "
"movie.")
# --------------------------------------------------------------------- TV
for key in tv_keys:
items = eps[key]
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
yr = year.most_common(1)[0][0] if year else None
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
tops = {parts_of(r["path"])[0] for r in items}
members = sorted([r["path"] for r in items]
+ [q for q in extras | marks if len(parts_of(q)) > 1
and parts_of(q)[0] in tops],
key=lambda q: q)
for src in members:
rel_parts = rel(src).split("/")
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
book(src, f"{ROOT}/{base}/{tail}", "tv")
for p in sorted(marks):
stem, ext = os.path.splitext(os.path.basename(p))
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
os.path.dirname(p)), "watermark")
for q in rows:
if is_qb(q["path"]):
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
book(q["path"], q["path"], "qb-partial") # stays: resume data
# --------------------------------------------------------------- debug hook
if os.environ.get("JF_ISO"):
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
raise SystemExit
if os.environ.get("JF_DEBUG"):
import json
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
print("at_target", len(at_target))
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
for up in unp[:20]:
d = os.path.dirname(up)
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
# -------------------------------------------------------------- validation
srcs = [m[0] for m in moves]
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
for s in dup_src:
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
all_src = set(srcs)
untouched = {r["path"] for r in rows} - all_src
bad_names, into_alive, lost = [], [], []
for src, dst, kind in moves:
for comp in dst[len(ROOT) + 1:].split("/"):
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
or len(comp) > 200):
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
if dst in untouched: # something staying put is there
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
if dst.lower() in {s.lower() for s in untouched}:
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
if dst.startswith(src + "/") and src in all_src:
lost.append(f"{rel(src)} moves into its own subtree")
# every accounted-for source must exist exactly once in the plan
covered = {r["path"] for r in rows}
not_planned = sorted(covered - all_src - at_target)
with open(PLAN, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
if d.startswith(ROOT + "/") and d != ROOT:
out.write(f"mkdir\t\t{d}\tdir\n")
for p in sorted(booked_src - {m[0] for m in moves}):
out.write(f"keep\t{p}\t{p}\n")
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
out.write(f"move\t{src}\t{dst}\t{kind}\n")
kinds = collections.Counter(k for _s, _d, k in moves)
films_n = sum(1 for _s, _d, k in moves if k == "film")
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
with open(EXCEPTS, "w") as out:
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
out.write(f"kinds={dict(kinds)}\n")
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
out.write("\n".join(sorted(set(readmes))) + "\n")
out.write("\n=== PROBLEMS ===\n")
out.write("\n".join(sorted(set(problems))) + "\n")
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+ "\n".join(sorted(set(into_alive))) + "\n")
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
f"mkdirs={len(mkdirs)} film-files={films_n}")
print(f"kinds={dict(kinds)}")
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
print(f"left-unmoved={len(not_planned)}")
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
f"disc-trees={len(disc_children)}")
+572
View File
@@ -0,0 +1,572 @@
#!/usr/bin/env python3
r"""The fleet's release-filename title parser: the grouping authority the relayout
plan uses for every file the media server never matched.
report: media-titles.py report [--probed TSV] [--out TSV]
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
ffprobe census scripts/media-census.py writes.
parse: media-titles.py parse NAME [NAME…]
what title_of() makes of a name — the way to check a naming accident
before it merges two films into one folder.
scripts/media-plan.py imports this file; over there the paths are arguments.
Method — three independent signals, so a naming accident cannot merge two films:
* franchise KEY : first two significant title tokens. Stable across parts of a
collection ("resident evil", "jump street") but NOT film-specific — it only
selects candidates.
* DURATION : the identity check. Same master encodes differ by at most a
couple of seconds; different films / cuts do not collide.
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
are byte-twins (the hash pass confirms); same duration but different sizes are
two different releases of one film (deletable redundancy).
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
would otherwise collapse a whole series into one "duplicate" group.
"""
import argparse
import collections, os, re, sys, unicodedata
# Library root: the folder directly under this one names the film, and every
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
# their own; nothing here reads media, only the census TSVs about it.
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
# Release-tag vocabulary: never part of a title, so the title stops here.
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
JUNK_RE = re.compile(
r"^("
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
r")$", re.I)
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
"vs", "avec", "con"}
def norm(s):
s = unicodedata.normalize("NFKD", s)
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
def fold_sep(s):
"""norm(), plus: every character that is left over is not a letter, digit or
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
of the word. Release groups file titles with lookalikes for punctuation the
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
those (no decomposition mapping), and the scrubber inside title_of DELETES
what it does not recognise, so five different V/H/S films squashed into the one
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
they separate. Accents are gone by then — norm() removed the combining marks —
so "Amélie" stays "amelie" and never splits into "ame lie"."""
return " ".join("".join(c if c.isalnum() or c == "'" else " "
for c in norm(s)).split())
def title_of(path):
"""(title, year) from the basename.
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
(2) Find the boundary: the first release-noise token (resolution / codec /
source / audio / language / group vocabulary) ends the title, so trailing
tags can never leak in. (3) Collect up to that boundary: leading index
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
are skipped while nothing is collected; once a real word appears a sentinel
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
"""
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
# docstring): the tokens below are then plain alnum/apostrophe words.
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
raw = [x for x in SEP_RE.split(name) if x]
years = [t for t in raw if YEAR_RE.match(t)]
two_year = len(years) > 1 # "2001…1968": one of them is the title
toks, year = [], None
for t in raw:
if YEAR_RE.match(t):
year = t # last year wins
toks.append("\x00")
else:
toks.append(t)
stop, seen_word = len(toks), False
for i, t in enumerate(toks):
if t == "\x00":
continue
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
stop = i
break
if c.isalpha():
seen_word = True
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
# it is three digits or more ("007", "000", "1408") or the name carries no year
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
# loses its zero to the parser — Plex's matched title overrides it anyway.
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
keep, num_title = [], None
for t in toks[:stop]:
if t == "\x00":
if keep:
break # year closes the title
if two_year and t is not None and years[0] != year:
# two DIFFERENT years ("2001…1968"): the leading one is the title;
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
num_title = num_title or years[0]
continue # leading year: not the title
if track_idx and IDX_RE.match(t) and not keep and t == lead:
continue # "03." / "1c." ordering
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if not keep and not c.isalpha():
num_title = num_title or (c or None)
continue # numeric title candidate
keep.append(c)
if len(keep) >= 8:
break
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
title = " ".join(keep)
if not keep:
# nothing collected: a pure-numeric leading token is the title even when it
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
for t in raw:
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
return c, year or "?" # never "2160p" / "1080": they are not pure digits
if (num_title and title and not keep[0].isdigit()
and (two_year or not track_idx)):
# a leading number that only echoes the release year ("1917.2019.2160p…") is
# not a title; one that stands alone ("300.2006…", "1900") is
title = f"{num_title} {title}"
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
return title, year or "?"
# Bonus-material vocabulary: names that say nothing about the film, so an extras
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
# Clips) — and the running-time guard in is_extra() carries the rest.
SEP_CLS = r"[\s._-]+"
def _bonus_rx(word):
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
EXTRA_WORDS = [
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
"tv spot", "advert", "commercial", "behind the scenes", "making of",
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
"menus", "gallery", "still gallery", "stills", "poster", "posters",
"montage", "preview", "previews", "epilogue", "visual effects",
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
"test footage", "screen test", "audition", "cast and crew", "isolated score",
"picture in picture", "original score", "session piece", "sample",
"watermark",
]
# Words that describe bonus material which can run LONG (a feature-length
# commentary-track cut, a documentary about the film). Trusted only below
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
"archival", "documentary"]
EXTRA_NAME_RE = re.compile(
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
EXTRA_LONG_NAME_RE = re.compile(
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
# search = it only mentions extras ("... + Extras" box sets hold feature films).
EXTRA_FOLDER_RE = re.compile(
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
# Anything longer than this is a feature film whatever it is called: a 118-minute
# file named "The Menu" or "Trailer" is a movie, not bonus material.
SHORT_MAX = 45 * 60
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
# commentary cut is an alternate version of the film, not a featurette.
LONG_BONUS_MAX = 25 * 60
def name_words(x):
"""Bare words of a file or folder name: no release metadata, no year."""
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
return [w for w in SEP_RE.split(fold_sep(x))
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
and not JUNK_RE.match(w) and w not in JUNK]
def mentions(short, long_):
"""True when one word list appears in the other, anywhere: a release names
the film at the front ("Alien (1979) - Trailer") or in the middle
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
a, b = sorted([short, long_], key=len)
if not a:
return False
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
def folder_is_bonus(folder):
"""True when a folder name carries bonus vocabulary as WHOLE words: the
vocabulary is matched against a padded word string, so "Extras" decides and
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
"""True for bonus material, which must never be counted as a film: a file
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
extras otherwise pair with each other across the whole library.
Evidence, strongest first:
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
unless it sits in an explicit bonus folder and metadata never matched it.
Bonus words that describe long-form material (commentary, documentary,
interview) are believed only
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
* The file's OWN name carries bonus vocabulary -> bonus.
* ANY folder of the path below the film's own root folder carries bonus
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
is exempt because it names the film, and a film may be titled with a
bonus word; and a file whose own name IS that folder's film name is the
feature, not the bonus ("Ghost in the Shell Movies/Extras/
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
there stays a film).
Bonus vocabulary is matched as whole words only, so the film
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
the folder rule, and "…Extras" as a fragment of a word is never evidence.
"""
parts = [q for q in path.split("/") if q]
if not parts:
return False
name = (os.path.splitext(parts[-1])[0]
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
# A bonus folder anywhere BELOW the film's own folder decides for its
# contents; the film's own folder is the film's name, whatever words it
# contains, so it is exempt. It is the first component AFTER ROOT's own
# components — those are the mount path, not folders of the library.
head = len([q for q in root.split("/") if q]) + 1
bonus_folder = False
for folder in parts[head:-1]:
if folder_is_bonus(folder):
bonus_folder = not mentions(name_words(name), name_words(folder))
break
if dur and dur > SHORT_MAX:
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
# it as a film. Long-form bonus words still describe a film, so a
# feature-length commentary cut or documentary keeps its runtime veto.
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
return True
return False
if EXTRA_NAME_RE.search(name):
return True
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
return True
return bonus_folder
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
# "RARBG.com.mp4") into many different films, so its name carries no film
# identity at all and it would otherwise pair with itself across the library.
WATERMARK_MAX = 20 * 2**20
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
def is_watermark(path, size):
"""True for a release group's own watermark video, not a film or an extra."""
return (size < WATERMARK_MAX
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
def key_of(title):
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
Deliberately NOT film-specific — it only selects candidates for the duration
check, so a two-token collision (two films starting "star trek") cannot by
itself create a duplicate group."""
toks = [t for t in title.split() if t]
words = [t for t in toks if not t.isdigit()]
if not words:
return " ".join(toks)
sig = [t for t in words if t not in STOP] or words
return " ".join(sig[:2])
def ep_of(path):
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
The series is the name BEFORE the marker. The box-set convention writes
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
follows the marker is not the show's name: keying on the whole title gave
every episode of one series its own group, so a 124-episode box set never
reached the size that calls something a series and each episode was filed
as its own film. A file named only by its marker has nothing before it, so
its whole title stands in."""
name = os.path.splitext(os.path.basename(path))[0]
m = EP_RE.search(name)
if not m:
return None
head = name[:m.start()].strip(" ._\u2013-")
series = title_of(head)[0] if head else None
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
def base_words(title, key):
"""Title tokens beyond the franchise key — the FILM identity part.
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
kt = set(key.split())
return frozenset(t for t in title.split() if t not in kt and t not in STOP
and not t.isdigit() and not YEAR_RE.match(t))
def folder_year(path):
"""Soft year evidence: a 19xx/20xx token in any parent directory name
(collection folders are written both "(2002)" and "[2009]"; a range such as
"(1998-2000)" deliberately yields nothing)."""
for d in reversed(os.path.dirname(path).split("/")):
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
if len(ys) == 1:
return ys[0]
return None
def same_film(a, b):
"""Two files are copies of ONE film only when the film-identity tokens agree
(one side may carry none) AND no year evidence contradicts them. The key says
the franchise matches; this says the actual film matches — it is what keeps
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
Powers apart when their runtimes happen to agree."""
ba, bb = a["base"], b["base"]
if ba and bb and ba != bb:
return False
ya = a["year"] if a["year"] != "?" else a["soft"]
yb = b["year"] if b["year"] != "?" else b["soft"]
if ya and yb and ya != yb:
return False
return True
def dur_close(a, b):
if a <= 0 or b <= 0:
return True # unknown duration: let titles decide
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
# is folded into ONE entry, described by its longest main-feature clip (measured
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
BDMV_MAIN = {
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
}
def load(probed):
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
BDMV tree collapses to one entry carrying the tree's total size."""
trees = {}
rows = []
for line in open(probed):
f = line.rstrip("\n").split("\t")
if len(f) < 12:
continue
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
base = os.path.basename(p)
if base.startswith("._") or base == ".DS_Store":
continue # macOS sidecar, not media
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
root = p[:m.start()] if m else None
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
if not root:
continue # disc bookkeeping: folds into its tree, never a film
if root:
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
t["bytes"] += int(b)
t["audio"].update(a for a in aud.split("+") if a)
t["subs"] += int(subs or 0)
continue
try:
dur = float(dur)
except ValueError:
dur = 0.0
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
for root, t in trees.items():
folder = os.path.basename(root)
m = BDMV_MAIN.get(folder, {})
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
name_src=folder, dur=m.get("dur", 0.0),
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
fmt="BDMV tree"))
for r in rows:
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
r["key"] = key_of(r["title"])
r["base"] = base_words(r["title"], r["key"])
r["soft"] = folder_year(r["path"])
r["ep"] = ep_of(r["path"])
return rows
def dur_clusters(items):
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
buckets come from the franchise key; same_film then rejects the different
installments whose runtimes happen to agree."""
clusters = []
for r in sorted(items, key=lambda r: -r["dur"]):
for c in clusters:
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
c.append(r)
break
else:
clusters.append([r])
return clusters
def tier(groups):
"""Split same-film/same-duration clusters: DUP = different releases of one
film (real, deletable redundancy); REVIEW = identical size AND duration
(byte-twins — the hash pass is the authority) or clashing year evidence."""
dup, review = [], []
for g in groups:
g.sort(key=lambda r: -r["bytes"])
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
years.discard(None)
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
if len(years) > 1 or spread < 0.005:
review.append(g)
else:
dup.append(g)
return dup, review
def redun(g):
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
def line(r):
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
def dump(tag, groups, out, note=""):
for gi, g in enumerate(groups, 1):
g.sort(key=lambda r: -r["bytes"])
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
for rank, r in enumerate(g):
print(f" {line(r)} [{r['year']}] "
f"{r['path'].replace(ROOT + '/', '')}")
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
f"{r['path']}\n")
def report(probed, out_path):
rows = load(probed)
for r in rows:
r["mark"] = is_watermark(r["path"], r["bytes"])
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
marks = [r for r in rows if r["mark"]]
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
extras = [r for r in rows if r["extra"]]
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
f"TV episodes + {len(extras)} bonus/trailers "
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
f"+ {len(marks)} release-group watermarks")
out = open(out_path, "w")
buckets = collections.defaultdict(list)
for r in movies:
buckets[r["key"]].append(r)
cand = {k: v for k, v in buckets.items() if len(v) > 1}
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
dup, review = tier(clusters)
dup.sort(key=lambda g: -redun(g))
review.sort(key=lambda g: -redun(g))
dump("DUP", dup, out)
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
ebuf = collections.defaultdict(list)
for r in episodes:
ebuf[(r["ep"][0], r["ep"][1])].append(r)
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
for c in dur_clusters(v) if len(c) > 1])
edup.sort(key=lambda g: -redun(g))
dump("EP-DUP", edup, out)
shows = collections.Counter()
for r in episodes:
shows[r["ep"][0]] += 1
multi = {s: n for s, n in shows.items() if n > 4}
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
print(f" {n:4d} episodes {s}")
out.close()
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
print(f"REVIEW groups={len(review)} "
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
print(f"EP-DUP groups={len(edup)} "
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
def parse(names):
for n in names:
t, y = title_of(n)
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
rep.add_argument("--out", default="/tmp/media-similar.tsv")
par = sub.add_parser("parse", help="show the parsed title/year of a name")
par.add_argument("names", nargs="+")
a = ap.parse_args()
if a.cmd == "report":
report(a.probed, a.out)
else:
parse(a.names)
if __name__ == "__main__":
main()
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""media-tv.py — move TV episodes out of the movie library into a TV library.
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
`_TV needs a TV library - <series>` pile for a human. This is the other half of
that refusal. It walks the movie library, groups the files whose own name
carries an episode marker by SERIES, and writes a plan that places each series
in the shape both Jellyfin and Plex read without help:
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
It writes a plan and nothing else. Every rename belongs to
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
the executor's own path guard still has teeth, and give it its own journal —
one journal is one undo, and a TV move must not be welded to a movie relayout:
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py sweep --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
Guards, and why each one is here:
* The same five-file threshold as the planner. An episode-shaped name on one
file is not a series — `The Godfather Part 2` is a film — so a smaller group
is reported and left exactly where it is.
* An episode whose name carries no season number is reported and left, never
filed into an invented `Season 01`.
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
episode; an episode without its subtitles is a half-move.
* A group that would claim one destination twice is not resolved here. The
executor refuses the whole plan before its first rename, which is the right
outcome: nobody but Frank decides which copy of an episode is the keeper.
* An episode the destination library already holds at the same byte count is
left where it is. The TV library has held a complete Monk box set for years
while the movie library carried a second copy of all 124 episodes; a mover
that did not look where it was filing would have put both inside one library.
* A symlink is never followed and never moved.
"""
import argparse
import collections
import importlib.util
import os
import re
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="the movie library to empty of TV")
ap.add_argument("--library", default="/mnt/entropy/TVShows",
help="the TV library to fill (a different share or folder)")
ap.add_argument("--out", default=None,
help="plan TSV for scripts/media-apply.py"
" (default: media-tv-plan.tsv beside --root)")
ap.add_argument("--min", type=int, default=5,
help="episode files that make something a series")
a = ap.parse_args()
if not a.out:
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
"media-tv-plan.tsv")
return a
ARGS = parse_args()
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
BAD_CHARS = re.compile(r'[<>:"/|?*]')
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
# Sidecars that follow an episode; macOS forks and resume data do not.
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
# A box set that stamps the show's year into every file name; the folder's year
# must not be added on top of it.
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
"by", "de", "la", "vs"}
def clean(name):
"""A name the filesystem and both matchers can read (same law as the plan)."""
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return name[:180] or "untitled"
def episode_name(path):
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
already carries spaces is left exactly as it was written."""
stem, ext = os.path.splitext(os.path.basename(path))
if " " not in stem and stem.count(".") >= 2:
stem = stem.replace(".", " ")
return clean(stem) + ext.lower()
def head_of(path):
"""The publisher's series name: the text BEFORE the episode marker.
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
after the marker is not the show's name, and case is kept because it is the
library's own spelling, not a normalised key."""
name = os.path.splitext(os.path.basename(path))[0]
m = eng.EP_RE.search(name)
head = name[:m.start()] if m else name
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
def title_series(head):
"""The series folder in the case a TV library is written in.
Only a name with no capital at all is retitled: a box set that was
lower-cased on the way in would otherwise file `monk (2002)` among
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
exactly as it was written."""
if any(c.isupper() for c in head):
return head # the publisher wrote it with case; keep it
words = head.split()
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
for i, w in enumerate(words))
def season_episode(token):
"""(season, episode) an episode token states, or None when it states none."""
m = SEASON_RE.match(token)
if not m:
return None
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
def sidecars(path):
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
d, stem = os.path.split(path)
stem = os.path.splitext(stem)[0]
out = []
try:
entries = sorted(os.listdir(d))
except OSError:
return out
for e in entries:
p = os.path.join(d, e)
if p == path or os.path.islink(p) or not os.path.isfile(p):
continue
s, _ext = os.path.splitext(e)
if s == stem and not SKIP_SIDECAR.match(e):
out.append(p)
return out
def episodes_under(root):
"""(path, series, token) for every episode-shaped video file under root."""
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames
if not d.startswith(".") and d != "@eaDir")
for f in filenames:
p = os.path.join(dirpath, f)
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
continue
e = eng.ep_of(p)
if e:
yield p, e[0], e[1]
def walk(root, library):
"""Episode files under ROOT keyed by series, ignoring the destination."""
groups = collections.defaultdict(list)
lib = os.path.abspath(library)
for p, series, token in episodes_under(root):
if os.path.abspath(p).startswith(lib + os.sep):
continue
groups[series].append((p, token))
return groups
def holdings(library):
"""What the destination already carries: (series, season, episode) -> sizes.
The one expensive read here, and the one that matters. The movie library
holds a complete Monk box set that the TV library has held for years: 124
episodes, the same episode list, identical byte counts. Moving it would
have filed a second copy of a whole show into the library that already had
it. Byte count is the test on purpose — two encodes of one episode are two
files and both are wanted; the same bytes are the same file."""
out = collections.defaultdict(set)
for p, series, token in episodes_under(library):
se = season_episode(token)
if se is None:
continue
try:
out[(series, *se)].add(os.path.getsize(p))
except OSError:
continue
return out
def main():
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
groups = walk(root, library)
held_here = holdings(library)
mkdirs = {library}
rows, lines, skipped = set(), [], []
for series, items in sorted(groups.items()):
if len(items) < ARGS.min:
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
f"episode-shaped file{'s' if len(items) != 1 else ''},"
f" below {ARGS.min}: not a series")
continue
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
if y)
year = years.most_common(1)[0][0] if years else None
heads = collections.Counter(head_of(p) for p, _t in items)
head = heads.most_common(1)[0][0]
if year and YEAR_TAIL.search(head):
year = None
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
series_dir = os.path.join(library, name)
moved = held = 0
for path, token in items:
se = season_episode(token)
if se is None:
skipped.append(f"skipped: {path} — no season number in its name,"
" nothing here invents one")
continue
try:
size = os.path.getsize(path)
except OSError:
continue
if size in held_here.get((series, *se), ()):
held += 1
continue
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
mkdirs.update([series_dir, dst_dir])
base = episode_name(path)
rows.add((path, os.path.join(dst_dir, base)))
moved += 1
for side in sidecars(path):
rows.add((side, os.path.join(
dst_dir, clean(os.path.splitext(base)[0]
+ os.path.splitext(side)[1].lower()))))
if moved:
lines.append(f"{name}: {moved} episodes"
f"{'' if year else ' (no year in any folder name)'}"
f" -> {series_dir}")
if held:
lines.append(f"{name}: {held}"
f" {'episode' if held == 1 else 'episodes'} already in"
f" {library} at the same byte count — left there")
# An empty plan creates nothing, not even the library folder: a run that
# found no series must not be able to leave a stray directory behind.
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
with open(ARGS.out, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in dirs:
out.write(f"mkdir\t\t{d}\tdir\n")
for src, dst in sorted(rows):
out.write(f"move\t{src}\t{dst}\ttv\n")
for line in lines + skipped:
print(line)
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
if rows:
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
f" --root {os.path.dirname(root)}"
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
" (add --apply to run it)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,230 @@
---
type: reference
title: Media Relayout
description: "Census, plan and apply the movie library into Jellyfin format: the gates to read, and the rollback."
tags: [dankserver, media-server, jellyfin, plex, procedure]
timestamp: 2026-10-30T00:00:00Z
---
# Media Relayout
The movie library on [[Entropy Volume]] arrived as release folders —
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
the release. [[Jellyfin]] matches a movie by folder and file name, so the
release noise cost posters, extras and versions. These scripts turn that
library into Jellyfin's own shape — and move TV back out of it — and nothing
else in the repo touches a media file.
| Script | Touches media? | Output |
|---|---|---|
| `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
The seam is `tests/media_relayout.bats`; the target contract is quoted from
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
in `media-plan.py`'s docstring.
## Procedure
Run the census on the machine that mounts the library. `probe` is the slow step
(a header read per container) and is `nice`d — it is safe beside a live service
because it never competes for the GPU and never writes inside the library.
```bash
C=scripts/media-census.py
$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
$C probe --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
$C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
```
`plex` reads the Plex metadata cache (`--db` defaults to the path the
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
`mode=ro&immutable=1`: a live server is never locked or written.
Then plan — this reads only the three TSVs, so it costs seconds and can run
anywhere, including a laptop:
```bash
scripts/media-plan.py --root /mnt/entropy/Movies \
--enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
--plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
```
**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
is a file the plan could not name a film for, a pile that needs a README, or a
collision it refused. The printed gate lines are the go/no-go:
`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
0 and `unplanned=0` must appear in the report. Any non-zero number means a file
that would be filed wrong, and the plan is not run until it is understood.
```bash
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
```
An apply also leaves the release folders it emptied standing where they were —
hundreds of them at the library root — which is what `sweep` removes, dry-run
first, from the same journal.
After it lands, both servers need a scan or the change is invisible.
```bash
# Run this from the host shell through `podman exec`, never from a neighbour
# container: Plex's HTTPD closes without a single byte on any request that
# arrives from another container's netns (10.89.x.x reads as "empty reply", not
# as a 401). The token rides stdin, never argv, where any process on the host
# could read it out of /proc/<pid>/cmdline.
umask 077
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
podman exec -i plex sh -c 'read -r T
for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
-H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
done' </tmp/px.tok
```
[[Jellyfin]] has no credential on this box that can trigger a scan: the two keys
in its database are Seerr's, unused since 2026-09-01, and answer 401 to every
call. Mint one (*Dashboard → Advanced → API Keys*, Administrator) and store it in
`~/.config/secrets/jellyfin`, or press *Library → Scan All Libraries* in the web
UI. A scripted scan passes the key in a mode-600 curl config, never in a header
argument:
```bash
umask 077; printf 'header = "X-Emby-Token: %s"\n' "$(cat ~/.config/secrets/jellyfin)" >/tmp/jf.hdr
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n" \
-X POST http://127.0.0.1:8096/Library/Refresh
```
## TV in the movie library
A box set that arrives in the movie share stays there until someone moves it:
`media-plan.py` refuses to file TV and parks each series in
`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
walks the movie library itself (no census needed), groups episode-shaped names
by series, and writes a plan in the executor's own format:
```bash
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
--out /mnt/entropy/media-tv-plan.tsv
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
--root /mnt/entropy --apply
```
The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
because the publisher's spelling beats a re-derivation, with the year taken
from the box-set folder. What it refuses it prints and leaves alone:
- fewer than `--min` (5) files under one series name: one file named `Part 2`
is a film, not a series;
- an episode whose name states no season number — no invented `Season 01`;
- a destination two files both claim: the executor refuses the whole plan
before its first rename, and only Frank decides which copy is the keeper.
- an episode the destination library already holds at the same byte count. The
movie share carried a second copy of all 124 Monk episodes — the TV library
has had the same box set for years, every episode the same size, ~31 GiB of
it. The mover leaves such a series in place and says so; a duplicate is
deleted or kept by Frank, never copied into one library by a tool.
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
## Laws
- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
database keeps the release-folder path, so playback fails `Couldn't find the
file to stream` while the scanner works through the section. Never read such a
404 as a lost file: the file is where the journal says it is.
- **Plex's matched title outranks the filename**, and a Plex year of `0` means
*indexed but never matched* — trusting it named folders `Title (0)`, which is
a Jellyfin item with no year, no poster and no identity. A year that is not a
plausible year is no information.
- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
`V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
levels deep that neither matcher reads as a movie.
- **A folder inside a movie's folder is bonus territory even when its name is
misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
`Extras`; the vocabulary rule never fired, the featurette became a movie
folder, and its words then out-scored the real film and drew that box set's
other extras into it. The rule that replaced it needs no metadata: the folder
holding the longest film owns the shorter video filed under it, unless that
file's own name carries a different year — which is another film speaking.
- **Matching words are split on everything that is not a letter or a digit.**
Splitting on separators but not on `/` left tokens like `(1960)/extras` that
no other name could ever agree with, so a film could not be found by the
folder holding it, and extras drifted into a neighbour. A bare number counts
only when it could be a release year, so `1080` and `2160` never pose as
agreement between two names.
- **A file whose name carries no words is bonus material, not a folder.**
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
is not a movie. It becomes a review line telling a human to name the file.
- **The executor renames and nothing else.** `rename(2)` cannot cross a
filesystem, so a plan that would is refused rather than silently copying tens
of terabytes.
- **The journal is a write-ahead log: the intent is fsynced before the rename.**
An interrupted run therefore leaves only two states — an intent whose move
happened, and an intent whose move never ran — and the next run finishes the
job without moving anything twice. A file that moved with no journal line did
not move through this tool, and the tool refuses to guess on its behalf.
- **The journal lives BESIDE the library root, never inside it** (default:
`media-relayout.jsonl` next to the root): a log inside the library is a file
Jellyfin indexes.
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
duplicate destination stops the run *before the first rename*, so the operator
fixes one thing and re-runs instead of untangling a half-applied plan.
`--force` relaxes only the opinion guards; it never permits overwriting.
- **A sweep deletes only what the journal proves the run emptied.** A folder is
eligible when a journalled file moved out of it or out of something under it,
it now holds nothing but empty directories, no journalled destination lives
beneath it, and it is not a folder the run created. A folder nobody moved —
someone's placeholder — cannot be selected at all, and a symlink anywhere
inside disqualifies the subtree, because the thing behind a link is not this
tool's to delete. Every removal is journalled as `rmdir` and `rollback`
recreates the folder, so the sweep is undone by the same command as the moves.
Proof carries downward: inside a folder the journal proves emptied, every
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
move source itself, so 146 of them held their emptied parents on disk after
the first sweep.
A failed `rmdir` is not a removal: only a folder that is really gone leaves
scope, so the next sweep finishes what a permission problem stopped.
- **An episode-shaped name marks a SERIES, not a film — and the series is the
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
`Monk`. Keyed on the whole title instead, every episode became a group of one
and never reached the five-episode size that calls something a series, so a
box set was filed as 124 movies in 124 folders. The threshold still earns its
keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
in `_TV needs a TV library - <series>` for a human to move to a TV source —
the movie tool never files TV.
- **One journal is one undo.** The TV move runs on its own journal, and with
`--root` set to the share that holds BOTH libraries: the executor refuses any
path outside `--root`, and a journal shared with the movie relayout would
undo both runs when only one of them is wanted.
## Rollback
One command, from the journal of the run:
`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
any folder a sweep removed, restores every source path, removes only the folders
that run created *and that are now empty*, and refuses everything if any moved
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
## See Also
[[Jellyfin]], [[Plex]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
## Citations
- Jellyfin Movie documentation — folder/file naming, extras subfolders,
`Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
<https://jellyfin.org/docs/general/server/media/movies>
+1
View File
@@ -51,6 +51,7 @@ is a locator: one row, one clause, no numbers.
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. |
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
| [[Media Relayout]] | Census, plan and apply the movie library into Jellyfin format: the gates to read, and the rollback. |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
+620
View File
@@ -0,0 +1,620 @@
#!/usr/bin/env bats
# Seam: the movie-library relayout — scripts/media-census.py (the read-only
# census), scripts/media-plan.py (the plan) and scripts/media-apply.py (the only
# thing that moves files).
#
# The library is 8k files on a 100 %-full array, so the two things this seam
# exists to catch are the ones that cost a re-run of a 20-minute census: a file
# that ends up in the WRONG movie folder, and a move that destroys something.
# Everything below runs on a synthetic library in the test's own tmpdir: no
# ffprobe, no network, no server. The probe step is the one part a fixture
# cannot run (it opens real containers), so the tests feed the planner the
# probed TSV in the column order scripts/media-census.py exports — which pins
# that contract as a side effect.
#
# The plan's own gate lines (unplanned, left-unmoved, double-claim, occupied)
# are asserted as output, because they are how the operator learns the plan is
# trustworthy before pointing the executor at 24 TB.
load helpers
PLAN="$REPO_ROOT/scripts/media-plan.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
TV="$REPO_ROOT/scripts/media-tv.py"
# probe_row FILE DUR [RES] [AUDIO] [SUBS] — one probed-TSV row: the twelve
# columns media-census.py probe writes (bytes, path, dur, vcodec, res, fps,
# vbit, pix, audio, subs, spare, fmt).
probe_row() {
local p="$1" dur="$2" res="${3:-1920x1080}" aud="${4:-aac+aac}" subs="${5:-0}"
printf '%s\t%s\t%s\thevc\t%s\t23.976\t5000000\t8bit\t%s\t%s\t0\tmatroska,webm\n' \
"$(stat -c %s "$p")" "$p" "$dur" "$res" "$aud" "$subs" >> "$PROBED"
}
# plex_row FILE TITLE YEAR — one Plex metadata row; the planner reads these
# three columns and treats a row without a plausible year as no match at all.
plex_row() {
printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$PLEX"
}
# at PATH SIZE — a file with deterministic content, big enough to matter.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
setup() {
LIB="$BATS_TEST_TMPDIR/Movies"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
PLEX="$BATS_TEST_TMPDIR/plex.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
TVLIB="$BATS_TEST_TMPDIR/TVShows"
TVPLAN="$BATS_TEST_TMPDIR/tv.tsv"
TVJ="$BATS_TEST_TMPDIR/tv.jsonl"
: > "$PROBED"
printf 'file\tleaf_title\tleaf_year\n' > "$PLEX"
# Two films that share a box set and a word, plus their bonus material:
# the shape that made one film's featurettes land in another's folder.
at "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4000
at "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 3300000
at "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 900000
at "$LIB/Alien (1979)/subs/3_Eng.srt" 40000
at "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 5000000
at "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 2400000
# A long extra: over 45 minutes, so only the folder may call it bonus.
at "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 2600000
# A misspelled bonus folder: no bonus vocabulary, so only the folder that
# holds the feature can decide that this is not a film.
at "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2500000
# A Plex title that carries "/" and a Plex row that was never matched.
at "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 4200000
at "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 3100000
# Junk the layout has to park somewhere other than a movie folder.
at "$LIB/.DS_Store" 6148
at "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv" 220
# A bonus file whose name is a number: never a movie folder named "1".
at "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 2700000
# A TV box set sitting in the movie library. This one names each file
# "Show - SxxEyy - Episode Title", so the episode title comes AFTER the
# marker; that convention is what made 124 episodes of one series look like
# 124 different films, each under its own threshold.
for e in 01 02 03 04 05; do
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
# The same trouble named the other way round, with no episode title at all.
for e in 01 02 03 04 05; do
at "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3000000
done
# A film whose own name carries a bare "Part 2": one episode-shaped token is
# not a series, and it must not be dragged into the TV pile.
at "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 9000000
probe_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4700
probe_row "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 2700
probe_row "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 30
probe_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 4800
probe_row "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 1200
probe_row "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 3000
probe_row "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2600
probe_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 5700
probe_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 6300
probe_row "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 5220
for e in 01 02 03 04 05; do
probe_row "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 2600
probe_row "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3300
done
probe_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 6200
plex_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" "Alien" "1979"
plex_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" "Marnie" "1964"
plex_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" "V/H/S/2" "2013"
# An unmatched Plex row: the leaf title echoes the file and the year is 0.
plex_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" "AMERICAN POP" "0"
plex_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" \
"The Godfather Part 2" "1974"
export MEDIA_ROOT="$LIB"
run python3 "$CENSUS" enumerate --root "$LIB" --out "$ENUM"
[ "$status" -eq 0 ]
}
# plan — run the planner over the fixture, keeping its plan TSV, its exception
# report and the gate lines it prints (that stdout is the operator's evidence
# that the plan accounts for every file, so the tests read it too).
plan() {
PLANOUT="$BATS_TEST_TMPDIR/plan.out"
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --probed "$PROBED" \
--plex "$PLEX" --out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$PLANOUT"
}
# dst_of SRC — the destination the plan gives one file.
dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$OUT" "$1"
}
# tv_plan — the TV mover over the fixture library; its stdout is the operator's
# census of what it called a series and what it refused to touch.
tv_plan() {
run python3 "$TV" --root "$LIB" --library "$TVLIB" --out "$TVPLAN"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$BATS_TEST_TMPDIR/tv.out"
}
# tv_apply — that plan through the same executor. The destination is a sibling
# share, so the plan is applied at the root that holds both libraries.
tv_apply() {
run python3 "$APPLY" apply --plan "$TVPLAN" --root "$BATS_TEST_TMPDIR" \
--journal "$TVJ" --apply
[ "$status" -eq 0 ]
}
# tv_dst_of SRC — the destination the TV plan gives one file, empty if none.
tv_dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$TVPLAN" "$1"
}
@test "a film lands in Title (year)/Title (year).mkv, named by Plex" {
plan
[ "$(dst_of "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv")" \
= "$LIB/Alien (1979)/Alien (1979).mkv" ]
}
@test "a Plex title that carries a slash becomes one folder, not a nested tree" {
plan
local d; d="$(dst_of "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv")"
[ "$(dirname "$d")" = "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$LIB/V" ]
# the folder must be ONE mkdir: an un-sanitized name would make the
# executor build Movies/V/H/S/2 (2013) instead
run grep -F "mkdir" "$OUT"
[[ "$output" == *"$(printf '%s' "$LIB/V-H-S-2 (2013)")"* ]]
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != *"$LIB/V/"* ]]
}
@test "an unmatched Plex row never names a folder with a zero year" {
plan
run awk -F'\t' '{print $3}' "$OUT"
[[ "$output" != *"(0)"* ]]
# the filename's own year carries the folder, the folder and the file share
# the name (Jellyfin's rule), and the film sits at the library root rather
# than in an unsorted pile
local d; d="$(dst_of "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB" ]
[[ "$(basename "$(dirname "$d")")" == *" (2023)" ]]
[ "$(basename "$d")" = "$(basename "$(dirname "$d")").mkv" ]
}
@test "a bonus folder keeps its featurettes inside its own film, not a neighbour's" {
plan
local d
d="$(dst_of "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
d="$(dst_of "$LIB/Alien (1979)/Extras/Extra - Logo.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
# the feature-length extra is bonus too: it sits in a bonus folder and no
# metadata matched it, so it may not become a movie of its own
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a folder inside a movie folder is bonus even when its name is misspelled" {
plan
local d; d="$(dst_of "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a subtitle follows the film of the folder it sits in" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/subs/3_Eng.srt")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
}
@test "macOS sidecars are parked in one pile, never inside a movie folder" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
d="$(dst_of "$LIB/.DS_Store")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
grep -q "AppleDouble" "$EXC"
}
@test "a file whose name is a number becomes a review line, not a folder named 1" {
plan
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != "$LIB/1"$'\n'* ]] && [[ "$output" != "$LIB/1" ]]
[[ "$output" != *"$LIB/1/"* ]]
grep -q "UNNAMED" "$EXC"
}
@test "a box set in the movie library is quarantined, not filed as movies" {
plan
local d
d="$(dst_of "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ monk* ]]
d="$(dst_of "$LIB/Big Little Lies Season 1/Big Little Lies S01E01.mp4")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ big\ little\ lies* ]]
# one series is one pile, whatever its episodes are called
run python3 -c '
import re, sys
pile = {}
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[0] == "move":
m = re.search(r"/_TV needs a TV library - ([^/]*)/", f[2] + "/")
if m:
pile[m.group(1)] = pile.get(m.group(1), 0) + 1
for k in sorted(pile):
print(f"{k}={pile[k]}")
' "$OUT"
[ "${#lines[@]}" -eq 2 ]
[[ "${lines[0]}" == big\ little\ lies*=5 ]]
[[ "${lines[1]}" == monk*=5 ]]
}
@test "a film named Part 2 is not called a series" {
plan
[ "$(dst_of "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv")" \
= "$LIB/The Godfather Part 2 (1974)/The Godfather Part 2 (1974).mkv" ]
}
@test "the TV mover files a box set into the TV library in Jellyfin shape" {
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ ! -e "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
# films are not the TV mover's business
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "the TV mover leaves a film whose name only looks like an episode" {
tv_plan
tv_apply
local gf="$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv"
[ "$(tv_dst_of "$gf")" = "" ]
[ -e "$gf" ]
grep -q "Godfather.*not a series" "$BATS_TEST_TMPDIR/tv.out"
}
@test "the TV mover keeps an episode's own subtitle with the episode" {
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.srt" 40000
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.srt" ]
}
@test "a second TV move finds nothing left to move" {
tv_plan
tv_apply
tv_plan
grep -q "0 moves" "$BATS_TEST_TMPDIR/tv.out"
}
@test "a show whose own name carries the year is not given it twice" {
local d="$LIB/Leah Remini Scientology and the Aftermath (2016)/Season 1"
for e in 01 02 03 04 05; do
at "$d/Leah Remini Scientology and the Aftermath (2016) - S01E$e - Episode.mkv" 3000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016)/Season 01/Leah Remini Scientology and the Aftermath (2016) - S01E01 - Episode.mkv" ]
[ ! -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016) (2016)" ]
}
@test "an all-lowercase series name is written the way the library writes names" {
for e in 01 02 03 04 05; do
at "$LIB/downton abbey 2010/Season 1/downton abbey - s01e$e - episode title.mkv" 3000000
done
tv_plan
tv_apply
# the FOLDER is titled; the file keeps the name the library already holds
[ -e "$TVLIB/Downton Abbey (2010)/Season 01/downton abbey - s01e01 - episode title.mkv" ]
}
@test "the mover leaves an episode the TV library already holds" {
local src="$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv"
for e in 01 02 03 04 05; do
at "$TVLIB/Monk (2002)/Season 01/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
tv_plan
[ -z "$(tv_dst_of "$src")" ]
[ -e "$src" ]
grep -q "Monk.*already in" "$BATS_TEST_TMPDIR/tv.out"
}
@test "an episode the TV library holds in another encode is still moved" {
for e in 01 02 03 04 05; do
at "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E$e 1080p.mp4" 4000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01 1080p.mp4" ]
}
@test "every file in the census is placed or kept — nothing is dropped" {
plan
python3 - "$ENUM" "$OUT" <<'PY'
import sys
enum = {l.rstrip("\n").split("\t")[2] for l in open(sys.argv[1])}
placed = set()
for line in open(sys.argv[2]):
f = line.rstrip("\n").split("\t")
if f[0] in ("move", "keep"):
placed.add(f[1])
missing = enum - placed
assert not missing, f"{len(missing)} census files with no plan row: {sorted(missing)[:3]}"
PY
grep -q "left-unmoved=0" "$PLANOUT"
grep -q "double-claim=0" "$PLANOUT"
grep -q "occupied=0" "$PLANOUT"
grep -q "bad-names=0" "$PLANOUT"
grep -q "unplanned=0" "$EXC"
}
# ---------------------------------------------------------------- executor
# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
apply_plan() {
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
}
@test "a run without --apply says what it would do and touches nothing" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# a dry run may not even build the folders: an operator must be able to
# run it against the live library with no --apply and change nothing
[ ! -d "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$JOURNAL" ]
}
@test "apply moves every planned file, keeps its bytes, and journals each move" {
plan
apply_plan
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
local planned moved
planned=$(awk -F'\t' '$1=="move"' "$OUT" | wc -l)
moved=$(python3 -c 'import json,sys
print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "$JOURNAL")
[ "$moved" -eq "$planned" ]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
}
@test "a target already holding a different file stops the whole run" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
at "$LIB/Alien (1979)/Alien (1979).mkv" 7
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"occupied"* ]]
# nothing moved at all: the other films are still where the census saw them
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ ! -e "$JOURNAL" ]
}
@test "a plan row whose source has gone stops the run instead of inventing a move" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
rm "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "a run interrupted between the journal write and the move finishes the job" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
# The journal is written before each move, so an interruption leaves two
# states and only two: an intent whose move happened, and an intent whose
# move never ran. Both are simulated here by hand.
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"][:3]
with open(sys.argv[2], "w") as fh:
for i, r in enumerate(moves):
fh.write(json.dumps({"op": "move", "src": r[1], "dst": r[2],
"bytes": os.path.getsize(r[1])}) + "\n")
if i < 2: # the first two moves actually ran
os.makedirs(os.path.dirname(r[2]), exist_ok=True)
os.rename(r[1], r[2])
PY
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"]
journal = [json.loads(l) for l in open(sys.argv[2])]
seen = [j["dst"] for j in journal]
assert len(seen) == len(set(seen)), "a destination was journalled twice"
for r in moves:
assert os.path.exists(r[2]) and not os.path.exists(r[1]), r[2]
PY
}
@test "rollback puts every file back where the census found it" {
plan
apply_plan
[ "$status" -eq 0 ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# the folders the run created are empty afterwards and must not litter the
# library: this one never existed before the plan ran
[ -d "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/V-H-S-2 (2013)" ]
}
@test "rollback refuses when a moved file has gone missing" {
plan
apply_plan
[ "$status" -eq 0 ]
rm "$LIB/Alien (1979)/Alien (1979).mkv"
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
# the refusal is all-or-nothing: another film did not move back
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
}
# ---------------------------------------------------------------- sweep
#
# A relayout leaves the release folders it emptied standing at the library root:
# 1675 of them after the live run, each holding nothing but empty subfolders. The
# sweep pass removes only folders THIS RUN emptied — proven from the journal,
# never from "it looks empty" — so a folder someone made by hand is untouchable,
# and every removal rides the same rollback as the moves.
@test "sweep removes only the folders this run emptied" {
plan
apply_plan
mkdir -p "$LIB/A Empty Placeholder" # someone's folder, never a move source
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept"* ]]
[ ! -e "$LIB/AMERICAN POP" ]
[ ! -e "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/Alien (1979)/Featurettes" ]
[ ! -e "$LIB/Alien (1979)/subs" ]
[ ! -e "$LIB/Marnie (1964)/Exrtas" ]
# a folder the run never moved a file out of survives, empty or not
[ -d "$LIB/A Empty Placeholder" ]
}
@test "a swept folder carries the empty junk folders inside it" {
mkdir -p "$LIB/AMERICAN POP/Screens" # screenshots, never a move source
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"0 left in place"* ]]
[ ! -e "$LIB/AMERICAN POP/Screens" ]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "a sweep whose rmdir failed tries those folders again" {
plan
apply_plan
mkdir -p "$LIB/AMERICAN POP/Screens"
chmod 500 "$LIB/AMERICAN POP" # readable, but nothing can be removed
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"left in place"* ]]
[ -e "$LIB/AMERICAN POP/Screens" ]
chmod 700 "$LIB/AMERICAN POP" # the operator cleared the blocker
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" != *"left in place: "* ]]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "sweep without --apply reports the husks and changes nothing" {
plan
apply_plan
local before jlines
before=$(find "$LIB" -type d | wc -l)
jlines=$(wc -l < "$JOURNAL")
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
[ "$(wc -l < "$JOURNAL")" -eq "$jlines" ]
[ -e "$LIB/AMERICAN POP" ]
}
@test "a folder that still holds content is never swept" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# the film folder, the folder the run created, and the piles all stay
[ -d "$LIB/Marnie (1964)" ]
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
[ -d "$LIB/V-H-S-2 (2013)" ]
[ -d "$LIB/_unsorted bonus/extras" ]
}
@test "rollback after a sweep brings the folders and the files back" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ ! -e "$LIB/AMERICAN POP" ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" ]
[ -e "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
}
@test "a second sweep has nothing left to do and says so" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
local before
before=$(find "$LIB" -type d | wc -l)
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept: 0"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
}
@test "sweep never follows a symlink and never removes the folder it guards" {
plan
apply_plan
mkdir -p "$LIB/Elsewhere"
at "$LIB/Elsewhere/keep.txt" 10
ln -s "$LIB/Elsewhere" "$LIB/AMERICAN POP/pointer"
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# a link is content, not nothing: the husk it sits in survives untouched
[ -d "$LIB/AMERICAN POP" ]
[ -L "$LIB/AMERICAN POP/pointer" ]
[ -e "$LIB/Elsewhere/keep.txt" ]
}
@test "sweep refuses a journal that belongs to another library" {
plan
apply_plan
mkdir -p "$BATS_TEST_TMPDIR/Other"
run python3 "$APPLY" sweep --root "$BATS_TEST_TMPDIR/Other" \
--journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"outside"* ]]
[ -d "$LIB/AMERICAN POP" ]
}