dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(media): measure a series per episode, and let the tag outrank a borrowed id

Browse Source 
media-tvdupes.py censuses a TV library by episode key and never by folder. 10 of
the 25 shows that looked duplicated in the 2026-11-02 census held complementary
halves (Game of Thrones Complete carries no S01 or S08, Modern Family S05 is the
only copy of season 5), so a folder-level verdict would have staged some 340 GB
of unique episodes to recover 54 GiB of true copies.

Show identity comes from the folder's [tvdbid-N] tag, from the tvshow.nfo the
metadata view exposes, or from an operator --ids map. A derived id that
contradicts a tag is dropped and reported, never unioned: Jellyfin's provider
table carries a 2026 special's id on the folder tagged [tvdbid-73838], and
unioning the two makes one show of a 2000 series and a 2026 special whose bigger
files then dominate the original's pilot. Files named by an id key only when the
server's own episode table (the --witness) names them, so a bare number is never
mistaken for an episode.

Seam: tests/media_tvdupes.bats — 12 assertions on which files land in the plan
and which the tool refuses to touch, on a synthetic library.
This commit is contained in:
Frank committed 2026-11-02 00:44:46 -08:00
1 parent a25f139281
commit 9f9043c037
5 files changed
+790 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/media-tvdupes.py
Executable
+473
View File
@@ -0,0 +1,473 @@
#!/usr/bin/env python3
"""media-tvdupes.py — find duplicate TV episodes and plan their staging, one
EPISODE at a time.
A TV library grows the same show into several folders: a tagged season box, an
untagged packager drop, a `Complete` folder that is not complete. The folders
look like copies of each other and are not, so the unit of comparison is the
episode and never the folder. A file is droppable only when EVERY episode it
carries is also held, somewhere else in the same show, by a file at least as
large — Frank's rule: the same episode in two places keeps the larger one. A
file that holds one episode nothing else has stays, whatever else it carries.
It writes a plan and moves nothing; `scripts/media-apply.py` is the only thing
that renames. The plan stages into a dated hold, so the pass is undoable from
the journal and frees no space until the hold is purged.
python3 scripts/media-tvdupes.py --library /mnt/entropy/TVShows \
--hold /mnt/entropy/.AGENTTRASH/20261102-tv-duplicates \
--out /mnt/entropy/tv-dupes-plan.tsv
python3 scripts/media-apply.py apply --plan /mnt/entropy/tv-dupes-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/tv-dupes.jsonl \
--metadata-view /var/lib/media-meta/view --apply
python3 scripts/media-apply.py verify --plan /mnt/entropy/tv-dupes-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/tv-dupes.jsonl
python3 scripts/media-apply.py sweep --root /mnt/entropy \
--journal /mnt/entropy/tv-dupes.jsonl --apply # empty folders only
scripts/media-view.sh reap # ghost sidecars above
Then both servers have to be told: the plan moves files on the raw array, and no
watcher crosses the metadata view — [[Media Relayout]] owns the refresh order.
Guards, and why each one is here:
* **Identity before geometry.** Two folders holding the same words are not the
same show: the 2026-11-02 census found `Avatar: The Last Airbender` (1999,
tvdb 74852) beside the 2024 remake (385925), `Star Trek: Voyager` beside
`Star Trek: The Animated Series`, and a German thriller Jellyfin had titled
`Millennium (2022) [tvdbid-472933]` beside `Millennium (1996)`. A folder's
identity is read from three places, in this order: its `[tvdbid-N]` tag, the
`tvshow.nfo` Jellyfin wrote — which lives in the metadata view, not on the
array, and is what `--meta` reads — and a `folder<TAB>id` map passed with
`--ids`. The tag outranks the derived layers: an id that contradicts a folder's
own tag is reported and ignored, never unioned — measured 2026-11-02, Jellyfin's
provider table carried a 2026 special's id on a folder tagged `[tvdbid-73838]`,
and unioning them makes one show of the 2000 series and the special, whose bigger
files then dominate the original's pilot. Two folders are one show when they share
an id, or when neither has one and name and year both agree. Two disjoint id sets
is an absolute stop: the pair is reported as a conflict and nothing under either is
staged. Same name with nothing to decide on is reported unresolved — the fix is the
tag, not a guess.
* **A folder-level verdict destroys content.** 10 of the 25 shows that looked
duplicated held COMPLEMENTARY halves: `Game of Thrones Complete` carries no
S01 and no S08, `Modern Family S05` is the only copy of season 5, `Loiter
Squad Uncensored` is the only copy of season 1. Judged per folder, 20 shows
and some 340 GB of unique episodes would have been staged; judged per episode
the true figure was a sixth of that.
* **A bare number is never an episode.** `256491.mp4` may be an episode id, a
poster, or a season, so a name that states no `SxxEyy` / `1x04` marker is
unkeyed: never staged, and counted in the report. A media server's own episode
table does know those files — pass it with `--witness` and its rows key
whatever the server has matched.
* **The audit re-reads the disk.** Sizes and the survivor's existence are
checked again when the plan is written, because the census is a snapshot: a
file that grew, vanished, or landed in the hold in between must fail the plan,
not surprise the executor.
* **A sidecar that shares an episode's stem follows it.** Subtitles and `.nfo`
left at the old coordinates are ghost bait for `media-view.sh reap`.
* A multi-episode file (`S01E01-02`) is dominated only when every episode in its
range is held elsewhere — and it competes for the survivor seat on each of
them, because the byte law compares files. A pack that states its range in
the name is a name both media servers parse the same way, so the pack winning
against two single-episode rips is the law applied, not a misread.
"""
import argparse
import collections
import importlib.util
import json
import os
import re
import sys
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(eng)
# The executor's definition of a media file, plus the two the title engine omits:
# an audit that disagrees with media-apply.py is a plan that dies at its
# preflight instead of at the census.
VIDEO = tuple(sorted(set(eng.VIDEO_SUFFIX) | {".ts", ".wmv"}))
TAG_RE = re.compile(r"\[\s*tvdbid[\s._-]*(\d+)\s*\]", re.I)
YEAR_RE = re.compile(r"[(\[]\s*((?:19|20)\d\d)\s*[)\]]")
SEASON_EP = re.compile(r"(?i)\bs(?:eason)?[\s._-]*(\d{1,2})[\s._-]*e(?:p(?:isode)?)?"
r"[\s._-]*(\d{1,3})(?:[\s._-]*(?:-|to|\+)[\s._-]*e?"
r"[\s._-]*(\d{1,3}))?")
NXN = re.compile(r"(?i)(?<!\d)(\d{1,2})x(\d{1,3})(?!\d)")
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--library", default="/mnt/entropy/TVShows",
help="the TV library to census (never written)")
ap.add_argument("--hold", required=True,
help="dated trash the plan stages into, e.g. "
"/mnt/entropy/.AGENTTRASH/20261102-tv-duplicates")
ap.add_argument("--out", default=None,
help="plan TSV for scripts/media-apply.py (default: "
"tv-dupes-plan.tsv beside --library)")
ap.add_argument("--witness", default=None,
help="TSV of path<TAB>season<TAB>episode out of a media "
"server's episode table; keys files whose names carry"
" no marker")
ap.add_argument("--meta", default=None,
help="the merged metadata view (on the host "
"/var/lib/media-meta/view): tvshow.nfo lives in the"
" upper layer, so this is where an untagged folder's"
" tvdb id is readable")
ap.add_argument("--ids", default=None,
help="TSV of folder<TAB>tvdbid, one line per folder, for"
" identity a lookup settled and the folder name does"
" not state")
ap.add_argument("--json", default=None,
help="also dump the analysis as JSON")
return ap.parse_args()
ARGS = parse_args()
def episode_keys(name):
"""Every (season, episode) an episode name states.
`S01E03`, `Season 1 E3`, `1x04`, and a range `S01E01-02`, which expands so a
two-in-one file is only droppable when both halves are held elsewhere. A
bare number states nothing and yields nothing."""
keys = set()
for m in SEASON_EP.finditer(name):
s, first = int(m.group(1)), int(m.group(2))
keys.add((s, first))
if m.group(3):
keys.update((s, e) for e in range(first + 1, int(m.group(3)) + 1))
keys.update((int(m.group(1)), int(m.group(2))) for m in NXN.finditer(name))
return keys
def folder_identity(folder):
"""(fold_sep name, year, tvdb ids) for a top-level show folder.
Tag and year come in both orders — `Name (2008) [tvdbid-84859]` and `Name
[tvdbid-84859] (2008)` are both in the library — so neither is assumed to
sit at the end, and both are cut out of the name folders compare on."""
ids = frozenset(int(x) for x in TAG_RE.findall(folder))
year = None
for _ in range(3):
y = YEAR_RE.search(folder)
if not y:
break
year = year or y.group(1)
folder = TAG_RE.sub(" ", f"{folder[:y.start()]} {folder[y.end():]}")
return eng.fold_sep(TAG_RE.sub(" ", folder)), year, ids
def nfo_id(path):
"""The tvdb id a `tvshow.nfo` states, or None.
Three shapes are in the library: Jellyfin's `<uniqueid type="tvdb">`, Kodi's
`<tvdbid>`, and a packager's `<id>` + `<id_type>` pair; some carry the id
only in the title line as `[tvdbid-N]`."""
try:
with open(path, encoding="utf-8", errors="replace") as fh:
text = fh.read(16384)
except OSError:
return None
for rx in (r"<tvdbid\s*>\s*(\d+)\s*<",
r"<uniqueid[^>]*type\s*=\s*[\"']?tvdb[^>]*>\s*(\d+)\s*<",
r"<id\s*>\s*(\d+)\s*</id>\s*<id_type>\s*(?:tvdb|thedotdb)",
r"\[\s*tvdbid[\s._-]*(\d+)\s*\]"):
m = re.search(rx, text, re.I)
if m:
return int(m.group(1))
return None
def collect_ids(library, meta, ids_file):
"""top-level folder -> {tvdb ids} from the nfo layer and an operator map.
`meta` is the merged metadata view: Jellyfin's sidecars live in the upper
layer, so a folder's `tvshow.nfo` is not on the raw array. Reading the id
from there is what makes an untagged packager folder identifiable at all —
without it the pair stays unresolved and nothing under it is staged."""
out = collections.defaultdict(set)
if meta:
try:
tops = sorted(os.listdir(library))
except OSError:
tops = []
for folder in tops:
i = nfo_id(os.path.join(meta, folder, "tvshow.nfo"))
if i:
out[folder].add(i)
if ids_file:
with open(ids_file, encoding="utf-8", errors="replace") as fh:
for line in fh:
cols = line.rstrip("\n").split("\t")
if len(cols) < 2 or not cols[1].strip().isdigit():
continue
out[cols[0]].add(int(cols[1].strip()))
return out
def witness_rows(path):
"""absolute path -> {(season, episode)} from a media server's episode table."""
out = collections.defaultdict(set)
if not path:
return out
with open(path, encoding="utf-8", errors="replace") as fh:
for line in fh:
cols = line.rstrip("\n").split("\t")
if len(cols) < 3 or not cols[0].startswith("/"):
continue
try:
out[os.path.abspath(cols[0])].add((int(cols[1]), int(cols[2])))
except ValueError:
continue
return out
def sidecars(path):
"""The files beside PATH carrying its exact stem — subtitles, nfo, artwork."""
folder, stem = os.path.split(path)
stem = os.path.splitext(stem)[0]
out = []
try:
entries = sorted(os.listdir(folder))
except OSError:
return out
for e in entries:
p = os.path.join(folder, e)
if p == path or os.path.islink(p) or not os.path.isfile(p):
continue
if os.path.splitext(e)[0] == stem:
out.append(p)
return out
def scan(library, hold, witness):
"""Every video file under LIBRARY: relative path, size, episode keys.
A file keys from the witness when it has a row (a server matched it), and
from its own name otherwise; with neither it is unkeyed."""
files = []
for dirpath, dirnames, filenames in os.walk(library):
dirnames[:] = sorted(d for d in dirnames
if not d.startswith(".") and d != "@eaDir"
and os.path.abspath(os.path.join(dirpath, d)) != hold)
for f in sorted(filenames):
p = os.path.join(dirpath, f)
if os.path.islink(p) or not f.lower().endswith(VIDEO):
continue
try:
size = os.path.getsize(p)
except OSError:
continue
files.append({"path": os.path.abspath(p),
"rel": os.path.relpath(os.path.abspath(p), library),
"size": size,
"keys": set(witness.get(os.path.abspath(p))
or episode_keys(os.path.splitext(f)[0]))})
return files
def show_groups(folders, extra_ids):
"""(show groups, conflicts, unresolved, disagreements) over top-level folders.
A folder's id is its `[tvdbid-N]` tag; the nfo layer and `--ids` fill a folder
only when it carries no tag. The rule, in order of trust: an id in common makes
one show; two disjoint id sets are two shows, reported as a conflict and never
merged; no id on either side merges only on name AND year (the 1999 miniseries
and the 2015 series are not one show); one side ided and the other bare is
unresolved, because the fix is the tag.
The derived layers never overrule a tag. Measured 2026-11-02: Jellyfin's
provider table carried a 2026 special's tvdb id on the folder tagged
`[tvdbid-73838]`, and unioning the two made one show of the 2000 series and the
special — whose bigger files then dominate the original's pilot. A contradiction
is reported instead, because the tag is the operator's word and the provider
table is a guess that can be wrong.
"""
names = {f: folder_identity(f) for f in folders}
tags = {f: set(names[f][2]) for f in folders}
derived = {f: set(extra_ids.get(f, ())) for f in folders}
disagree = {f: (sorted(tags[f]), sorted(derived[f] - tags[f]))
for f in folders if tags[f] and derived[f] - tags[f]}
ids = {f: (tags[f] or derived[f]) for f in folders}
parent = {f: f for f in folders}
def find(f):
while parent[f] != f:
parent[f] = parent[parent[f]]
f = parent[f]
return f
conflicts, unresolved = [], []
order = sorted(folders)
for i, a in enumerate(order):
for b in order[i + 1:]:
na, ya, _ia = names[a]
nb, yb, _ib = names[b]
if na != nb:
continue
ia, ib = ids[a], ids[b]
if ia & ib:
parent[find(b)] = find(a)
elif ia and ib:
conflicts.append((a, b, f"tvdb {sorted(ia)} vs {sorted(ib)}"))
elif not ia and not ib and ya and ya == yb:
parent[find(b)] = find(a)
elif not ia and not ib:
conflicts.append((a, b,
f"same name, years {ya or '?'} vs {yb or '?'}"))
else:
unresolved.append((a, b, "one folder carries a tvdb id,"
" the other none"))
groups = collections.defaultdict(list)
for f in order:
groups[find(f)].append(f)
return sorted(groups.values()), conflicts, unresolved, disagree
def rank(rec):
"""Survivor order: biggest, then shortest path, then lexicographic.
A strict total order per key means a key's winner is never itself dominated,
so no chain of staging can cycle and exactly one copy of a byte-identical
pair is staged."""
return (-rec["size"], len(rec["path"]), rec["path"])
def main():
library = os.path.abspath(ARGS.library)
hold = os.path.abspath(ARGS.hold)
out = ARGS.out or os.path.join(os.path.dirname(library), "tv-dupes-plan.tsv")
if not os.path.isdir(library):
sys.exit(f"refused: {library} is not a directory")
if hold == library or hold.startswith(library + os.sep):
sys.exit(f"refused: --hold {hold} sits inside the library; a hold the"
" census walks is a plan that eats its own output")
files = scan(library, hold, witness_rows(ARGS.witness))
folders = sorted({r["rel"].split(os.sep)[0] for r in files})
ids = collect_ids(library, ARGS.meta, ARGS.ids)
groups, conflicts, unresolved, disagree = show_groups(folders, ids)
by_show = collections.defaultdict(list)
for members in groups:
for r in files:
if r["rel"].split(os.sep)[0] in members:
by_show[members[0]].append(r)
rows, audits, report, shows_json = [], [], [], []
for show in sorted(by_show):
recs = by_show[show]
holders = collections.defaultdict(list)
for r in recs:
for k in r["keys"]:
holders[k].append(r)
winner = {k: min(v, key=rank) for k, v in holders.items()}
dominated = [r for r in recs if r["keys"]
and all(winner[k] is not r for k in r["keys"])]
if not dominated:
continue
dup_keys = sorted({k for r in dominated for k in r["keys"]})
sole = sum(1 for v in holders.values() if len(v) == 1)
staged_folders = {os.path.dirname(r["rel"]) for r in dominated}
for r in dominated:
for k in r["keys"]:
t = winner[k]
try:
now_src = os.path.getsize(r["path"])
now_twin = os.path.getsize(t["path"])
except OSError:
audits.append(f"gone since the census: {r['rel']}"
f" or its survivor {t['rel']}")
break
if now_twin < now_src:
audits.append(f"survivor smaller on disk: {r['rel']}"
f" {now_src} > {t['rel']} {now_twin}")
for r in dominated:
dst = os.path.join(hold, r["rel"])
rows.append((r["path"], dst))
for side in sidecars(r["path"]):
rows.append((side, os.path.join(hold, os.path.relpath(side, library))))
shows_json.append({"show": show, "files": len(dominated),
"bytes": sum(r["size"] for r in dominated),
"episodes": len(dup_keys),
"sole-episode files left": sole,
"folders": sorted(staged_folders)})
n_ep, n_folder = len(dup_keys), len(staged_folders)
report.append(
f"{show}: stage {len(dominated)} files"
f" ({sum(r['size'] for r in dominated) / 2**30:.1f} GiB) — {n_ep}"
f" episode{n_ep != 1 and 's' or ''} held twice or more; {sole}"
f" episode{sole != 1 and 's' or ''} exist in one copy only, so"
f" {n_folder} folder{n_folder != 1 and 's' or ''} keep their files")
rows = sorted(set(rows))
dests = set()
for src, dst in rows:
if dst in dests:
audits.append(f"destination twice: {dst}")
dests.add(dst)
if os.path.exists(dst):
audits.append(f"hold target occupied: {dst}")
if not os.path.isfile(src):
audits.append(f"source missing: {src}")
unkeyed = collections.Counter(os.path.dirname(r["rel"]) for r in files
if not r["keys"])
if ARGS.json:
with open(ARGS.json, "w") as fh:
json.dump({"shows": shows_json,
"conflicts": [list(c) for c in conflicts],
"unresolved": [list(u) for u in unresolved],
"disagreement": {f: [t, d] for f, (t, d) in disagree.items()},
"unkeyed": dict(unkeyed)}, fh, indent=1)
for line in report:
print(line)
for a, b, why in conflicts:
print(f"CONFLICT (never merged): {a} | {b} — {why}")
for a, b, why in unresolved:
print(f"UNRESOLVED (tag it, then re-run): {a} | {b} — {why}")
for f, (tagged, extra) in sorted(disagree.items()):
print(f"DISAGREE (the tag wins): {f} — tag tvdb {tagged},"
f" derived tvdb {extra}")
if unkeyed:
print(f"unkeyed, never staged: {sum(unkeyed.values())} files in"
f" {len(unkeyed)} folders name no SxxEyy and have no --witness row;"
" most: " + ", ".join(f"{d} ({n})" for d, n in unkeyed.most_common(5)))
# The gate runs before the plan is written: a plan file that exists is a plan
# somebody will point the executor at, so a failed audit leaves no artifact.
if audits:
print(f"AUDIT FAILED ({len(audits)})", file=sys.stderr)
for a in audits[:10]:
print(f" {a}", file=sys.stderr)
print(f"refused: no plan written to {out}", file=sys.stderr)
return 1
mkdirs = sorted({os.path.dirname(dst) for _s, dst in rows},
key=lambda p: (p.count(os.sep), p))
with open(out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in mkdirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst in rows:
fh.write(f"move\t{src}\t{dst}\ttvdup\n")
held = sum(os.path.getsize(s) for s, _d in rows if os.path.isfile(s))
print(f"plan: {len(rows)} moves ({held / 2**30:.1f} GiB) into"
f" {len(mkdirs)} hold folders -> {out}")
print("audit: every staged file has a survivor in the same show holding each of"
" its episodes at >= its size on disk, one destination each")
if rows:
root = os.path.dirname(library)
print(f"next: python3 scripts/media-apply.py apply --plan {out} --root"
f" {root} --journal {os.path.join(root, 'tv-dupes.jsonl')}"
" --metadata-view /var/lib/media-meta/view (dry run; add --apply)")
return 0
if __name__ == "__main__":
raise SystemExit(main())