From 7aa32d1b6498764ce5b9f5a51786fe679cdcc96f Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 31 Oct 2026 13:10:32 -0700 Subject: [PATCH] feat(media): the operator's map gives a folder the id its name cannot reach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tvdb titles a record in the show's own language and keeps the English name in aliases the name search never reads, so the rule that accepts only tvdb's own spelling of a query can never resolve them: ブラッククローバー (331753) holds no alias at all, and the bare alias "One Piece" belongs to 392276, the 2023 live action, not to the 1999 record (81797) that 1,072-file pack is. Seventeen pack folders on the array — 3,507 files — are in that class. media-shows.py now reads scripts/media-shows-tvdb-ids.tsv by default: one line per pack folder, folder tvdb id note, every id read from tvdb's own record. The map outranks a provider id, because a box set's provider id is the server's own guess at a folder name, and it merges the release folders that are one show. Everything it touched goes to the exceptions: every folder it named, every row that matched no folder, every disagreement with a provider id, every id tvdb never answered. The file is read fail-closed — an id that is not a number, or one folder named twice, stops the plan, because a typo there renames a pack of 1,268 files to the wrong show. scripts/media-shows-tvdb.py is the sheet the map was built from, promoted out of /tmp so the next run is reproducible: it reads a plan's mkdir rows, asks tvdb about every destination folder carrying no [tvdbid-], and grades the answer EXACT / PUNCT / NEAR / NONE by how much was guessed to reach it. Acceptance is unchanged from the passes it replaces — bare-form equality plus the year-span guard — and that gate is what rejected three confident picks: Fairy Tail's 2024 sequel (410031), the 2026 Hokuto no Ken remake (465216) and Mugyutto! Black Clover (378762) offered for Black Clover. The sheet is for the operator; the map is what the planner reads, and only after he approves the rows. Tests: seven for the map's behaviour — rename to the record it names, two release folders becoming one series, precedence over a provider id and the line stating the disagreement, an id tvdb never answered, a row whose folder is gone, a malformed id refusing the run, a folder named twice refusing the run — plus one that the shipped file parses and never stops a plan. --- scripts/media-shows-tvdb-ids.tsv | 35 ++ scripts/media-shows-tvdb.py | 306 ++++++++++++++++++ scripts/media-shows.py | 115 ++++++- .../Dankserver/Media Relayout.md | 40 ++- tests/media_shows.bats | 129 ++++++++ 5 files changed, 609 insertions(+), 16 deletions(-) create mode 100644 scripts/media-shows-tvdb-ids.tsv create mode 100755 scripts/media-shows-tvdb.py diff --git a/scripts/media-shows-tvdb-ids.tsv b/scripts/media-shows-tvdb-ids.tsv new file mode 100644 index 0000000..f840f8f --- /dev/null +++ b/scripts/media-shows-tvdb-ids.tsv @@ -0,0 +1,35 @@ +# The operator's tvdb id map: `folder tvdb id note`. +# +# These are the pack folders whose show tvdb's NAME SEARCH cannot reach: the +# record exists, but tvdb titles it in the show's own language and the English +# name rides in aliases the search never reads. Without a row here the folder +# keeps the release folder's words and no episode title; with one, both matchers +# read `[tvdbid-NNNN]` and the files rename to tvdb's own name and titles. +# +# Every id was read from tvdb's own record (`/series//extended`) on +# 2026-10-31, and the note states what that record says plus the neighbouring +# record a name search would have picked instead — the row is checkable without +# re-searching, because one wrong id renames a whole pack to the wrong show. The +# plan refuses a line whose id is not a number and lists every folder the map +# touched in the exceptions file. +# +# Folders the name search DOES reach (the EXACT and PUNCT classes of the proposal +# table) stay off this file on purpose: a map row would hide a live lookup behind +# a number nobody re-checks. +[Anime Time] Pokemon Complete Series (Season 01-25+Movies+Specials+Shorts+OST) [DVD+BD][1080p][HEVC 10bit x265][AAC] 76703 tvdb: ポケットモンスター (1997), the umbrella record for every era this pack spans; 1268 files +[Anime Time] One Piece (0001-1071+Movies+Specials) [BD+CR] [Dual Audio] [1080p][HEVC 10bit x265][AAC][Multi Sub] 81797 tvdb: ワンピース (1999), alias ONE PIECE; the bare alias "One Piece" belongs to 392276, the 2023 live action; 1072 files +[Judas] Naruto (Complete Series) [BD 1080p][HEVC x265 10bit][Multi-Audio][Multi-Subs] 78857 tvdb: NARUTO－ナルト－ (2002), alias "Naruto" carries the full-width dashes; 220 files +Star Trek Deep Space 9 Deluxe DVD Boxset + Extras DVDRip TSV 72073 tvdb: Star Trek: Deep Space Nine (1993), alias "Star Trek Deep Space 9"; the box set's own words match nothing; 279 files +Top Gear UK 1-17 74608 tvdb: Top Gear (2002), no "UK" alias — the US version is 199051; 131 files +Black Clover 001 331753 tvdb: ブラッククローバー (2017), which holds NO alias at all; 378762 is Mugyutto! Black Clover (2019), a different show; 108 files +Shin Chan - Complete Eng Dub Series 2160p Ai Upscaled -Mesc 437356 tvdb: Shin Chan (2006) — the English dub record this pack is; 79654 is the 1992 Japanese original; 78 files +Fairy Tail (2009-2014) (Seasons 1-8 + OVAs) (BD 1080p)(HEVC x265 10bit)(Dual-Audio)(Eng-Subs)-Judas[TGx] 114801 tvdb: フェアリーテイル (2009), alias FAIRY TAIL; 410031 is FAIRY TAIL 100 NEN QUEST (2024), the sequel; 59 files +(CBB) Tokyo Ghoul - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-10bit-x265) 281249 tvdb: 東京喰種トーキョーグール (2014) — :re and √A are seasons of THIS record, confirmed by search; 58 files +[Anime Time] Demon Slayer - Kimetsu no Yaiba (S01+S02+S03+S04+Mugen Train Arc+Movie) [BD] [Dual Audio][1080p][HEVC 10bit x265][AAC][Eng Sub] 348545 tvdb: 鬼滅の刃 (2019), alias "Demon Slayer: Kimetsu no Yaiba" — the folder's hyphen splits it differently; 56 files +(CBB) Tengen Toppa Gurren Lagann - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-x265-10bit) 80096 tvdb: 天元突破グレンラガン (2007), alias Tengen Toppa Gurren Lagann; 51 files +Fullmetal Alchemist - Brotherhood 85249 tvdb: 鋼の錬金術師 FULLMETAL ALCHEMIST (2009), alias "Full Metal Alchemist Brotherhood" — spaced, unlike this folder; 64 files +[NewbSubs] Attack on Titan~Shingeki no Kyojin Series (1080p Blu-ray x265 Dual Audio) 267440 tvdb: 進撃の巨人 (2013), aliases carry "Attack on Titan" and "Shingeki no Kyojin", joined here by a tilde; 27 files +Kimetsu no Yaiba 348545 the same record as the Anime Time pack above: two release folders, ONE show; 12 files +Rick and Morty The Anime 420279 tvdb: リック・アンド・モーティ：ジ・アニメ (2024), no English alias on the record; 10 files +[Anime Time] Dorohedoro (Season 1+OVA+Extra) [Dual Audio][BD][1080p][HEVC 10bit x265][AAC][Eng Sub] 370761 tvdb: ドロヘドロ (2020), alias Dorohedoro; 9 files +Ken Il Guerriero HOKUTO NO KEN 79156 tvdb: 北斗の拳 (1984), alias "Hokuto no Ken"; the Italian title matches nothing and 465216 is the 2026 remake; 5 files diff --git a/scripts/media-shows-tvdb.py b/scripts/media-shows-tvdb.py new file mode 100755 index 0000000..399457c --- /dev/null +++ b/scripts/media-shows-tvdb.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python3 +r"""Ask tvdb what it calls the shows a plan renamed WITHOUT a tvdb id. + + media-shows-tvdb.py [--plan TSV] [--root DIR] [--park DIR] [--map TSV] + [--out TSV] [--credentials FILE] + +`media-shows.py` renames a series to tvdb's own words only when an id backs the +name; everything else keeps the metadata provider's words and no episode title, +and the plan's mkdir rows are where those folders show up. This script turns that +list into a decision sheet: one row per destination series folder that carries no +`[tvdbid-NNNN]`, with what tvdb's search answers for it and a verdict that says +HOW MUCH was guessed to get there. + +It changes nothing. The sheet is for the operator; the file that changes +behaviour is `scripts/media-shows-tvdb-ids.tsv`, whose rows he approves. That is +the loop: plan -> this sheet -> approved map rows -> re-plan. + +Verdicts, weakest last: + + EXACT tvdb's own name equals the folder's words as they stand. + PUNCT it equals them only after release junk is stripped (pack numbers, + codecs, group names, `Complete Series`); the difference is spelling, + not identity. + NEAR tvdb answered something whose name is NOT the query. The best hit is + printed so a human can judge it, and it is never applied by anything. + NONE nothing left to ask with, or no answer at all. + +Acceptance is deliberately unforgiving — a hit counts only when tvdb's name +equals the query with every non-alphanumeric dropped from both sides, and the +series' year still has to fall inside any year span the folder names. The +cleaner is the only thing that got smarter over the passes this tool replaces; +the acceptance test never moved, because a wrong id renames a whole pack to the +wrong show. A query that needs the cleaner is reported as PUNCT for exactly that +reason: the machine stripped words, so a human reads the row. + +Read-only against tvdb. The credential file is the planner's own +(`api_key=`, optional `pin=`), read here and never printed. +""" +import argparse +import collections +import importlib.util +import json +import os +import re +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0], + formatter_class=argparse.RawDescriptionHelpFormatter) +ARGS.add_argument("--plan", default="/tmp/tv-plan.tsv", + help="the plan media-shows.py just wrote (its mkdir rows are " + "the folders this sheet asks about)") +ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"), + help="the TV library the plan renames") +ARGS.add_argument("--park", default="", + help="the review pile inside it (its folders are not shows)") +ARGS.add_argument("--map", + default=os.path.join(os.path.dirname(os.path.abspath(__file__)), + "media-shows-tvdb-ids.tsv"), + help="the operator's approved map; its folders are reported as " + "MAPPED and never asked again") +ARGS.add_argument("--credentials", + default=os.path.expanduser("~/.config/secrets/tvdb"), + help="file with an api_key= line (and a pin= for a user-issued " + "key); mode 600, and never printed") +ARGS.add_argument("--out", default="/tmp/tvdb-proposal.tsv", + help="the decision sheet") +ARGS = ARGS.parse_args() + +ROOT = os.path.abspath(ARGS.root) +PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted")) + +_spec = importlib.util.spec_from_file_location( + "eng", os.path.join(os.path.dirname(os.path.abspath(__file__)), + "media-titles.py")) +eng = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(eng) + +# Release metadata, not the show: everything a scene folder name says about the +# ENCODING, the packaging or the languages, which tvdb has never heard of. +JUNK = re.compile( + r"complete|series|dual|multi|audio|sub|subs|multisub|dualaudio|" + r"\d{3,4}p|1080p|720p|2160p|480p|4k|hd|hdtv|cam|hdcam|telesync|x264|x265|hevc|avc|" + r"\d+bit|bit|bluray|blu-?ray|bdrip|web-?rip|web-?dl|webrip|hdtvrip|" + r"dvdrip|dvdr|dvdscr|tsr|tsv|mkv|avi|mp4|aac|ac3|dts|eac3|flac|ddp?5[.1]1|" + r"bd|dvd|rip|boxset|box|season|seasons|episodes?|specials?|shorts|ost|" + r"ova|ovas|oad|repack|proper|internal|mi?xed|budgetbits|deluxe|remux|" + r"remastered|uncut|collector|edition|extras?|volume|volumes|part|" + r"disc|disk|cd|ntsc|pal|region|fixed|patched|proof|english|esp|" + r"espagnol|french|vostfr|subtitul\w*|spanish|german|italian|ita|eng|fre|spa|lat|" + r"upscaled?|ai|dd|movie|movies", + re.I) +GROUP = re.compile(r"KONTRAST|TGx|Judas|Silence|FreetheFilms|NTb|DEADPOOL|Xploit|" + r"SM737|TSA|3OISL3R|PLAYWEB|EPiC|FLUX|HONEY|Zaxton|Cca3rW|" + r"monikeres|frank|hdh|psi|rtfpemaker|cbm|web|asser|astro|" + r"yfn|uindex|www|org|com|net|mesc|ataraxiaprime|seph\d*|" + r"[a-z]+subs?|edge", re.I) +BRACKET = re.compile(r"[\[({]([^\])}]*)[\])}]") +NUM_RANGE = re.compile(r"(?= 3 or (not out and after >= 2): + continue + out.append(t) + while out and out[-1].lower() in ("and", "with", "the", "a", "an", "to", "of"): + out.pop() + return " ".join(out).strip() + + +def years(name): + return [int(y) for y in re.findall(r"\b(19\d\d|20\d\d)\b", name)] + + +def bare(s): + return re.sub(r"[^a-z0-9]", "", s.lower()) + + +CRED = {} +try: + for _line in open(ARGS.credentials): + if "=" in _line: + k, v = _line.strip().split("=", 1) + CRED[k.strip()] = v.strip() +except OSError: + print(f"tvdb: no credentials at {ARGS.credentials} — the sheet would be all " + "NEAR and NONE, which is not a decision sheet", file=sys.stderr) + sys.exit(2) +if not CRED.get("api_key"): + print(f"tvdb: no api_key in {ARGS.credentials}", file=sys.stderr) + sys.exit(2) + +TOKEN = {"jwt": ""} + + +def call(path): + """One tvdb v4 GET, logging in again once when the JWT expires mid-run.""" + for attempt in (1, 2): + if not TOKEN["jwt"]: + body = {"apiKey": CRED["api_key"]} + if CRED.get("pin"): + body["token"] = CRED["pin"] + req = urllib.request.Request("https://api4.thetvdb.com/v4/login", + data=json.dumps(body).encode(), + headers={"Content-Type": + "application/json"}) + TOKEN["jwt"] = json.load( + urllib.request.urlopen(req, timeout=30))["data"]["token"] + try: + req = urllib.request.Request("https://api4.thetvdb.com/v4" + path, + headers={"Authorization": + "Bearer " + TOKEN["jwt"]}) + return json.load(urllib.request.urlopen(req, timeout=30))["data"] + except urllib.error.HTTPError as e: + if e.code == 401 and attempt == 1: + TOKEN["jwt"] = "" + continue + raise + + +MAPPED = {} +if ARGS.map and os.path.exists(ARGS.map): + for _line in open(ARGS.map): + _line = _line.rstrip("\n") + if not _line.strip() or _line.startswith("#"): + continue + cols = _line.split("\t") + if len(cols) > 1 and cols[1].strip().isdigit(): + MAPPED[cols[0].strip()] = cols[1].strip() + +# The folders the plan created inside the library, with the file count each one +# will hold. The review pile is not a show: its files belong to no identity, and +# naming the pile would put one junk series on the sheet for all of them. +shows = collections.Counter() +for line in open(ARGS.plan).read().splitlines()[1:]: + parts = line.split("\t") + if len(parts) < 4 or parts[0] != "mkdir": + continue + dst = os.path.abspath(parts[2]) + if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep): + continue + top = os.path.relpath(dst, ROOT).split(os.sep)[0] + if top.startswith((".", "_")): + continue + if "[tvdbid-" in top: + continue # an id already backs it + shows.setdefault(top, 0) +for line in open(ARGS.plan).read().splitlines()[1:]: + parts = line.split("\t") + if len(parts) < 4 or parts[0] != "move": + continue + dst = os.path.abspath(parts[2]) + if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep): + continue + top = os.path.relpath(dst, ROOT).split(os.sep)[0] + if "[tvdbid-" in top: + continue + shows[top] += 1 + +tally = collections.Counter() +rows = [] +for folder in sorted(shows, key=lambda f: (-shows[f], f)): + files, tag, note = shows[folder], "NONE", "" + if folder in MAPPED: + rows.append([files, "MAPPED", folder, "", "", "", MAPPED[folder], + "the operator's map already states this id"]) + tally["MAPPED"] += 1 + continue + query, untouched = clean(folder), clean(folder) == folder + ys = years(folder) + if not query: + rows.append([files, "NONE", folder, "", "", "", "", + "nothing is left of the folder's name once release junk" + " is gone"]) + else: + try: + hits = call("/search?query=" + urllib.parse.quote(query) + + "&type=series") or [] + except Exception as e: # noqa: BLE001 + hits = [] + note = f"tvdb did not answer: {e}" + want = bare(query) + pick = None + for h in hits: + if bare(h.get("name") or "") != want: + continue + hy = str(h.get("year") or "") + if ys and hy.isdigit() and not (min(ys) <= int(hy) <= max(ys)): + continue + pick = h + break + if pick: + tag = "EXACT" if untouched else "PUNCT" + rows.append([files, tag, folder, query, pick.get("name", ""), + str(pick.get("year", "") or ""), + str(pick.get("tvdb_id", "") or ""), ""]) + elif hits: + best = hits[0] + rows.append([files, "NEAR", folder, query, best.get("name", ""), + str(best.get("year", "") or ""), + str(best.get("tvdb_id", "") or ""), + f"{len(hits)} answer(s); tvdb's own name is not the " + "query, so nothing would take it"]) + else: + rows.append([files, "NONE", folder, query, "", "", "", + note or "tvdb has no series by that name"]) + tally[rows[-1][1]] += 1 + time.sleep(0.12) + +# Two folders mapping to one record is not two shows; it is one show the release +# folders split. Say it on every row involved, or the operator approves the same +# id twice and reads the second as a mistake. +by_id = collections.Counter(r[6] for r in rows if r[6] and r[1] != "MAPPED") +for r in rows: + if r[6] and r[1] != "MAPPED" and by_id[r[6]] > 1: + r[7] = (f"MERGE: {by_id[r[6]]} folders map to tvdb {r[6]} — one show, " + "the releases split it; " + r[7]).strip("; ") + +with open(ARGS.out, "w") as fh: + fh.write("files\tverdict\tfolder\tquery\ttvdb_name\ttvdb_year\ttvdb_id\tnote\n") + for r in rows: + fh.write("\t".join(str(c) for c in r) + "\n") +print(f"sheet: {len(rows)} folders" + + "".join(f" {t}={n}" for t, n in sorted(tally.items())) + + f" -> {ARGS.out}") diff --git a/scripts/media-shows.py b/scripts/media-shows.py index 4dff323..ebacd3d 100755 --- a/scripts/media-shows.py +++ b/scripts/media-shows.py @@ -3,7 +3,7 @@ r"""Plan a Jellyfin-format relayout of a TV library. DRY RUN ONLY: nothing moves media-shows.py [--root DIR] [--enum TSV] [--titles TSV] [--probed TSV] [--tmdb-cache JSON] [--tmdb] [--movies DIR] [--park DIR] - [--out TSV] [--exceptions TXT] + [--tvdb-ids TSV] [--out TSV] [--exceptions TXT] Inputs are the TSVs scripts/media-census.py exports: `enumerate` lists what exists, `jellyfin` says which episode each file IS (its series' TMDb and Tvdb @@ -24,6 +24,14 @@ The series name and the episode title are the ones the IDENTITY authority — tv — states for the id the metadata server matched, not the words a release folder or a server's folder-derived Series name happens to carry. +A folder tvdb's search cannot state is not a folder tvdb does not know: the +record exists, it is titled in the show's own language, and the English name +rides in aliases the search never reads. --tvdb-ids is the human door for those: +one line per pack folder, `folder tvdb id note`, every id checked by +hand against the record it names. It is an input file like the census, so a plan +stays reproducible, and every folder it touched is listed in the exceptions — +one wrong row is a whole pack renamed to the wrong show. + PLAN is the TSV scripts/media-apply.py replays: `action src dst kind`, every mkdir row before any move, and `keep` for a file already where the contract puts it. EXCEPTIONS is the review list: everything parked, and why. @@ -106,6 +114,16 @@ ARGS.add_argument("--tvdb-credentials", default=os.path.expanduser("~/.config/secrets/tvdb"), help="file with an api_key= line (and a pin= line for a " "user-issued key); mode 600, and never the cache") +ARGS.add_argument("--tvdb-ids", + default=os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "media-shows-tvdb-ids.tsv"), + help="the operator's map: folder tvdb id note, for the\n" + "packs whose name tvdb's search cannot state (anime is titled\n" + "by its own record, the English name rides in aliases). Used\n" + "only when the file exists; a line without a numeric id stops\n" + "the plan, because a typo renames a whole pack to the wrong\n" + "show.") ARGS.add_argument("--movies", default="/mnt/entropy/Movies", help="movie library a mis-filed film moves into") ARGS.add_argument("--park", default="", @@ -611,6 +629,63 @@ def extras_kind(path, extra_type): return "extras" +# --------------------------------------------------------- the operator's map +# The map is the human answer to a name the authority cannot be searched by. It +# is read as an input file — like the census and the cache — so a plan stays +# reproducible from the files beside it, and it outranks a provider id: a box +# set's provider id is the server's own guess at a folder name, while a map row +# is an id someone checked against the record it names. +MAP = {} # top-level pack folder -> {"id", "note"} +MAP_SEEN, MAP_CLASH = set(), [] # what the map actually touched +if ARGS.tvdb_ids and os.path.exists(ARGS.tvdb_ids): + for _n, _line in enumerate(open(ARGS.tvdb_ids), 1): + _line = _line.rstrip("\n") + if not _line.strip() or _line.startswith("#"): + continue + cols = _line.split("\t") + folder = cols[0].strip() + sid = cols[1].strip() if len(cols) > 1 else "" + if not folder or not sid.isdigit(): + print(f"{ARGS.tvdb_ids}:{_n}: not a numeric tvdb id: {_line}", + file=sys.stderr) + sys.exit(2) + if folder in MAP: + print(f"{ARGS.tvdb_ids}:{_n}: folder named twice: {folder}", + file=sys.stderr) + sys.exit(2) + MAP[folder] = {"id": sid, + "note": cols[2].strip() if len(cols) > 2 else ""} + + +def top_folder(path): + """The library folder a file's pack sits in. An id belongs to a show, and a + show is a folder here, so that is what the map keys on.""" + return os.path.relpath(path, ROOT).split(os.sep)[0] + + +def map_id_for(path, provider_id): + """The id the operator states for this file's folder. The map wins over the + provider's — it is the only door for a name tvdb cannot state — and a + disagreement between the two is written where the operator reads it, never + resolved in silence.""" + if not MAP: + return provider_id + top = top_folder(path) + entry = MAP.get(top) + if not entry: + return provider_id + MAP_SEEN.add(top) + if provider_id and provider_id != entry["id"]: + MAP_CLASH.append((top, provider_id, entry["id"])) + return entry["id"] + + +def map_note(): + if not MAP: + return "" + return f" (+{len(MAP_SEEN)} folders from the operator's map)" + + # ----------------------------------------------------------- classify every file series = {} # norm(name) -> {name, year, tmdb} claimants = [] # (row, name, season, [episodes]) @@ -644,7 +719,7 @@ def as_extra(r, owner, extra_type): if not owner: unknown.append((r, "bonus material with no series to own it")) return - note(owner) + note(owner, tvdb_id=map_id_for(r["path"], "")) extras.append((r, owner, extras_kind(r["path"], extra_type))) for r in video: @@ -686,12 +761,12 @@ for r in video: eps = [ep] claimants.append((r, owner, season, eps)) note(owner, t["year"] if t else "", t["tmdb"] if t else "", - t["tvdb"] if t else "") + map_id_for(r["path"], t["tvdb"] if t else "")) continue owner = series_of_dir(os.path.dirname(p)) if fs is not None and owner and leading_index(r["base"]): claimants.append((r, owner, fs, [leading_index(r["base"])])) - note(owner) + note(owner, tvdb_id=map_id_for(r["path"], "")) continue if eng.is_extra(p, title_of(r["base"])[0], dur.get(p, 0.0), ROOT): as_extra(r, owner, "") @@ -1073,7 +1148,7 @@ def write_plan(): sized = sum(held[s]["bytes"] for s, _d, _k in moves if s in held) print(f"plan: {len(moves) - len(parked)} moves, {len(parked)} parked" f" ({sized / 2**40:.2f} TiB touched), {len(keeps)} already placed," - f" {len(dirs)} folders{provider_note()} -> {ARGS.out}") + f" {len(dirs)} folders{map_note()}{provider_note()} -> {ARGS.out}") def write_exceptions(): @@ -1123,6 +1198,36 @@ def write_exceptions(): fh.write(" " + "; ".join( f"{n} S{'/'.join(f'{x:02d}' for x in ss)}" for n, ss in clash[:300]) + "\n") + map_ids = {e["id"] for e in MAP.values()} + unseen = sorted(f for f in MAP if f not in MAP_SEEN) + unanswered = sorted({f"{identity(s)[0]} ({s['tvdb']})" + for s in canon.values() + if s.get("tvdb") in map_ids and not s.get("canon")}) + if MAP: + fh.write(f"\n{len(MAP_SEEN):>6} folders took their tvdb id from the\n") + fh.write(" operator's map — the human answer to a name tvdb's search\n" + " cannot state. A wrong row renames a whole pack to the wrong\n" + " show, so every folder it touched is listed here; fix the map\n" + " and re-plan, because a rename is idempotent.\n") + fh.write(" " + "; ".join( + f"{f} -> {MAP[f]['id']}" for f in sorted(MAP_SEEN)) + "\n") + if unseen: + fh.write(f"\n{len(unseen):>6} map rows name no folder on the array: the\n") + fh.write(" pack moved, or the row carries a name the array never held.\n") + fh.write(" " + "; ".join(unseen[:300]) + "\n") + if MAP_CLASH: + fh.write(f"\n{len(MAP_CLASH):>6} folders carry a provider tvdb id the map\n") + fh.write(" contradicts. The map won: a box set's provider id is the\n") + fh.write(" server's guess at a folder name. Read each one:\n") + fh.write(" " + "; ".join( + f"{f} provider {p} map {m}" + for f, p, m in sorted(MAP_CLASH)[:300]) + "\n") + if unanswered: + fh.write(f"\n{len(unanswered):>6} map ids tvdb never answered, so those\n") + fh.write(" folders carry the id and the provider's own name; both\n") + fh.write(" matchers resolve by the id in the folder tag, so the next\n" + " re-plan takes tvdb's title. Nothing was invented.\n") + fh.write(" " + "; ".join(unanswered[:300]) + "\n") print(f"exceptions -> {ARGS.exceptions}") diff --git a/serverAppConfigs-docs/Dankserver/Media Relayout.md b/serverAppConfigs-docs/Dankserver/Media Relayout.md index a1ad5f1..568a3a0 100644 --- a/serverAppConfigs-docs/Dankserver/Media Relayout.md +++ b/serverAppConfigs-docs/Dankserver/Media Relayout.md @@ -301,17 +301,35 @@ visible series folder that real files get renamed into. `--park` outside the roo makes the next plan re-file the pile instead of skipping it. Inside a hidden top-level folder a file's own words may MATCH a registered show, never invent one. -Release-group folder names are the one thing no provider can read, so the -pending TVShows naming plan files 179 top-level directories (9,931 files) under -the junk name they arrived with. A TVDB id map can name them, under the ladder's -own acceptance rule: a hit counts only when TVDB's spelling equals the cleaned -query, or differs from it by punctuation alone — the same letters are not a -guess. Everything else is a choice for Frank, and the map is never applied -unreviewed: `/mnt/entropy/media-relayout-shows-tvdb-proposal.tsv` carries one -row per folder with `verdict` = `EXACT` / `PUNCT` / `NEAR` / `NONE` and a `note` -naming the two traps — a folder marked `(US)` that matches the UK original, and -several folders mapping to one id, which is a MERGE whose duplicate episodes the -executor will refuse until a keeper is chosen. +**The operator's id map is the door for a name tvdb cannot be searched by.** +`scripts/media-shows.py` reads `scripts/media-shows-tvdb-ids.tsv` by default: one +line per pack folder, `folder tvdb id note`. The file exists because +tvdb titles a record in the show's own language and the English name rides in +*aliases* the name search never reads: `ブラッククローバー` (331753) answers to no +spelling of "Black Clover", and the bare alias `One Piece` belongs to 392276, the +2023 live action, not to the 1999 record (81797) the pack here is. The map +outranks a provider id — a box set's provider id is the server's own guess at a +folder name — and it merges the release folders that are one show. Everything it +touched is written to the exceptions: every folder it named, every row that +matched no folder, every disagreement with a provider id, every id tvdb never +answered. A line whose id is not a number stops the plan, because a typo renames +a whole pack to the wrong show. Only folders the search CANNOT reach belong in +it; a row for a folder the search would resolve hides a live lookup behind a +number nobody re-checks. + +`scripts/media-shows-tvdb.py` makes the sheet the map is built from. It reads a +plan's `mkdir` rows, asks tvdb about every destination folder carrying no +`[tvdbid-`, and writes one row per folder with the verdict that says how much was +guessed to get there: `EXACT` (tvdb's own spelling), `PUNCT` (equal only after +release junk is stripped from the folder's words), `NEAR` (tvdb answered +something else — the best hit is printed for a human and nothing applies it), +`NONE`. Acceptance never moved across the passes that built this: a hit counts +only when tvdb's name equals the query with every non-alphanumeric dropped from +both sides, and the record's year still has to fall inside any year span the +folder names. That gate is what caught three confident picks — Fairy Tail's 2024 +sequel (410031), the 2026 Hokuto no Ken remake (465216) and `Mugyutto! Black +Clover` (378762) offered for Black Clover. It is why the map is approved by hand +and the sheet is never fed to the planner. ## The music and audiobook arms diff --git a/tests/media_shows.bats b/tests/media_shows.bats index e78bf1d..7b1bbd2 100755 --- a/tests/media_shows.bats +++ b/tests/media_shows.bats @@ -566,3 +566,132 @@ PY run grep -i "numbering tvdb contradicts" "$EXC" [ "$status" -ne 0 ] } + +# ------------------------------------------------------------- the operator map +# Most of this library carries no tvdb id anywhere: the authority titles anime by +# its own record — in the show's language — and the English name rides in aliases +# the planner's name search never reads, so a search that demands tvdb's own +# spelling cannot reach it. The operator's map is the other door: one line per +# folder, `folder tvdb id note`, each id checked by hand against the +# record it names. It outranks a provider id, it merges the release folders that +# are one show, and every folder it touched is named in the exceptions: one wrong +# row there is 1,268 renames to the wrong show, so nothing about it is silent. + +plan_map() { # the plan, with the operator's map in hand + run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \ + --probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \ + --tvdb-ids "$1" --out "$OUT" --exceptions "$EXC" +} + +@test "a folder the operator mapped to a tvdb id renames to that record" { + printf '{"tvdb:series:331753": %s}\n' \ + '{"name": "Black Clover", "year": "2017"}' > "$CACHE" + { + printf '# the note column carries why the name search cannot reach it\n' + printf 'Black Clover 001\t331753\tthe record holds no alias\n' + } > "$BATS_TEST_TMPDIR/ids.tsv" + file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000 + title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \ + "Black Clover 001" "" 1 1 + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + want="$LIB/Black Clover (2017) [tvdbid-331753]/Season 01" + want+="/Black Clover (2017) - S01E01.mkv" + [ "$(dst "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv")" = "$want" ] +} + +@test "release folders one id maps are ONE series folder" { + printf '{"tvdb:series:81797": %s}\n' \ + '{"name": "One Piece", "year": "1999"}' > "$CACHE" + { + printf '[Anime Time] One Piece (0001-1071)\t81797\tbare "One Piece" is the 2023 live action\n' + printf 'One Piece Ship Collection\t81797\tthe same record, split by release\n' + } > "$BATS_TEST_TMPDIR/ids.tsv" + file "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv" 5000 + file "$LIB/One Piece Ship Collection/One.Piece.0002.mkv" 6000 + title_row "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv" \ + "One Piece" 1999 1 1 + title_row "$LIB/One Piece Ship Collection/One.Piece.0002.mkv" \ + "One Piece Ship Collection" "" 1 2 + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + want="$LIB/One Piece (1999) [tvdbid-81797]/Season 01" + want+="/One Piece (1999) - S01E01.mkv" + [ "$(dst "$LIB/[Anime Time] One Piece (0001-1071)/One.Piece.0001.mkv")" \ + = "$want" ] + want="$LIB/One Piece (1999) [tvdbid-81797]/Season 01" + want+="/One Piece (1999) - S01E02.mkv" + [ "$(dst "$LIB/One Piece Ship Collection/One.Piece.0002.mkv")" = "$want" ] +} + +@test "the map outranks a provider id and says which two disagreed" { + printf '{"tvdb:series:4242": %s, "tvdb:series:78850": %s}\n' \ + '{"name": "Real Ship", "year": "1999"}' \ + '{"name": "Box Junk", "year": "2016"}' > "$CACHE" + printf '%s\t4242\thand-verified against the record\n' \ + "Show Box (2016) Season 1 S01 (1080p WEB-DL)" > "$BATS_TEST_TMPDIR/ids.tsv" + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \ + | grep -q "^$LIB/Real Ship (1999) \[tvdbid-4242\]/Season 01/" + run grep -E '78850.*4242|4242.*78850' "$EXC" + [ "$status" -eq 0 ] +} + +@test "a map id the authority never answered keeps the provider's name and is named" { + printf 'Black Clover 001\t331753\ttvdb never answered\n' \ + > "$BATS_TEST_TMPDIR/ids.tsv" + file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000 + title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \ + "Black Clover 001" "" 1 1 + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + dst "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \ + | grep -q "^$LIB/Black Clover 001 \[tvdbid-331753\]/Season 01/" + run grep -q "331753" "$EXC" + [ "$status" -eq 0 ] +} + +@test "a map row for a folder the array no longer holds is reported" { + printf 'Gone Pack\t4242\tmoved away\n' > "$BATS_TEST_TMPDIR/ids.tsv" + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + run grep "Gone Pack" "$EXC" + [ "$status" -eq 0 ] +} + +@test "a map line whose id is not a number stops the plan" { + printf 'Black Clover 001\tnotanid\ttypo\n' > "$BATS_TEST_TMPDIR/ids.tsv" + file "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" 5000 + title_row "$LIB/Black Clover 001/Season 01/Black.Clover.001.mkv" \ + "Black Clover 001" "" 1 1 + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + [ "$status" -ne 0 ] + [[ "$output" == *"notanid"* ]] +} + +# The map the fleet ships is scripts/media-shows-tvdb-ids.tsv, and the planner +# reads it by default: an unusable line there stops every plan the operator runs +# from the checkout, so the seam validates the real file, not a fixture copy. +@test "the shipped operator map parses and never stops a plan" { + MAPS="$REPO_ROOT/scripts/media-shows-tvdb-ids.tsv" + run awk -F '\t' '!/^#/ && NF { + if (NF < 2 || $2 !~ /^[0-9]+$/) { print "unusable row: " $0; bad = 1 } + } END { exit bad }' "$MAPS" + [ "$status" -eq 0 ] + dup=$(awk -F '\t' '!/^#/ && NF {print $1}' "$MAPS" | sort | uniq -d) + [ -z "$dup" ] + plan_map "$MAPS" + [ "$status" -eq 0 ] + grep -qi "operator's map" "$EXC" +} + +@test "a map that names one folder twice stops the plan" { + printf '{"tvdb:series:78850": %s}\n' \ + '{"name": "Naruto", "year": "2002"}' > "$CACHE" + { + printf 'Naruto Complete Series\t78850\tthe dub pack\n' + printf 'Naruto Complete Series\t331753\ta paste of another row\n' + } > "$BATS_TEST_TMPDIR/ids.tsv" + file "$LIB/Naruto Complete Series/Season 01/naruto 01.mkv" 5000 + title_row "$LIB/Naruto Complete Series/Season 01/naruto 01.mkv" \ + "Naruto Complete Series" "" 1 1 + plan_map "$BATS_TEST_TMPDIR/ids.tsv" + [ "$status" -eq 2 ] + [[ "$output" == *"named twice"* ]] +}