Frank
7aa32d1b64
Tvdb titles a record in the show's own language and keeps the English name in aliases the name search never reads, so the rule that accepts only tvdb's own spelling of a query can never resolve them: ブラッククローバー (331753) holds no alias at all, and the bare alias "One Piece" belongs to 392276, the 2023 live action, not to the 1999 record (81797) that 1,072-file pack is. Seventeen pack folders on the array — 3,507 files — are in that class. media-shows.py now reads scripts/media-shows-tvdb-ids.tsv by default: one line per pack folder, folder <TAB> tvdb id <TAB> note, every id read from tvdb's own record. The map outranks a provider id, because a box set's provider id is the server's own guess at a folder name, and it merges the release folders that are one show. Everything it touched goes to the exceptions: every folder it named, every row that matched no folder, every disagreement with a provider id, every id tvdb never answered. The file is read fail-closed — an id that is not a number, or one folder named twice, stops the plan, because a typo there renames a pack of 1,268 files to the wrong show. scripts/media-shows-tvdb.py is the sheet the map was built from, promoted out of /tmp so the next run is reproducible: it reads a plan's mkdir rows, asks tvdb about every destination folder carrying no [tvdbid-], and grades the answer EXACT / PUNCT / NEAR / NONE by how much was guessed to reach it. Acceptance is unchanged from the passes it replaces — bare-form equality plus the year-span guard — and that gate is what rejected three confident picks: Fairy Tail's 2024 sequel (410031), the 2026 Hokuto no Ken remake (465216) and Mugyutto! Black Clover (378762) offered for Black Clover. The sheet is for the operator; the map is what the planner reads, and only after he approves the rows. Tests: seven for the map's behaviour — rename to the record it names, two release folders becoming one series, precedence over a provider id and the line stating the disagreement, an id tvdb never answered, a row whose folder is gone, a malformed id refusing the run, a folder named twice refusing the run — plus one that the shipped file parses and never stops a plan.
307 lines
13 KiB
Python
Executable File
307 lines
13 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
r"""Ask tvdb what it calls the shows a plan renamed WITHOUT a tvdb id.
|
||
|
||
media-shows-tvdb.py [--plan TSV] [--root DIR] [--park DIR] [--map TSV]
|
||
[--out TSV] [--credentials FILE]
|
||
|
||
`media-shows.py` renames a series to tvdb's own words only when an id backs the
|
||
name; everything else keeps the metadata provider's words and no episode title,
|
||
and the plan's mkdir rows are where those folders show up. This script turns that
|
||
list into a decision sheet: one row per destination series folder that carries no
|
||
`[tvdbid-NNNN]`, with what tvdb's search answers for it and a verdict that says
|
||
HOW MUCH was guessed to get there.
|
||
|
||
It changes nothing. The sheet is for the operator; the file that changes
|
||
behaviour is `scripts/media-shows-tvdb-ids.tsv`, whose rows he approves. That is
|
||
the loop: plan -> this sheet -> approved map rows -> re-plan.
|
||
|
||
Verdicts, weakest last:
|
||
|
||
EXACT tvdb's own name equals the folder's words as they stand.
|
||
PUNCT it equals them only after release junk is stripped (pack numbers,
|
||
codecs, group names, `Complete Series`); the difference is spelling,
|
||
not identity.
|
||
NEAR tvdb answered something whose name is NOT the query. The best hit is
|
||
printed so a human can judge it, and it is never applied by anything.
|
||
NONE nothing left to ask with, or no answer at all.
|
||
|
||
Acceptance is deliberately unforgiving — a hit counts only when tvdb's name
|
||
equals the query with every non-alphanumeric dropped from both sides, and the
|
||
series' year still has to fall inside any year span the folder names. The
|
||
cleaner is the only thing that got smarter over the passes this tool replaces;
|
||
the acceptance test never moved, because a wrong id renames a whole pack to the
|
||
wrong show. A query that needs the cleaner is reported as PUNCT for exactly that
|
||
reason: the machine stripped words, so a human reads the row.
|
||
|
||
Read-only against tvdb. The credential file is the planner's own
|
||
(`api_key=`, optional `pin=`), read here and never printed.
|
||
"""
|
||
import argparse
|
||
import collections
|
||
import importlib.util
|
||
import json
|
||
import os
|
||
import re
|
||
import sys
|
||
import time
|
||
import urllib.error
|
||
import urllib.parse
|
||
import urllib.request
|
||
|
||
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
|
||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||
ARGS.add_argument("--plan", default="/tmp/tv-plan.tsv",
|
||
help="the plan media-shows.py just wrote (its mkdir rows are "
|
||
"the folders this sheet asks about)")
|
||
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
|
||
help="the TV library the plan renames")
|
||
ARGS.add_argument("--park", default="",
|
||
help="the review pile inside it (its folders are not shows)")
|
||
ARGS.add_argument("--map",
|
||
default=os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||
"media-shows-tvdb-ids.tsv"),
|
||
help="the operator's approved map; its folders are reported as "
|
||
"MAPPED and never asked again")
|
||
ARGS.add_argument("--credentials",
|
||
default=os.path.expanduser("~/.config/secrets/tvdb"),
|
||
help="file with an api_key= line (and a pin= for a user-issued "
|
||
"key); mode 600, and never printed")
|
||
ARGS.add_argument("--out", default="/tmp/tvdb-proposal.tsv",
|
||
help="the decision sheet")
|
||
ARGS = ARGS.parse_args()
|
||
|
||
ROOT = os.path.abspath(ARGS.root)
|
||
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
|
||
|
||
_spec = importlib.util.spec_from_file_location(
|
||
"eng", os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||
"media-titles.py"))
|
||
eng = importlib.util.module_from_spec(_spec)
|
||
_spec.loader.exec_module(eng)
|
||
|
||
# Release metadata, not the show: everything a scene folder name says about the
|
||
# ENCODING, the packaging or the languages, which tvdb has never heard of.
|
||
JUNK = re.compile(
|
||
r"complete|series|dual|multi|audio|sub|subs|multisub|dualaudio|"
|
||
r"\d{3,4}p|1080p|720p|2160p|480p|4k|hd|hdtv|cam|hdcam|telesync|x264|x265|hevc|avc|"
|
||
r"\d+bit|bit|bluray|blu-?ray|bdrip|web-?rip|web-?dl|webrip|hdtvrip|"
|
||
r"dvdrip|dvdr|dvdscr|tsr|tsv|mkv|avi|mp4|aac|ac3|dts|eac3|flac|ddp?5[.1]1|"
|
||
r"bd|dvd|rip|boxset|box|season|seasons|episodes?|specials?|shorts|ost|"
|
||
r"ova|ovas|oad|repack|proper|internal|mi?xed|budgetbits|deluxe|remux|"
|
||
r"remastered|uncut|collector|edition|extras?|volume|volumes|part|"
|
||
r"disc|disk|cd|ntsc|pal|region|fixed|patched|proof|english|esp|"
|
||
r"espagnol|french|vostfr|subtitul\w*|spanish|german|italian|ita|eng|fre|spa|lat|"
|
||
r"upscaled?|ai|dd|movie|movies",
|
||
re.I)
|
||
GROUP = re.compile(r"KONTRAST|TGx|Judas|Silence|FreetheFilms|NTb|DEADPOOL|Xploit|"
|
||
r"SM737|TSA|3OISL3R|PLAYWEB|EPiC|FLUX|HONEY|Zaxton|Cca3rW|"
|
||
r"monikeres|frank|hdh|psi|rtfpemaker|cbm|web|asser|astro|"
|
||
r"yfn|uindex|www|org|com|net|mesc|ataraxiaprime|seph\d*|"
|
||
r"[a-z]+subs?|edge", re.I)
|
||
BRACKET = re.compile(r"[\[({]([^\])}]*)[\])}]")
|
||
NUM_RANGE = re.compile(r"(?<!\w)\d{1,4}\s*(?:-|–|to|through)\s*\d{1,4}(?!\w)", re.I)
|
||
INSIDE = re.compile(r"all|complete|full|dual|multi|audio|sub|subs|ova|ovas|"
|
||
r"movies?|episodes?|extras?|bonus|pack|remux|region\s*\d*|"
|
||
r"bd|dvd|cd|ntsc|pal|hr-?dvd", re.I)
|
||
|
||
|
||
def drop_bracket(m):
|
||
"""A bracket says nothing when its words are junk; keep it when they aren't."""
|
||
toks = [t for t in re.split(r"[\s_.,+~&]+", m.group(1)) if t]
|
||
if not toks or all(INSIDE.fullmatch(t) or JUNK.fullmatch(t)
|
||
or GROUP.fullmatch(t)
|
||
or re.fullmatch(r"\d+[a-z]?", t, re.I)
|
||
or re.fullmatch(r"(19|20)\d\d", t) for t in toks):
|
||
return " "
|
||
return " " + m.group(1) + " "
|
||
|
||
|
||
def clean(name):
|
||
"""The folder's words with everything about the RELEASE taken out, so what is
|
||
left is what a person would type into tvdb. `Star Trek Deep Space 9 Deluxe
|
||
DVD Boxset + Extras DVDRip TSV` becomes `Star Trek Deep Space 9`; `SNL 16-20`
|
||
becomes `SNL`."""
|
||
name = re.sub(r"^\s*www\b.*?\b(org|com|net)\b\s*[-–]\s*", " ", name, flags=re.I)
|
||
name = re.sub(r"\bS\d{1,2}([ .Ee]\d+)?[-–Ee\d.]*", " ", name)
|
||
name = re.sub(r"\b\d{1,2}[xX]\d{1,3}\b", " ", name)
|
||
name = NUM_RANGE.sub(" ", name)
|
||
name = re.sub(r"(19|20)\d\d(?:\s*[-/–]\s*(?:19|20)\d\d)?", " ", name)
|
||
name = re.sub(r"\bby\s+[A-Za-z0-9_.-]+$", " ", name)
|
||
name = BRACKET.sub(drop_bracket, name)
|
||
name = re.sub(r"[\[()\]:,;!?*]", " ", name)
|
||
toks = re.split(r"[\s_.\-+~&]+", name)
|
||
out = []
|
||
for i, t in enumerate(toks):
|
||
if not t or JUNK.fullmatch(t) or GROUP.fullmatch(t):
|
||
continue
|
||
if re.fullmatch(r"\d+", t):
|
||
after = sum(1 for x in toks[i + 1:] if x
|
||
and not JUNK.fullmatch(x) and not GROUP.fullmatch(x)
|
||
and not re.fullmatch(r"\d+", x))
|
||
# A pack index (`001`) and a lead-in number over a real title
|
||
# (`01 Charles Barkley`) are noise; `30 Rock`, `24`, `Deep Space 9`
|
||
# are the title itself.
|
||
if len(t) >= 3 or (not out and after >= 2):
|
||
continue
|
||
out.append(t)
|
||
while out and out[-1].lower() in ("and", "with", "the", "a", "an", "to", "of"):
|
||
out.pop()
|
||
return " ".join(out).strip()
|
||
|
||
|
||
def years(name):
|
||
return [int(y) for y in re.findall(r"\b(19\d\d|20\d\d)\b", name)]
|
||
|
||
|
||
def bare(s):
|
||
return re.sub(r"[^a-z0-9]", "", s.lower())
|
||
|
||
|
||
CRED = {}
|
||
try:
|
||
for _line in open(ARGS.credentials):
|
||
if "=" in _line:
|
||
k, v = _line.strip().split("=", 1)
|
||
CRED[k.strip()] = v.strip()
|
||
except OSError:
|
||
print(f"tvdb: no credentials at {ARGS.credentials} — the sheet would be all "
|
||
"NEAR and NONE, which is not a decision sheet", file=sys.stderr)
|
||
sys.exit(2)
|
||
if not CRED.get("api_key"):
|
||
print(f"tvdb: no api_key in {ARGS.credentials}", file=sys.stderr)
|
||
sys.exit(2)
|
||
|
||
TOKEN = {"jwt": ""}
|
||
|
||
|
||
def call(path):
|
||
"""One tvdb v4 GET, logging in again once when the JWT expires mid-run."""
|
||
for attempt in (1, 2):
|
||
if not TOKEN["jwt"]:
|
||
body = {"apiKey": CRED["api_key"]}
|
||
if CRED.get("pin"):
|
||
body["token"] = CRED["pin"]
|
||
req = urllib.request.Request("https://api4.thetvdb.com/v4/login",
|
||
data=json.dumps(body).encode(),
|
||
headers={"Content-Type":
|
||
"application/json"})
|
||
TOKEN["jwt"] = json.load(
|
||
urllib.request.urlopen(req, timeout=30))["data"]["token"]
|
||
try:
|
||
req = urllib.request.Request("https://api4.thetvdb.com/v4" + path,
|
||
headers={"Authorization":
|
||
"Bearer " + TOKEN["jwt"]})
|
||
return json.load(urllib.request.urlopen(req, timeout=30))["data"]
|
||
except urllib.error.HTTPError as e:
|
||
if e.code == 401 and attempt == 1:
|
||
TOKEN["jwt"] = ""
|
||
continue
|
||
raise
|
||
|
||
|
||
MAPPED = {}
|
||
if ARGS.map and os.path.exists(ARGS.map):
|
||
for _line in open(ARGS.map):
|
||
_line = _line.rstrip("\n")
|
||
if not _line.strip() or _line.startswith("#"):
|
||
continue
|
||
cols = _line.split("\t")
|
||
if len(cols) > 1 and cols[1].strip().isdigit():
|
||
MAPPED[cols[0].strip()] = cols[1].strip()
|
||
|
||
# The folders the plan created inside the library, with the file count each one
|
||
# will hold. The review pile is not a show: its files belong to no identity, and
|
||
# naming the pile would put one junk series on the sheet for all of them.
|
||
shows = collections.Counter()
|
||
for line in open(ARGS.plan).read().splitlines()[1:]:
|
||
parts = line.split("\t")
|
||
if len(parts) < 4 or parts[0] != "mkdir":
|
||
continue
|
||
dst = os.path.abspath(parts[2])
|
||
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
|
||
continue
|
||
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
|
||
if top.startswith((".", "_")):
|
||
continue
|
||
if "[tvdbid-" in top:
|
||
continue # an id already backs it
|
||
shows.setdefault(top, 0)
|
||
for line in open(ARGS.plan).read().splitlines()[1:]:
|
||
parts = line.split("\t")
|
||
if len(parts) < 4 or parts[0] != "move":
|
||
continue
|
||
dst = os.path.abspath(parts[2])
|
||
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
|
||
continue
|
||
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
|
||
if "[tvdbid-" in top:
|
||
continue
|
||
shows[top] += 1
|
||
|
||
tally = collections.Counter()
|
||
rows = []
|
||
for folder in sorted(shows, key=lambda f: (-shows[f], f)):
|
||
files, tag, note = shows[folder], "NONE", ""
|
||
if folder in MAPPED:
|
||
rows.append([files, "MAPPED", folder, "", "", "", MAPPED[folder],
|
||
"the operator's map already states this id"])
|
||
tally["MAPPED"] += 1
|
||
continue
|
||
query, untouched = clean(folder), clean(folder) == folder
|
||
ys = years(folder)
|
||
if not query:
|
||
rows.append([files, "NONE", folder, "", "", "", "",
|
||
"nothing is left of the folder's name once release junk"
|
||
" is gone"])
|
||
else:
|
||
try:
|
||
hits = call("/search?query=" + urllib.parse.quote(query)
|
||
+ "&type=series") or []
|
||
except Exception as e: # noqa: BLE001
|
||
hits = []
|
||
note = f"tvdb did not answer: {e}"
|
||
want = bare(query)
|
||
pick = None
|
||
for h in hits:
|
||
if bare(h.get("name") or "") != want:
|
||
continue
|
||
hy = str(h.get("year") or "")
|
||
if ys and hy.isdigit() and not (min(ys) <= int(hy) <= max(ys)):
|
||
continue
|
||
pick = h
|
||
break
|
||
if pick:
|
||
tag = "EXACT" if untouched else "PUNCT"
|
||
rows.append([files, tag, folder, query, pick.get("name", ""),
|
||
str(pick.get("year", "") or ""),
|
||
str(pick.get("tvdb_id", "") or ""), ""])
|
||
elif hits:
|
||
best = hits[0]
|
||
rows.append([files, "NEAR", folder, query, best.get("name", ""),
|
||
str(best.get("year", "") or ""),
|
||
str(best.get("tvdb_id", "") or ""),
|
||
f"{len(hits)} answer(s); tvdb's own name is not the "
|
||
"query, so nothing would take it"])
|
||
else:
|
||
rows.append([files, "NONE", folder, query, "", "", "",
|
||
note or "tvdb has no series by that name"])
|
||
tally[rows[-1][1]] += 1
|
||
time.sleep(0.12)
|
||
|
||
# Two folders mapping to one record is not two shows; it is one show the release
|
||
# folders split. Say it on every row involved, or the operator approves the same
|
||
# id twice and reads the second as a mistake.
|
||
by_id = collections.Counter(r[6] for r in rows if r[6] and r[1] != "MAPPED")
|
||
for r in rows:
|
||
if r[6] and r[1] != "MAPPED" and by_id[r[6]] > 1:
|
||
r[7] = (f"MERGE: {by_id[r[6]]} folders map to tvdb {r[6]} — one show, "
|
||
"the releases split it; " + r[7]).strip("; ")
|
||
|
||
with open(ARGS.out, "w") as fh:
|
||
fh.write("files\tverdict\tfolder\tquery\ttvdb_name\ttvdb_year\ttvdb_id\tnote\n")
|
||
for r in rows:
|
||
fh.write("\t".join(str(c) for c in r) + "\n")
|
||
print(f"sheet: {len(rows)} folders"
|
||
+ "".join(f" {t}={n}" for t, n in sorted(tally.items()))
|
||
+ f" -> {ARGS.out}")
|