Frank
2625ad1083
The Attack on Titan box set on the array names two files S03E2019 and S04E47370355, and the provider rows the metadata server recorded for them carry the same junk. EP_RANGE capped its first episode group at three digits without guarding the run, so the pass-2 plan filed both under E201 and E473 — slots that belong to real episodes — inside 進撃の巨人 (2013) [tvdbid-267440]. - guard the first episode group the way the second was already guarded, so a longer digit run states no marker at all; a glued resolution stays legal, because S01E01-1080p is still ONE episode - cap a provider row's episode at EP_MAX: such a row is renumbered from tvdb's title when the series' id answers, and parks with a reason that names the junk when nothing answers - split the operator-map exceptions into the two things they are: a folder that IS on the array whose files never became a series (bonus material with no owner: Fairy Tail, Kimetsu no Yaiba, Dorohedoro) is no longer reported as a folder the array no longer holds, and the missing-folder hint says the review pile counts as moved because the map reads top-level folders only - the vault's evidence list carries the law; the four tests were read red against the shipping planner before any of it changed Replanning the two real files now parks them under their own names with the reasons "a provider row of season 3 episode 2019" and "… season 4 episode 47370355".
1259 lines
56 KiB
Python
Executable File
1259 lines
56 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
r"""Plan a Jellyfin-format relayout of a TV library. DRY RUN ONLY: nothing moves.
|
|
|
|
media-shows.py [--root DIR] [--enum TSV] [--titles TSV] [--probed TSV]
|
|
[--tmdb-cache JSON] [--tmdb] [--movies DIR] [--park DIR]
|
|
[--tvdb-ids TSV] [--out TSV] [--exceptions TXT]
|
|
|
|
Inputs are the TSVs scripts/media-census.py exports: `enumerate` lists what
|
|
exists, `jellyfin` says which episode each file IS (its series' TMDb and Tvdb
|
|
ids, and the episode title the row carries), and `probe` is optional duration
|
|
evidence for the movie rule. Every name and title answer lives in the cache JSON
|
|
so a plan stays reproducible: answers are read from it, and a miss is looked up
|
|
only with --tmdb/--tvdb and their credentials (a key never enters argv, and
|
|
never the cache).
|
|
|
|
Target contract (jellyfin.org/docs/general/server/media/shows):
|
|
|
|
<root>/<Series> (<year>)/Season <NN>/<Series> (<year>) - SxxEyy - <Title>.ext
|
|
one file holding two episodes SxxE01-E02 specials Season 00
|
|
a documented part of an episode SxxE03 Part 1 sidecars <base>.<lang>.srt
|
|
bonus material <series>/trailers|featurettes|… — never a season
|
|
|
|
The series name and the episode title are the ones the IDENTITY authority — tvdb
|
|
— states for the id the metadata server matched, not the words a release folder
|
|
or a server's folder-derived Series name happens to carry.
|
|
|
|
A folder tvdb's search cannot state is not a folder tvdb does not know: the
|
|
record exists, it is titled in the show's own language, and the English name
|
|
rides in aliases the search never reads. --tvdb-ids is the human door for those:
|
|
one line per pack folder, `folder <TAB> tvdb id <TAB> note`, every id checked by
|
|
hand against the record it names. It is an input file like the census, so a plan
|
|
stays reproducible, and every folder it touched is listed in the exceptions —
|
|
one wrong row is a whole pack renamed to the wrong show.
|
|
|
|
PLAN is the TSV scripts/media-apply.py replays: `action src dst kind`, every
|
|
mkdir row before any move, and `keep` for a file already where the contract puts
|
|
it. EXCEPTIONS is the review list: everything parked, and why.
|
|
|
|
The authority chain, strongest first, for one video file:
|
|
|
|
1. a provider row — Jellyfin matched this exact path and numbered it. The
|
|
row's NAME is only evidence once an id backs it: the server names a Series
|
|
after the folder it scanned, so `24 (2001) Season 1` is a folder echo, and
|
|
the series' tvdb id is what undoes it.
|
|
2. the file's own SxxEyy / 1x04 marker, the series taken from a sibling's
|
|
provider row, else from the nearest ancestor folder that has one, else from
|
|
a registered show its own words name, else — outside the review pile — from
|
|
the release folder's words. Inside a hidden top-level folder a file may
|
|
MATCH a registered show but never INVENT one: invention is what filled the
|
|
pile to begin with.
|
|
3. nothing. An unnumbered file is PARKED under --park with its relative path
|
|
kept: an invented `Season 01` steals an episode number from the file that
|
|
really is S01E01, and a wrong number is worse than no number.
|
|
|
|
Guards, and why each is here:
|
|
|
|
* One series, one folder, one year: the year ANY provider row of a series
|
|
states is shared by all of them, so a box set split across five release
|
|
folders lands inside ONE series folder instead of five.
|
|
* Two files claiming one episode: the bigger keeps the standard name, the other
|
|
parks with the reason. Nothing is overwritten and nothing is deleted — which
|
|
encode survives is Frank's call, and a parked file is one rename from home.
|
|
* A file with no episode evidence that runs long and that the MOVIE provider
|
|
names leaves the TV library for --movies under the movie naming contract.
|
|
Long and unnamed parks; short and unnumbered is bonus material.
|
|
* Series art and the show nfo follow the SERIES, a season image follows its
|
|
SEASON, a subtitle or episode nfo follows the episode's new base — an episode
|
|
moved without its subtitles is a half-move.
|
|
* Hidden files, macOS forks, qBittorrent resume parts are never moved. A file
|
|
already inside --park is never moved either: that is what makes re-planning
|
|
stable. --park belongs OUTSIDE the library — parked files inside it get
|
|
indexed as one show named after the park folder, and a row like that is a
|
|
folder echo, never evidence (the guard in `row_names_the_park`).
|
|
|
|
The plan crosses three trees (the library, its park, the movie library), so the
|
|
executor runs with --root at the share that holds them, on its own journal —
|
|
one journal is one undo:
|
|
|
|
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
|
|
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
|
|
--tvdb --tmdb --out /mnt/entropy/media-tv-plan.tsv \
|
|
--park /mnt/entropy/media-park/TVShows
|
|
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
|
--root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
|
|
"""
|
|
import argparse
|
|
import collections
|
|
import importlib.util
|
|
import json
|
|
import os
|
|
import re
|
|
import sys
|
|
import urllib.parse
|
|
import urllib.request
|
|
|
|
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
|
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
|
|
help="the TV library the plan reorganises")
|
|
ARGS.add_argument("--enum", default="/tmp/tv-enum.tsv",
|
|
help="file census (media-census.py enumerate)")
|
|
ARGS.add_argument("--titles", default="/tmp/jellyfin-tv.tsv",
|
|
help="the provider's numbering (media-census.py jellyfin)")
|
|
ARGS.add_argument("--probed", default="",
|
|
help="ffprobe census for the movie rule; absent = no durations")
|
|
ARGS.add_argument("--tmdb-cache", default="/tmp/tv-names.json",
|
|
help="name answers (TMDb and Tvdb) as ONE input file, so a plan "
|
|
"is reproducible; keys carry the provider they came from")
|
|
ARGS.add_argument("--tmdb", action="store_true",
|
|
help="look up cache misses at TMDb (needs TMDB_API_KEY)")
|
|
ARGS.add_argument("--tvdb", action="store_true",
|
|
help="look up cache misses at Tvdb (needs the credentials file)")
|
|
ARGS.add_argument("--tvdb-credentials",
|
|
default=os.path.expanduser("~/.config/secrets/tvdb"),
|
|
help="file with an api_key= line (and a pin= line for a "
|
|
"user-issued key); mode 600, and never the cache")
|
|
ARGS.add_argument("--tvdb-ids",
|
|
default=os.path.join(
|
|
os.path.dirname(os.path.abspath(__file__)),
|
|
"media-shows-tvdb-ids.tsv"),
|
|
help="the operator's map: folder <TAB> tvdb id <TAB> note, for the\n"
|
|
"packs whose name tvdb's search cannot state (anime is titled\n"
|
|
"by its own record, the English name rides in aliases). Used\n"
|
|
"only when the file exists; a line without a numeric id stops\n"
|
|
"the plan, because a typo renames a whole pack to the wrong\n"
|
|
"show.")
|
|
ARGS.add_argument("--movies", default="/mnt/entropy/Movies",
|
|
help="movie library a mis-filed film moves into")
|
|
ARGS.add_argument("--park", default="",
|
|
help="review pile OUTSIDE the library (default: <root>/_unsorted)")
|
|
ARGS.add_argument("--out", default="/tmp/tv-plan.tsv")
|
|
ARGS.add_argument("--exceptions", default="/tmp/tv-exceptions.txt")
|
|
ARGS = ARGS.parse_args()
|
|
|
|
ROOT = os.path.abspath(ARGS.root)
|
|
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
|
|
MOVIES = os.path.abspath(ARGS.movies)
|
|
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
|
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
|
eng = importlib.util.module_from_spec(_spec)
|
|
_spec.loader.exec_module(eng)
|
|
|
|
try:
|
|
CACHE = json.load(open(ARGS.tmdb_cache))
|
|
except (OSError, ValueError):
|
|
CACHE = {}
|
|
API_KEY = os.environ.get("TMDB_API_KEY", "")
|
|
TMDB = {"tries": 0, "failed": 0}
|
|
|
|
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
|
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".rm",
|
|
".rmvb", ".iso", ".img"}
|
|
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
|
|
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
|
|
NFO_EXT = {".nfo", ".xml"}
|
|
# The container's own word, for a sidecar named "episode.mkv.srt": that component
|
|
# is not a language flag, and both matchers read the plain form better.
|
|
CONTAINER = {e[1:] for e in VIDEO_EXT}
|
|
# Files that are never anybody's business: hidden, forks, resume state, thumbnails.
|
|
SKIP_NAME = re.compile(r"^(\.|#|dl_state|@eaDir|Thumbs\.db|desktop\.ini)|\.!q$")
|
|
# Show-level metadata keeps its own name at the series root; season-level at the
|
|
# season root; folder.jpg stays with the folder it was written for.
|
|
SERIES_META = re.compile(r"^(tvshow|series|show)(\.nfo|\.xml)?$"
|
|
r"|^(poster|tvposter|banner|clearlogo|clearart|logo|art"
|
|
r"|landscape|fanart|showbackdrop|backdrop)\d*\."
|
|
r"(jpg|jpeg|png|webp|bmp)$", re.I)
|
|
SEASON_META = re.compile(r"^season\.nfo$", re.I)
|
|
FOLDER_IMAGE = re.compile(r"^folder\.(jpg|jpeg|png|webp|bmp)$", re.I)
|
|
SEASON_DIR_RE = re.compile(r"(?i)^\s*(?:season[\s._-]*(\d{1,3})|s(\d{1,2}))\s*$")
|
|
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
|
# SxxEyy with an optional second episode, plus the 1x04 form. Both episode numbers
|
|
# are guarded so a longer digit run never reads as an episode. A resolution glued
|
|
# on stays legal — "S01E01-1080p" is ONE episode — while "S03E2019" and
|
|
# "S04E47370355" (both real names on the array, from a box set whose provider rows
|
|
# say the same junk) state no episode at all: the run is an airdate or a pack
|
|
# index, and the first three digits of it would steal E201 and E473 from the
|
|
# episodes that own them. No marker means the file is numbered by its provider
|
|
# row, or by tvdb's title, or it parks.
|
|
EP_RANGE = re.compile(r"(?i)(?<![a-z0-9])s(?:eason[\s._-]*)?(\d{1,2})[\s._-]*"
|
|
r"e(?:p(?:isode)[\s._-]*)?(\d{1,3})(?![\dpP])"
|
|
r"(?:[\s._-]*(?:-|to[\s._-]*|ep)[\s._-]*(?:s\d{1,2}[\s._-]*)?e?"
|
|
r"(\d{1,3})(?![\dpP]))?")
|
|
NX_RE = re.compile(r"(?i)(?<![a-z0-9])(\d{1,2})x(\d{2,3})(?![a-z0-9])")
|
|
PART_RE = re.compile(r"(?i)(?:^|[\s._-])(?:part|pt|cd|dvd|disc|disk)[\s._-]?"
|
|
r"([1-9]|i{1,3})(?:$|[\s._-])")
|
|
# Jellyfin reads these subfolder names as bonus material. The planner emits only
|
|
# these spellings, so 40k extras end up in nine folders instead of nine hundred.
|
|
EXTRA_DIRS = [
|
|
("trailers", r"trailer|teaser|promo|advert|commercial|tv[\s._-]spot|preview"),
|
|
("deleted scenes", r"deleted"),
|
|
("behind the scenes", r"behind[\s._-]the[\s._-]scenes|making[\s._-]of|bts"
|
|
r"|b[\s._-]roll|gag[\s._-]reel|blooper|outtake|archival"
|
|
r"|video[\s._-]archive|scene[\s._-]selection|menu|gallery"
|
|
r"|stills?|visual[\s._-]effects|workprint"),
|
|
("interviews", r"interview|cast[\s._-]and[\s._-]crew|audition|commentary"),
|
|
("scene cuts", r"scene[\s._-]cuts?|scenecuts"),
|
|
("featurettes", r"featurettes?"),
|
|
("shorts", r"shorts?"),
|
|
("clips", r"clips?"),
|
|
]
|
|
# Jellyfin's own ExtraType values, folded onto the same folder names.
|
|
EXTRA_KIND = {"trailer": "trailers", "tmbd": "trailers", "intro": "other",
|
|
"movie": "other", "scene": "scene cuts", "clip": "clips",
|
|
"interview": "interviews", "behindthescenes": "behind the scenes",
|
|
"deleted": "deleted scenes", "other": "other", "epilogue": "other",
|
|
"opening": "other", "featurette": "featurettes", "short": "shorts"}
|
|
SHORT_MAX = 45 * 60 # over this, a file with no episode evidence is film-shaped
|
|
# An episode number above this is not an episode number: the array's box sets
|
|
# carry 2019 (an airdate fragment) and 47370355 (a pack index) in the column the
|
|
# metadata server recorded as the match. tvdb's episode list is the only door on
|
|
# such a row; nothing files at E1000 or beyond.
|
|
EP_MAX = 999
|
|
|
|
|
|
def rel(p):
|
|
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
|
|
|
|
|
|
def clean(name):
|
|
"""A name the filesystem and both matchers can read."""
|
|
name = BAD_CHARS.sub("-", name).replace("\\", "-").strip(" .")
|
|
return name[:180] or "untitled"
|
|
|
|
|
|
def title_of(text):
|
|
"""The engine's parser, applied to a name that has dots but no extension:
|
|
splitext would read "The.Examples" as stem "The", so a sentinel extension is
|
|
added and the parser sees an ordinary media name."""
|
|
return eng.title_of(text + ".mkv")
|
|
|
|
|
|
def markers(name):
|
|
"""(season, [episodes], position) the episode markers a name states; None when
|
|
it states none. The position is where the series' own words stop."""
|
|
stem = os.path.splitext(name)[0]
|
|
m = EP_RANGE.search(stem)
|
|
if m:
|
|
eps = [int(m.group(2))] + ([int(m.group(3))] if m.group(3) else [])
|
|
return int(m.group(1)), eps, m.start()
|
|
m = NX_RE.search(stem)
|
|
if m:
|
|
return int(m.group(1)), [int(m.group(2))], m.start()
|
|
return None
|
|
|
|
|
|
LEAD_IDX_RE = re.compile(r"^\s*(\d{1,3})\s*[-\u2013.\s]\s*[A-Za-z]")
|
|
PACK_IDX_RE = re.compile(r"^\s*\d+[a-z]?[.)]\s+")
|
|
|
|
|
|
def num(x):
|
|
try:
|
|
return int(x)
|
|
except (TypeError, ValueError):
|
|
return 0
|
|
|
|
|
|
def folder_season(path):
|
|
"""The season the FOLDER states (`Season 3`, `S3`). A human wrote it, and where
|
|
a scraped row disagrees this is the one that survives a re-plan."""
|
|
m = SEASON_DIR_RE.match(os.path.basename(os.path.dirname(path)))
|
|
return int(next(g for g in m.groups() if g)) if m else None
|
|
|
|
|
|
def leading_index(name):
|
|
"""The number a file leads with inside a season folder — `01 Kyle MacLachlan,
|
|
Sinead OC.avi` is episode 1. Both matchers read it that way."""
|
|
m = LEAD_IDX_RE.match(os.path.splitext(name)[0])
|
|
return int(m.group(1)) if m else None
|
|
|
|
|
|
def tmdb_request(url):
|
|
"""One TMDb call, authenticated. The two credential shapes are NOT
|
|
interchangeable and the difference is a 401 on every single call: a v3 key
|
|
travels as the `api_key` query parameter, and only a v4 request token is
|
|
accepted as a Bearer header. A token in argv is world-readable in
|
|
/proc/<pid>/cmdline — this URL is built inside the process and never reaches
|
|
argv, so the query form is safe here."""
|
|
TMDB["tries"] += 1
|
|
if API_KEY.count(".") == 2: # three segments: a v4 JWT
|
|
return http_json(url, headers={"Authorization": "Bearer " + API_KEY})
|
|
return http_json(url + ("&" if "?" in url else "?") + "api_key="
|
|
+ urllib.parse.quote(API_KEY))
|
|
|
|
|
|
def tmdb(kind, query):
|
|
"""A provider answer for a query: the cache first, TMDb once, then never again."""
|
|
ck = f"{kind}:{query}"
|
|
if ck in CACHE:
|
|
hit = CACHE[ck]
|
|
# A cache an operator edited by hand can carry the year as a number; the
|
|
# comparison below is text, so the wire form is imposed on every hit.
|
|
return None if hit is None else {**hit, "year": str(hit.get("year") or "")}
|
|
if not (ARGS.tmdb and API_KEY):
|
|
return None
|
|
path = "movie" if kind == "movie" else "tv"
|
|
url = ("https://api.themoviedb.org/3/search/" + path + "?query="
|
|
+ urllib.parse.quote(query) + "&include_adult=false")
|
|
try:
|
|
found = tmdb_request(url).get("results") or []
|
|
except Exception as e: # noqa: BLE001
|
|
TMDB["failed"] += 1
|
|
print(f"tmdb {kind} lookup failed for {query!r}: {e}", file=sys.stderr)
|
|
return None
|
|
if not found:
|
|
CACHE[ck] = None
|
|
return None
|
|
top = found[0]
|
|
# The two search endpoints date their rows differently: movies carry
|
|
# `release_date`, tv carries `first_air_date`. Reading only one of them leaves
|
|
# every film answer year-less, and a year-less answer can never be refused by
|
|
# the year guard below.
|
|
when = str(top.get("year") or "") or \
|
|
str(top.get("release_date") or top.get("first_air_date") or "")[:4]
|
|
CACHE[ck] = {"name": (top.get("title") or top.get("name") or "").strip(),
|
|
"year": when, "id": top.get("id")}
|
|
return CACHE[ck]
|
|
|
|
|
|
# ---------------------------------------------------------------- tvdb, the identity
|
|
# tvdb names the show and the episode: it is the only provider whose strings are
|
|
# the ones the operator asked for, so its answer replaces a folder-derived Series
|
|
# name. The login mints a JWT per run from a mode-600 file; neither the key nor
|
|
# the token reaches argv, the cache, or the plan. A provider that will not answer
|
|
# never invents a name: every lookup here returns None, and None keeps the
|
|
# provider's own words and lands in the exceptions.
|
|
TVDB = {"jwt": None, "dead": False}
|
|
|
|
|
|
def http_json(url, body=None, headers=None):
|
|
req = urllib.request.Request(
|
|
url, data=json.dumps(body).encode() if body is not None else None,
|
|
headers={"Accept": "application/json", "Content-Type": "application/json",
|
|
**(headers or {})})
|
|
with urllib.request.urlopen(req, timeout=30) as r:
|
|
return json.load(r)
|
|
|
|
|
|
def tvdb_login():
|
|
"""The run's JWT, or None: a project key needs only api_key=, a user-issued
|
|
key also needs the pin."""
|
|
if TVDB["jwt"] is not None:
|
|
return TVDB["jwt"] or None
|
|
TVDB["jwt"] = ""
|
|
try:
|
|
lines = open(ARGS.tvdb_credentials).read().splitlines()
|
|
except OSError:
|
|
print(f"tvdb: no credentials at {ARGS.tvdb_credentials}", file=sys.stderr)
|
|
return None
|
|
sec = {}
|
|
for line in lines:
|
|
line = line.strip()
|
|
if line and not line.startswith("#") and "=" in line:
|
|
k, v = line.split("=", 1)
|
|
sec[k.strip()] = v.strip()
|
|
if not sec.get("api_key"):
|
|
print(f"tvdb: no api_key in {ARGS.tvdb_credentials}", file=sys.stderr)
|
|
return None
|
|
body = {"apiKey": sec["api_key"]}
|
|
if sec.get("pin"):
|
|
body["token"] = sec["pin"]
|
|
try:
|
|
TVDB["jwt"] = (http_json("https://api4.thetvdb.com/v4/login", body)
|
|
.get("data") or {}).get("token") or ""
|
|
except Exception as e: # noqa: BLE001
|
|
print(f"tvdb login failed: {e}", file=sys.stderr)
|
|
return None
|
|
return TVDB["jwt"] or None
|
|
|
|
|
|
def tvdb(path):
|
|
"""One Tvdb v4 GET, or None when it cannot be answered."""
|
|
if not ARGS.tvdb or TVDB["dead"]:
|
|
return None
|
|
jwt = tvdb_login()
|
|
if not jwt:
|
|
TVDB["dead"] = True
|
|
return None
|
|
try:
|
|
return http_json("https://api4.thetvdb.com/v4" + path,
|
|
headers={"Authorization": "Bearer " + jwt}).get("data")
|
|
except Exception as e: # noqa: BLE001
|
|
print(f"tvdb {path} failed: {e}", file=sys.stderr)
|
|
return None
|
|
|
|
|
|
def tvdb_series(sid):
|
|
"""{name, year} for a tvdb series id."""
|
|
ck = f"tvdb:series:{sid}"
|
|
if ck in CACHE:
|
|
return CACHE[ck]
|
|
data = tvdb(f"/series/{sid}")
|
|
if not data:
|
|
return None
|
|
CACHE[ck] = {"name": (data.get("name") or "").strip(),
|
|
"year": (data.get("firstAired") or "")[:4]}
|
|
return CACHE[ck]
|
|
|
|
|
|
def tvdb_id_for_tmdb(mid):
|
|
"""The tvdb id TMDb records for a series id — the bridge for the bulk of this
|
|
library, which the server matched by TMDb."""
|
|
ck = f"tmdb:external:{mid}"
|
|
if ck in CACHE:
|
|
return CACHE[ck] or ""
|
|
if not (ARGS.tmdb and API_KEY):
|
|
return ""
|
|
try:
|
|
got = tmdb_request("https://api.themoviedb.org/3/tv/" + mid + "/external_ids")
|
|
except Exception as e: # noqa: BLE001
|
|
TMDB["failed"] += 1
|
|
print(f"tmdb external ids failed for {mid}: {e}", file=sys.stderr)
|
|
return ""
|
|
CACHE[ck] = str(got.get("tvdb_id") or "")
|
|
return CACHE[ck]
|
|
|
|
|
|
def tvdb_search(name):
|
|
"""The tvdb series a provider name states, or None: only tvdb's own spelling
|
|
counts, so a near-miss never renames a show."""
|
|
ck = f"tvdb:search:{name}"
|
|
if ck in CACHE:
|
|
return CACHE[ck]
|
|
for hit in tvdb("/search?query=" + urllib.parse.quote(name) + "&type=series") or []:
|
|
if eng.norm(hit.get("name") or "") != eng.norm(name):
|
|
continue
|
|
CACHE[ck] = {"name": (hit.get("name") or "").strip(),
|
|
"year": str(hit.get("year") or ""),
|
|
"id": str(hit.get("tvdb_id") or "")}
|
|
return CACHE[ck]
|
|
CACHE[ck] = None
|
|
return None
|
|
|
|
|
|
# Tvdb's default episode query answers AT MOST this many rows and states no
|
|
# total: One Piece (1999), Crayon Shin-chan (1992) and Detective Conan (1996)
|
|
# each answer exactly 500 while holding over a thousand. A list that long bounds
|
|
# nothing above itself, so it must never be read as "the series holds 500".
|
|
EPISODE_PAGE_CAP = 500
|
|
|
|
|
|
def tvdb_episodes(sid):
|
|
"""`season/episode` -> title for one series, in the one call tvdb answers.
|
|
Whether that call came back a cut-off page is recorded in the cache beside
|
|
it, so a warm cache answers the same question a cold one did."""
|
|
ck = f"tvdb:episodes:{sid}"
|
|
if ck not in CACHE:
|
|
data = tvdb(f"/series/{sid}/episodes/default")
|
|
if not data:
|
|
return None
|
|
rows = data.get("episodes") or []
|
|
CACHE[ck] = {f"{e.get('seasonNumber')}/{e.get('number')}":
|
|
(e.get("name") or "").strip() for e in rows}
|
|
CACHE[f"tvdb:capped:{sid}"] = len(rows) >= EPISODE_PAGE_CAP
|
|
return CACHE[ck]
|
|
|
|
|
|
def tvdb_list_capped(sid):
|
|
"""True when tvdb's episode answer for this series is a page, not a list."""
|
|
rows = tvdb_episodes(sid)
|
|
if not rows:
|
|
return False
|
|
return CACHE.get(f"tvdb:capped:{sid}", len(rows) >= EPISODE_PAGE_CAP)
|
|
|
|
|
|
|
|
# tvdb's titles read backwards, for the rows that carry a title and no numbering.
|
|
_TITLE_IDX = {}
|
|
|
|
|
|
def tvdb_title_index(sid):
|
|
"""tvdb's episode list as `title -> (season, episode)`. A title two episodes
|
|
share names no episode: both halves are dropped, because a file that parks is
|
|
honest and a file filed at the wrong number is a lie the server believes."""
|
|
if sid in _TITLE_IDX:
|
|
return _TITLE_IDX[sid]
|
|
index, tie = {}, set()
|
|
for key, name in (tvdb_episodes(sid) or {}).items():
|
|
season, _, ep = key.partition("/")
|
|
t = eng.norm(name)
|
|
if not t or t in tie:
|
|
continue
|
|
if t in index:
|
|
del index[t]
|
|
tie.add(t)
|
|
continue
|
|
try:
|
|
n_season, n_ep = int(season), int(ep)
|
|
except ValueError:
|
|
continue # half-numbers ("12.5") state no slot
|
|
index[t] = (n_season, n_ep)
|
|
_TITLE_IDX[sid] = index
|
|
return index
|
|
|
|
|
|
def tvdb_number(t):
|
|
"""The (season, episode) tvdb states for a provider row whose own numbering is
|
|
not a season at all. Jellyfin's date matcher files specials by AIRDATE, so the
|
|
row says season 2002 — a YEAR — and only the title the provider recorded says
|
|
which episode it is. No id, no title, or a title tvdb gives twice: no number,
|
|
and the file parks with the reason it always carried."""
|
|
sid = (t.get("tvdb") or "").strip()
|
|
title = (t.get("title") or "").strip()
|
|
if not sid or not title:
|
|
return None
|
|
return tvdb_title_index(sid).get(eng.norm(title))
|
|
|
|
# ------------------------------------------------------------------- the inputs
|
|
rows = []
|
|
for line in open(ARGS.enum):
|
|
size, ext, path = line.rstrip("\n").split("\t")
|
|
p = os.path.abspath(path)
|
|
base = os.path.basename(p)
|
|
if not p.startswith(ROOT + "/") or SKIP_NAME.search(base):
|
|
continue
|
|
if p.startswith(PARK + "/"):
|
|
continue # the park is already reviewed; re-parking is churn
|
|
rows.append({"bytes": int(size), "ext": ext.lower(), "path": p, "base": base})
|
|
held = {r["path"]: r for r in rows}
|
|
|
|
def row_names_the_park(path, name):
|
|
"""The library's own review pile is not a show. When the park sat INSIDE the
|
|
library the server indexed the whole folder as one series named after it, and
|
|
a row like that would rename a real show to the pile — so a row whose series is
|
|
the hidden top-level folder the file itself sits in is dropped. The server also
|
|
sanitizes the display name, dropping the leading underscore, so `unsorted` and
|
|
`_unsorted` are the same pile: compare them without their leading punctuation,
|
|
or the pile survives as a visible series folder."""
|
|
top = rel(path).split("/")[0]
|
|
return top.startswith("_") and eng.norm(name.lstrip("_")) == eng.norm(top.lstrip("_"))
|
|
|
|
|
|
def in_review_pile(path):
|
|
"""A file under the library's hidden top-level folder is review material by
|
|
that folder's own convention: its filename may MATCH a show a provider names
|
|
somewhere else, but it may never INVENT one — inventing one from a file's own
|
|
words is what filled the pile to begin with."""
|
|
return rel(path).split("/")[0].startswith("_")
|
|
|
|
|
|
titles = {}
|
|
with open(ARGS.titles) as fh:
|
|
cols = fh.readline().rstrip("\n").split("\t")
|
|
for line in fh:
|
|
f = line.rstrip("\n").split("\t")
|
|
f += [""] * (len(cols) - len(f))
|
|
row = dict(zip(cols, f))
|
|
p = os.path.abspath(row.get("file", ""))
|
|
if p in held and not row_names_the_park(p, row.get("series", "")):
|
|
titles[p] = row
|
|
|
|
dur = {}
|
|
if ARGS.probed and os.path.exists(ARGS.probed):
|
|
for line in open(ARGS.probed):
|
|
f = line.rstrip("\n").split("\t")
|
|
if len(f) > 2:
|
|
try:
|
|
dur[os.path.abspath(f[1])] = float(f[2] or 0)
|
|
except ValueError:
|
|
pass
|
|
|
|
video = [r for r in rows if r["ext"] in VIDEO_EXT]
|
|
other = [r for r in rows if r["ext"] not in VIDEO_EXT]
|
|
|
|
# What a folder already knows: one provider row names the series for every sibling
|
|
# that has no row of its own — that is how an anime pack gets its show.
|
|
by_dir = collections.defaultdict(list)
|
|
for r in video:
|
|
by_dir[os.path.dirname(r["path"])].append(r)
|
|
dir_series = {}
|
|
for d, rs in by_dir.items():
|
|
names = collections.Counter(titles[r["path"]]["series"] for r in rs
|
|
if r["path"] in titles and titles[r["path"]]["series"])
|
|
if names:
|
|
dir_series[d] = names.most_common(1)[0][0]
|
|
|
|
|
|
# The registry: every series a provider names, judged before any folder is. Folder
|
|
# words may resolve a file to one of these names; they may never invent one. That is
|
|
# the whole difference between "NCOP 01.mkv" inside the Gurren Lagann pack, which the
|
|
# pack names, and a new root folder called "01 NCOP".
|
|
registry = {}
|
|
for _p, _t in titles.items():
|
|
if _t["series"]:
|
|
registry.setdefault(eng.norm(_t["series"]), _t["series"])
|
|
|
|
|
|
def known_in(text):
|
|
"""The registered series named inside this text: the longest registered name
|
|
whose words appear in it as a run. "(CBB) Tengen Toppa Gurren Lagann - The
|
|
Complete Series (BD 1080p)" names "Tengen Toppa Gurren Lagann"."""
|
|
hay = " " + eng.norm(text) + " "
|
|
best = ""
|
|
for k, name in registry.items():
|
|
if len(k) >= 5 and f" {k} " in hay and len(k) > len(best):
|
|
best = name
|
|
return best
|
|
|
|
|
|
# Rows also name their own folders' ancestors, so an already-filed library stays
|
|
# where it is when it is re-planned.
|
|
for d, name in sorted(dir_series.items()):
|
|
a = os.path.dirname(d)
|
|
while a and a != ROOT and len(a) > len(ROOT):
|
|
if a not in dir_series:
|
|
got = known_in(os.path.basename(a))
|
|
if got:
|
|
dir_series[a] = got
|
|
a = os.path.dirname(a)
|
|
|
|
|
|
def series_of_dir(d):
|
|
"""The series an ancestor folder belongs to, nearest first: the folder holding
|
|
provider rows, else a folder whose own words name a registered show."""
|
|
while d and d != ROOT and len(d) > len(ROOT):
|
|
if d in dir_series:
|
|
return dir_series[d]
|
|
got = known_in(os.path.basename(d))
|
|
if got:
|
|
return got
|
|
d = os.path.dirname(d)
|
|
return ""
|
|
|
|
|
|
def extras_kind(path, extra_type):
|
|
"""The canonical bonus subfolder, from Jellyfin's own extra type first."""
|
|
if extra_type:
|
|
return EXTRA_KIND.get(eng.norm(extra_type).replace(" ", ""), "other")
|
|
for d in rel(path).split("/")[:-1]:
|
|
if eng.folder_is_bonus(d):
|
|
for name, rx in EXTRA_DIRS:
|
|
if re.search(rx, d, re.I):
|
|
return name
|
|
return "extras"
|
|
stem = os.path.splitext(os.path.basename(path))[0]
|
|
for name, rx in EXTRA_DIRS:
|
|
if re.search(rx, stem, re.I):
|
|
return name
|
|
return "extras"
|
|
|
|
|
|
# --------------------------------------------------------- the operator's map
|
|
# The map is the human answer to a name the authority cannot be searched by. It
|
|
# is read as an input file — like the census and the cache — so a plan stays
|
|
# reproducible from the files beside it, and it outranks a provider id: a box
|
|
# set's provider id is the server's own guess at a folder name, while a map row
|
|
# is an id someone checked against the record it names.
|
|
MAP = {} # top-level pack folder -> {"id", "note"}
|
|
MAP_SEEN, MAP_CLASH = set(), [] # what the map actually touched
|
|
if ARGS.tvdb_ids and os.path.exists(ARGS.tvdb_ids):
|
|
for _n, _line in enumerate(open(ARGS.tvdb_ids), 1):
|
|
_line = _line.rstrip("\n")
|
|
if not _line.strip() or _line.startswith("#"):
|
|
continue
|
|
cols = _line.split("\t")
|
|
folder = cols[0].strip()
|
|
sid = cols[1].strip() if len(cols) > 1 else ""
|
|
if not folder or not sid.isdigit():
|
|
print(f"{ARGS.tvdb_ids}:{_n}: not a numeric tvdb id: {_line}",
|
|
file=sys.stderr)
|
|
sys.exit(2)
|
|
if folder in MAP:
|
|
print(f"{ARGS.tvdb_ids}:{_n}: folder named twice: {folder}",
|
|
file=sys.stderr)
|
|
sys.exit(2)
|
|
MAP[folder] = {"id": sid,
|
|
"note": cols[2].strip() if len(cols) > 2 else ""}
|
|
|
|
|
|
def top_folder(path):
|
|
"""The library folder a file's pack sits in. An id belongs to a show, and a
|
|
show is a folder here, so that is what the map keys on."""
|
|
return os.path.relpath(path, ROOT).split(os.sep)[0]
|
|
|
|
|
|
def map_id_for(path, provider_id):
|
|
"""The id the operator states for this file's folder. The map wins over the
|
|
provider's — it is the only door for a name tvdb cannot state — and a
|
|
disagreement between the two is written where the operator reads it, never
|
|
resolved in silence."""
|
|
if not MAP:
|
|
return provider_id
|
|
top = top_folder(path)
|
|
entry = MAP.get(top)
|
|
if not entry:
|
|
return provider_id
|
|
MAP_SEEN.add(top)
|
|
if provider_id and provider_id != entry["id"]:
|
|
MAP_CLASH.append((top, provider_id, entry["id"]))
|
|
return entry["id"]
|
|
|
|
|
|
def map_note():
|
|
if not MAP:
|
|
return ""
|
|
return f" (+{len(MAP_SEEN)} folders from the operator's map)"
|
|
|
|
|
|
# ----------------------------------------------------------- classify every file
|
|
series = {} # norm(name) -> {name, year, tmdb}
|
|
claimants = [] # (row, name, season, [episodes])
|
|
extras = [] # (row, name, bonus folder)
|
|
films = [] # film-shaped, no episode evidence
|
|
unknown = [] # (row, reason)
|
|
|
|
|
|
def series_name(name):
|
|
"""The provider's name as a folder: the pack index a scene release leads with
|
|
("4a. Seed DESTINY") falls away, and the doubled spaces it leaves fold."""
|
|
return clean(re.sub(r"\s{2,}", " ", PACK_IDX_RE.sub("", name)).strip(" ."))
|
|
|
|
|
|
def note(name, year="", tmdb_id="", tvdb_id=""):
|
|
"""Register a series; the first name wins, the first year and ids stick."""
|
|
k = eng.norm(name)
|
|
s = series.setdefault(k, {"name": series_name(name), "year": "", "tmdb": "",
|
|
"tvdb": ""})
|
|
if year and not s["year"]:
|
|
s["year"] = str(year)
|
|
if tmdb_id and not s["tmdb"]:
|
|
s["tmdb"] = str(tmdb_id)
|
|
if tvdb_id and not s["tvdb"]:
|
|
s["tvdb"] = str(tvdb_id)
|
|
|
|
|
|
|
|
def as_extra(r, owner, extra_type):
|
|
"""Bonus material needs a series to file under; with none it is parked."""
|
|
if not owner:
|
|
unknown.append((r, "bonus material with no series to own it"))
|
|
return
|
|
note(owner, tvdb_id=map_id_for(r["path"], ""))
|
|
extras.append((r, owner, extras_kind(r["path"], extra_type)))
|
|
|
|
for r in video:
|
|
p, t = r["path"], titles.get(r["path"])
|
|
marked = markers(r["base"])
|
|
fs = folder_season(p)
|
|
if t and t["extra"]:
|
|
as_extra(r, t["series"] or series_of_dir(os.path.dirname(p)), t["extra"])
|
|
continue
|
|
row = bool(t and t["series"] and (t["season"] or t["episode"]))
|
|
if marked or row:
|
|
# The provider row names the SHOW: its registered name, its year. The number
|
|
# that orders the file comes from the file's own name first, then the folder
|
|
# it sits in, then the row — scraped pack rows number episodes badly, where
|
|
# every file under SNL 16-20/Season 16..19 carries season 16 episode 20.
|
|
owner = (t["series"] if t and t["series"]
|
|
else series_of_dir(os.path.dirname(p))
|
|
or known_in(r["base"])
|
|
or ("" if in_review_pile(p)
|
|
else title_of(r["base"][:marked[2]])[0]))
|
|
if not eng.norm(owner).replace(" ", ""):
|
|
unknown.append((r, "an episode marker with no series name to hang it on"))
|
|
continue
|
|
if marked:
|
|
season, eps = marked[0], marked[1]
|
|
else:
|
|
season = fs if fs is not None else num(t["season"])
|
|
ep = leading_index(r["base"]) or num(t["episode"])
|
|
if not 0 <= season <= 200 or not 1 <= ep <= EP_MAX:
|
|
# Season 0 IS a season: specials live in Season 00 by Jellyfin's own
|
|
# contract. A season that is a YEAR is the date matcher's doing and an
|
|
# episode above EP_MAX is a box set's airdate or pack index; neither is
|
|
# a number, and the only authority on either is tvdb's episode list.
|
|
recovered = tvdb_number(t)
|
|
if not recovered:
|
|
unknown.append(
|
|
(r, f"a provider row of season {season} episode {ep}"))
|
|
continue
|
|
season, ep = recovered
|
|
eps = [ep]
|
|
claimants.append((r, owner, season, eps))
|
|
note(owner, t["year"] if t else "", t["tmdb"] if t else "",
|
|
map_id_for(r["path"], t["tvdb"] if t else ""))
|
|
continue
|
|
owner = series_of_dir(os.path.dirname(p))
|
|
if fs is not None and owner and leading_index(r["base"]):
|
|
claimants.append((r, owner, fs, [leading_index(r["base"])]))
|
|
note(owner, tvdb_id=map_id_for(r["path"], ""))
|
|
continue
|
|
if eng.is_extra(p, title_of(r["base"])[0], dur.get(p, 0.0), ROOT):
|
|
as_extra(r, owner, "")
|
|
continue
|
|
if dur.get(p, 0.0) >= SHORT_MAX or (p not in dur and title_of(r["base"])[1] != "?"):
|
|
films.append(r)
|
|
continue
|
|
unknown.append((r, "no episode marker, no provider row, nothing long enough to be a film"))
|
|
|
|
# A provider that never states a year still has one: the search is exact, so a
|
|
# miss parks nothing — the folder simply carries no year, which Jellyfin accepts.
|
|
for k, s in series.items():
|
|
if not s["year"]:
|
|
got = tmdb("tv", s["name"])
|
|
if got and eng.norm(got["name"]) == k:
|
|
s["year"] = got["year"]
|
|
|
|
# tvdb is the identity authority. The row's own tvdb id is the door; a TMDb id
|
|
# crosses through TMDb's external ids; a bare name counts only when tvdb spells
|
|
# it the same and the years do not disagree. A series nothing resolves keeps the
|
|
# provider's words AND says so in the exceptions, which is where the operator
|
|
# finds the folders the plan did not rename to authority.
|
|
for s in series.values():
|
|
got = tvdb_series(s["tvdb"]) if s["tvdb"] else None
|
|
if not got and s["tmdb"]:
|
|
sid = tvdb_id_for_tmdb(s["tmdb"])
|
|
if sid:
|
|
s["tvdb"] = sid
|
|
got = tvdb_series(sid)
|
|
if not got:
|
|
hit = tvdb_search(s["name"])
|
|
if hit and hit["id"] and (not s["year"] or not hit["year"]
|
|
or hit["year"] == s["year"]):
|
|
s["tvdb"] = hit["id"]
|
|
got = hit
|
|
if got and got["name"]:
|
|
s["canon"], s["canon_year"] = got["name"], got["year"]
|
|
|
|
# One identity, one folder: the provider's spellings collapse to the shows tvdb
|
|
# says they are, so a box set split over five release folders is ONE series and
|
|
# two files claiming one episode MEET inside the duplicate rule below.
|
|
alias, canon = {}, {}
|
|
for k, s in series.items():
|
|
key = f"id:{s['tvdb']}" if s["tvdb"] else k
|
|
alias[k] = key
|
|
keep = canon.setdefault(key, s)
|
|
if keep is not s:
|
|
for f in ("year", "tvdb", "canon", "canon_year"):
|
|
if not keep.get(f) and s.get(f):
|
|
keep[f] = s[f]
|
|
|
|
if ARGS.tmdb or ARGS.tvdb:
|
|
with open(ARGS.tmdb_cache, "w") as fh:
|
|
json.dump(CACHE, fh, indent=1, sort_keys=True)
|
|
|
|
|
|
def identity(s):
|
|
"""(name, year) a folder and its files carry: tvdb's whenever it answered,
|
|
the provider's own words only for a series nothing resolved."""
|
|
if s.get("canon"):
|
|
return s["canon"], s["canon_year"] or s["year"]
|
|
return s["name"], s["year"]
|
|
|
|
|
|
def series_key(name):
|
|
"""The merged identity a provider spelling belongs to."""
|
|
return alias.get(eng.norm(name), eng.norm(name))
|
|
|
|
|
|
def label_of(s):
|
|
"""`<Series> (<year>)`, the form both matchers read. A provider's name can
|
|
arrive WITH the year already on it — TVDb hands back `Acapulco (2021)` for the
|
|
reboot — and appending a second one names a folder nothing will match again."""
|
|
nm, yr = identity(s)
|
|
nm = clean(nm)
|
|
if not yr or re.search(r"\(\d{4}\)\s*$", nm):
|
|
return nm
|
|
return f"{nm} ({yr})"
|
|
|
|
|
|
def contradicted_seasons(s):
|
|
"""Seasons whose numbering tvdb's own episode list contradicts — the one place
|
|
a tvdb title can be a confident lie. An anime pack numbered by its release group
|
|
books S05E01 in a season tvdb does not hold, or books more episodes there than
|
|
tvdb lists, and the slot its number points at is a DIFFERENT episode. Books <=
|
|
tvdb's count stays plausible: a partial pack is the normal shape. A list the
|
|
authority cut off at its page limit (EPISODE_PAGE_CAP) contradicts nothing —
|
|
it says nothing about the episodes it never sent."""
|
|
if "contradicted" in s:
|
|
return s["contradicted"]
|
|
bad = set()
|
|
eps = None if not s["tvdb"] or tvdb_list_capped(s["tvdb"]) \
|
|
else tvdb_episodes(s["tvdb"])
|
|
if eps:
|
|
have = collections.Counter()
|
|
for k in eps:
|
|
season, _, _ep = k.partition("/")
|
|
if season.isdigit():
|
|
have[int(season)] += 1
|
|
booked = collections.defaultdict(set)
|
|
for _r, name, season, claimed in claimants:
|
|
if canon.get(series_key(name)) is s:
|
|
booked[season].update(claimed)
|
|
bad = {n for n, seen in booked.items() if len(seen) > have.get(n, 0)}
|
|
s["contradicted"] = bad
|
|
return bad
|
|
|
|
|
|
def episode_title(r, s, season, ep):
|
|
"""The title this episode carries: tvdb's for the id the series resolved to,
|
|
else the title the matched row states, else nothing at all — a dangling ` - ` is
|
|
not a title, and an invented one is worse than none. A season tvdb's own list
|
|
contradicts gets no tvdb title: its number points somewhere else, and a wrong
|
|
title is worse than a bare one."""
|
|
if s["tvdb"] and season not in contradicted_seasons(s):
|
|
t = ((tvdb_episodes(s["tvdb"]) or {}).get(f"{season}/{ep}") or "").strip()
|
|
if t:
|
|
return clean(t)
|
|
t = ((titles.get(r["path"]) or {}).get("title") or "").strip()
|
|
return clean(t) if t else ""
|
|
|
|
|
|
def series_dir(name):
|
|
"""One series, one folder — and the folder CARRIES the tvdb id. Both matchers
|
|
read `[tvdbid-NNNN]` (Jellyfin's Tvdb agent, Plex's scanner), so a show whose
|
|
title is ambiguous resolves by number instead of by a human guessing which
|
|
folder is which. The id rides the FOLDER only: Jellyfin gives a file name 180
|
|
characters and the episode title needs the room. A series nothing resolved
|
|
keeps the provider's words with no tag at all."""
|
|
s = canon.get(series_key(name))
|
|
if not s:
|
|
return f"{ROOT}/{clean(series_name(name))}"
|
|
tag = f" [tvdbid-{s['tvdb']}]" if s["tvdb"] else ""
|
|
return f"{ROOT}/{label_of(s)}{tag}"
|
|
|
|
|
|
# ------------------------------------------------------------------- the plan
|
|
moves, keeps, parked = [], [], []
|
|
reasons = collections.defaultdict(list)
|
|
used = {} # dst -> the file that claimed it
|
|
|
|
|
|
def book(r, dst_dir, name, kind):
|
|
"""Claim one destination for one file, or park the file when it is taken."""
|
|
dst = f"{dst_dir}/{name}"
|
|
if dst == r["path"]:
|
|
keeps.append((r["path"], dst, kind))
|
|
return
|
|
if dst in used:
|
|
park(r, "something else in the plan already files there")
|
|
return
|
|
if dst in held:
|
|
park(r, f"{rel(dst)} already occupies that name")
|
|
return
|
|
used[dst] = r["path"]
|
|
moves.append((r["path"], dst, kind))
|
|
|
|
|
|
def park(r, reason):
|
|
parked.append((r, reason))
|
|
reasons[reason].append(r)
|
|
dst = f"{PARK}/{rel(r['path'])}"
|
|
if dst in used or dst in held:
|
|
dst = f"{PARK}/{rel(r['path'])}.{len(reasons)}"
|
|
used[dst] = r["path"]
|
|
moves.append((r["path"], dst, "park"))
|
|
|
|
|
|
# Episodes first: two files claiming one number is a decision Frank makes, not one
|
|
# the plan sneaks — the bigger copy takes the standard name, the other waits.
|
|
by_ep = collections.defaultdict(list)
|
|
for r, name, season, eps in claimants:
|
|
for e in eps:
|
|
by_ep[(series_key(name), season, e)].append(r)
|
|
winners, losers = set(), {}
|
|
for (key, season, e), rs in by_ep.items():
|
|
rs.sort(key=lambda r: (-r["bytes"], r["path"]))
|
|
winners.add(rs[0]["path"])
|
|
for r in rs[1:]:
|
|
losers.setdefault(r["path"],
|
|
f"duplicate of {canon[key]['name']} S{season:02d}E{e:02d}"
|
|
" (the bigger copy keeps the standard name)")
|
|
for r, name, season, eps in claimants:
|
|
if r["path"] in losers:
|
|
park(r, losers[r["path"]])
|
|
continue
|
|
s = canon[series_key(name)]
|
|
stem = f"{label_of(s)} - S{season:02d}E" + "-E".join(f"{e:02d}" for e in eps)
|
|
title = episode_title(r, s, season, eps[0])
|
|
if title:
|
|
stem += f" - {title}"
|
|
part = PART_RE.search(os.path.splitext(r["base"])[0])
|
|
if part and len(eps) == 1:
|
|
stem += f" Part {part.group(1).upper()}"
|
|
book(r, f"{series_dir(name)}/Season {season:02d}", stem + r["ext"], "tv")
|
|
|
|
for r, name, kind in extras:
|
|
book(r, f"{series_dir(name)}/{kind}",
|
|
clean(os.path.splitext(r["base"])[0]) + r["ext"], "extra")
|
|
|
|
# A film inside the TV library leaves for the movie library only when the movie
|
|
# provider answers with a title that matches the name and, when the name states a
|
|
# year, that year too.
|
|
for r in films:
|
|
parsed, year = title_of(r["base"])
|
|
got = tmdb("movie", parsed)
|
|
# The provider's title must be a run of words the FILE itself states. The query
|
|
# is only what title_of could salvage from the name, and a short one ("russia")
|
|
# lets the top hit be a different work entirely: a 1963 Bond film is not a 1991
|
|
# documentary episode, and Special Edition I is not Special Edition III.
|
|
said = eng.norm(os.path.splitext(r["base"])[0]).split()
|
|
named = eng.norm(got["name"]).split() if got else []
|
|
if not got or not eng.mentions(named, said) \
|
|
or (year != "?" and got["year"] and got["year"] != year):
|
|
park(r, "long file with no episode evidence, and no movie provider names it")
|
|
continue
|
|
label = f"{got['name']} ({got['year']})" if got["year"] else got["name"]
|
|
book(r, f"{MOVIES}/{clean(label)}", clean(label) + r["ext"], "movie")
|
|
|
|
# ------------------------------------------------- sidecars and folder metadata
|
|
moved = {} # src -> dst, for every booked file
|
|
for src, dst, _kind in moves:
|
|
moved[src] = dst
|
|
for src, dst, _kind in keeps:
|
|
moved[src] = dst
|
|
dst_dirs = collections.defaultdict(collections.Counter)
|
|
video_dst = collections.defaultdict(list)
|
|
for src, dst in moved.items():
|
|
if src in {r["path"] for r in video}:
|
|
video_dst[os.path.dirname(src)].append((src, dst))
|
|
dst_dirs[os.path.dirname(src)][os.path.dirname(dst)] += 1
|
|
|
|
|
|
def series_name_at(d):
|
|
"""The series the files of a folder belong to — a registered name, or nothing.
|
|
Folder words alone never name a show."""
|
|
names = [titles[src]["series"] for src, _ in video_dst.get(d, [])
|
|
if src in titles and titles[src]["series"]]
|
|
if names:
|
|
return collections.Counter(names).most_common(1)[0][0]
|
|
return series_of_dir(d)
|
|
|
|
|
|
def season_at(d):
|
|
"""The season number a folder carries: its own name, else its files' rows."""
|
|
m = SEASON_DIR_RE.match(os.path.basename(d))
|
|
if m:
|
|
return int(m.group(1) or m.group(2))
|
|
nums = [int(titles[src]["season"]) for src, _ in video_dst.get(d, [])
|
|
if src in titles and titles[src]["season"]]
|
|
return collections.Counter(nums).most_common(1)[0][0] if nums else None
|
|
|
|
|
|
def folder_dst(d):
|
|
"""Where a folder's own images go: the one place its files landed, or the
|
|
series root when they scattered across several seasons."""
|
|
c = dst_dirs.get(d)
|
|
if not c:
|
|
return None
|
|
if len(c) == 1:
|
|
return c.most_common(1)[0][0]
|
|
owner = series_name_at(d)
|
|
return series_dir(owner) if owner else None
|
|
|
|
|
|
def anchor_of(path, stem):
|
|
"""The video file this sidecar names: the longest file name it extends, in the
|
|
NEAREST folder that holds one. A scene pack buries its subtitles two levels
|
|
down (`Pack/Subs/<release-name>/<release>.jpn.srt`), so the search climbs to
|
|
the library root instead of stopping at the parent."""
|
|
d = os.path.dirname(path)
|
|
while len(d) >= len(ROOT) and d.startswith(ROOT):
|
|
best = None
|
|
for src, dst in video_dst.get(d, []):
|
|
base = os.path.basename(src)
|
|
for prefix in (os.path.splitext(base)[0], base):
|
|
if stem == prefix or stem.startswith(prefix + "."):
|
|
if best is None or len(prefix) > best[2]:
|
|
best = (src, dst, len(prefix))
|
|
if best:
|
|
return best[0], best[1]
|
|
up = os.path.dirname(d)
|
|
if up == d:
|
|
break
|
|
d = up
|
|
return None
|
|
|
|
|
|
def sidecar_name(r, src, dst):
|
|
"""The sidecar's own tail — language, flag, the media extension an nfo must
|
|
carry — hung on the video file's new base."""
|
|
stem = os.path.splitext(os.path.basename(src))[0]
|
|
tail = os.path.splitext(r["base"])[0][len(stem):]
|
|
# "Show.S01E01.mkv.srt" carries the container's name, not a flag: Jellyfin and
|
|
# Plex both read "Show S01E01.srt", so the container word is dropped here.
|
|
parts = tail.split(".")
|
|
if len(parts) > 1 and parts[1].lower() in CONTAINER:
|
|
tail = "".join("." + p for p in parts[2:])
|
|
if r["ext"] in NFO_EXT:
|
|
tail = os.path.splitext(os.path.basename(src))[1] + tail
|
|
return clean(os.path.splitext(os.path.basename(dst))[0] + tail + r["ext"])
|
|
|
|
|
|
for r in other:
|
|
p, base = r["path"], r["base"]
|
|
stem = os.path.splitext(base)[0]
|
|
d = os.path.dirname(p)
|
|
if r["ext"] in SUB_EXT or r["ext"] in NFO_EXT or r["ext"] in IMG_EXT:
|
|
a = anchor_of(p, stem)
|
|
if a:
|
|
book(r, os.path.dirname(a[1]), sidecar_name(r, a[0], a[1]),
|
|
"subtitle" if r["ext"] in SUB_EXT else
|
|
("nfo" if r["ext"] in NFO_EXT else "image"))
|
|
continue
|
|
if SERIES_META.match(base):
|
|
owner = series_name_at(d)
|
|
if owner:
|
|
book(r, series_dir(owner), base, "meta")
|
|
continue
|
|
elif SEASON_META.match(base):
|
|
owner, season = series_name_at(d), season_at(d)
|
|
if owner and season is not None:
|
|
book(r, f"{series_dir(owner)}/Season {season:02d}", base, "meta")
|
|
continue
|
|
elif r["ext"] in IMG_EXT or r["ext"] in NFO_EXT:
|
|
dest = folder_dst(d)
|
|
if dest:
|
|
book(r, dest, base, "meta")
|
|
continue
|
|
unknown.append((r, "not media, or media with nothing beside it to belong to"))
|
|
|
|
for r, reason in unknown:
|
|
park(r, reason)
|
|
|
|
|
|
# --------------------------------------------------------------------- output
|
|
def provider_broken():
|
|
"""TMDb was asked for names and did not answer: the plan is not the plan the
|
|
operator asked for, whatever the move count says."""
|
|
return bool(ARGS.tmdb and (not API_KEY or TMDB["failed"]))
|
|
|
|
|
|
def provider_note():
|
|
"""Provider health on the summary line. The 401s are already on stderr, and a
|
|
terminal that scrolled past them shows a plan that looks finished: the count
|
|
belongs where it cannot be missed, because a half-resolved plan renames
|
|
folders with the folder's own junk words."""
|
|
if not (ARGS.tmdb or ARGS.tvdb):
|
|
return ""
|
|
if not ARGS.tmdb:
|
|
return ", tmdb off (names from the cache and tvdb only)"
|
|
if not API_KEY:
|
|
return ", tmdb NOT ASKED (TMDB_API_KEY unset): NOT provider-complete"
|
|
if not TMDB["tries"]:
|
|
return ", tmdb cache-only (every name was already answered)"
|
|
done = TMDB["tries"] - TMDB["failed"]
|
|
return (f", tmdb {done}/{TMDB['tries']} answered"
|
|
+ (" — NOT provider-complete" if TMDB["failed"] else ""))
|
|
|
|
|
|
def write_plan():
|
|
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
|
Parents first: the executor creates directories in the order they are listed."""
|
|
chain = set()
|
|
for _s, dst, _k in moves:
|
|
d = os.path.dirname(dst)
|
|
while d and d not in (ROOT, MOVIES, PARK, os.path.dirname(d)):
|
|
chain.add(d)
|
|
d = os.path.dirname(d)
|
|
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
|
with open(ARGS.out, "w") as fh:
|
|
fh.write("action\tsrc\tdst\tkind\n")
|
|
for d in dirs:
|
|
fh.write(f"mkdir\t\t{d}\tdir\n")
|
|
for src, dst, kind in sorted(moves, key=lambda m: (m[1], m[0])):
|
|
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
|
for src, dst, kind in sorted(keeps, key=lambda m: (m[1], m[0])):
|
|
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
|
sized = sum(held[s]["bytes"] for s, _d, _k in moves if s in held)
|
|
print(f"plan: {len(moves) - len(parked)} moves, {len(parked)} parked"
|
|
f" ({sized / 2**40:.2f} TiB touched), {len(keeps)} already placed,"
|
|
f" {len(dirs)} folders{map_note()}{provider_note()} -> {ARGS.out}")
|
|
|
|
|
|
def write_exceptions():
|
|
no_year = sorted({s["name"] for s in canon.values() if not s["year"]})
|
|
no_tvdb = sorted({identity(s)[0] for s in canon.values() if not s.get("canon")})
|
|
clash = sorted({(identity(s)[0], tuple(sorted(contradicted_seasons(s))))
|
|
for s in canon.values() if contradicted_seasons(s)})
|
|
with open(ARGS.exceptions, "w") as fh:
|
|
fh.write(f"TV relayout exceptions for {ROOT}\n\n")
|
|
fh.write(f"Everything below is parked under {PARK}, keeping the relative path it\n")
|
|
fh.write("has today. Nothing is deleted and nothing is overwritten: a parked file\n")
|
|
fh.write("is one rename from wherever the plan is pointed next.\n\n")
|
|
if provider_broken():
|
|
fh.write("WARNING: " + provider_note().strip(", ").replace(": NOT", " — NOT")
|
|
+ "\n Names and episode titles that should have come from TMDb are\n"
|
|
" missing, so folders keep the metadata provider's words. Fix the\n"
|
|
" credential and re-plan: the cache keeps every answer that did\n"
|
|
" arrive, so the re-run costs only the lookups that failed.\n\n")
|
|
for reason in sorted(reasons, key=lambda r: -len(reasons[r])):
|
|
rs = reasons[reason]
|
|
fh.write(f"{len(rs):>6} files {reason}\n")
|
|
for r in sorted(rs, key=lambda r: rel(r["path"]))[:400]:
|
|
fh.write(f" {rel(r['path'])}\n")
|
|
if len(rs) > 400:
|
|
fh.write(f" … and {len(rs) - 400} more\n")
|
|
fh.write("\n")
|
|
if no_year:
|
|
fh.write(f"{len(no_year):>6} series carry no year from any provider, so their\n")
|
|
fh.write(" folders are named without one; Jellyfin still matches them.\n")
|
|
fh.write(" " + ", ".join(no_year[:300]) + "\n")
|
|
if no_tvdb:
|
|
fh.write(f"\n{len(no_tvdb):>6} series carry no tvdb identity, so their folders\n")
|
|
fh.write(" keep the name the metadata provider gave them and no episode\n")
|
|
fh.write(" title was taken. Nothing was invented: give these a resolvable\n")
|
|
fh.write(" id (--tvdb) and re-plan; a rename is idempotent.\n")
|
|
fh.write(" " + ", ".join(no_tvdb[:300]) + "\n")
|
|
if clash:
|
|
fh.write(f"\n{len(clash):>6} series carry seasons whose numbering tvdb"
|
|
" contradicts:\n")
|
|
fh.write(" a pack books more episodes in a season than tvdb holds"
|
|
" there — an anime\n pack numbered by its release group is"
|
|
" the usual shape. Those files still\n rename to the"
|
|
" standard shape, but they do NOT ride with tvdb's titles:"
|
|
"\n the slot tvdb answers for that number is a different"
|
|
" episode. Correct the\n numbering and re-plan; a rename"
|
|
" is idempotent.\n")
|
|
fh.write(" " + "; ".join(
|
|
f"{n} S{'/'.join(f'{x:02d}' for x in ss)}"
|
|
for n, ss in clash[:300]) + "\n")
|
|
map_ids = {e["id"] for e in MAP.values()}
|
|
on_array = {rel(p).split("/")[0] for p in held}
|
|
stalled = sorted(f for f in MAP if f not in MAP_SEEN and f in on_array)
|
|
unseen = sorted(f for f in MAP if f not in MAP_SEEN and f not in on_array)
|
|
unanswered = sorted({f"{identity(s)[0]} ({s['tvdb']})"
|
|
for s in canon.values()
|
|
if s.get("tvdb") in map_ids and not s.get("canon")})
|
|
if MAP:
|
|
fh.write(f"\n{len(MAP_SEEN):>6} folders took their tvdb id from the\n")
|
|
fh.write(" operator's map — the human answer to a name tvdb's search\n"
|
|
" cannot state. A wrong row renames a whole pack to the wrong\n"
|
|
" show, so every folder it touched is listed here; fix the map\n"
|
|
" and re-plan, because a rename is idempotent.\n")
|
|
fh.write(" " + "; ".join(
|
|
f"{f} -> {MAP[f]['id']}" for f in sorted(MAP_SEEN)) + "\n")
|
|
if stalled:
|
|
fh.write(f"\n{len(stalled):>6} map rows name a folder on the array that\n")
|
|
fh.write(" never became a series: every file in it parked, so the id\n"
|
|
" had nothing to name. The map names a RECORD — it numbers\n"
|
|
" no episodes and owns no bonus material, so read the park\n"
|
|
" reasons above this line: a folder of openings, endings and\n"
|
|
" promos with no show to claim them is the usual shape.\n")
|
|
fh.write(" " + "; ".join(
|
|
f"{f} -> {MAP[f]['id']}" for f in stalled[:300]) + "\n")
|
|
if unseen:
|
|
fh.write(f"\n{len(unseen):>6} map rows name no folder on the array: the\n")
|
|
fh.write(" pack moved (into the review pile counts: the map reads\n"
|
|
" top-level folders only), or the row carries a name the\n"
|
|
" array never held.\n")
|
|
fh.write(" " + "; ".join(unseen[:300]) + "\n")
|
|
if MAP_CLASH:
|
|
fh.write(f"\n{len(MAP_CLASH):>6} folders carry a provider tvdb id the map\n")
|
|
fh.write(" contradicts. The map won: a box set's provider id is the\n")
|
|
fh.write(" server's guess at a folder name. Read each one:\n")
|
|
fh.write(" " + "; ".join(
|
|
f"{f} provider {p} map {m}"
|
|
for f, p, m in sorted(MAP_CLASH)[:300]) + "\n")
|
|
if unanswered:
|
|
fh.write(f"\n{len(unanswered):>6} map ids tvdb never answered, so those\n")
|
|
fh.write(" folders carry the id and the provider's own name; both\n")
|
|
fh.write(" matchers resolve by the id in the folder tag, so the next\n"
|
|
" re-plan takes tvdb's title. Nothing was invented.\n")
|
|
fh.write(" " + "; ".join(unanswered[:300]) + "\n")
|
|
print(f"exceptions -> {ARGS.exceptions}")
|
|
|
|
|
|
write_plan()
|
|
write_exceptions()
|