Frank
441b908937
The Movies relayout planned from Plex; the TV library has no Plex rows and
58k files whose names lie about which episode they hold. Jellyfin has already
matched and numbered 47,676 of them, so the planner treats that numbering as
the authority and the file name as the fallback: an episode number the name
states, or a provider row, or nothing - and "nothing" parks the file under
_unsorted with a reason instead of inventing a number.
- media-census.py jellyfin: one TSV row per matched episode, its container
path mapped back to the host, series name/year/TMDb id from the Series row
(the episode row carries none of them). Both TSV writers now end lines with
\n so a shell consumer sees the last field clean.
- media-shows.py: the plan. Series folders take the provider's own spelling;
one file claiming two episodes keeps both numbers; two files claiming one
episode - the bigger keeps the standard name, the other parks; bonus
material files under the series' extras folder, never as an episode; a long
file inside the TV library leaves for Movies only when TMDb answers with a
matching title, otherwise it parks. Re-running the plan on a library it
already filed moves nothing.
- tests/media_shows.bats: 15 behaviors, including the executor preflight and
the re-plan stability law.
657 lines
28 KiB
Python
Executable File
657 lines
28 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
r"""Plan a Jellyfin-format relayout of a TV library. DRY RUN ONLY: nothing moves.
|
||
|
||
media-shows.py [--root DIR] [--enum TSV] [--titles TSV] [--probed TSV]
|
||
[--tmdb-cache JSON] [--tmdb] [--movies DIR] [--park DIR]
|
||
[--out TSV] [--exceptions TXT]
|
||
|
||
Inputs are the TSVs scripts/media-census.py exports: `enumerate` lists what
|
||
exists, `jellyfin` says which episode each file IS, and `probe` is optional
|
||
duration evidence for the movie rule. The TMDb cache is a plain JSON file so a
|
||
plan stays reproducible: answers are read from it, and a miss is looked up only
|
||
with --tmdb and TMDB_API_KEY in the environment (the key never enters argv, and
|
||
never the cache).
|
||
|
||
Target contract (jellyfin.org/docs/general/server/media/shows):
|
||
|
||
<root>/<Series> (<year>)/Season <NN>/<Series> SxxEyy.ext
|
||
one file holding two episodes SxxE01-E02 specials Season 00
|
||
a documented part of an episode SxxE03 Part 1 sidecars <base>.<lang>.srt
|
||
bonus material <series>/trailers|featurettes|… — never a season
|
||
|
||
PLAN is the TSV scripts/media-apply.py replays: `action src dst kind`, every
|
||
mkdir row before any move, and `keep` for a file already where the contract puts
|
||
it. EXCEPTIONS is the review list: everything parked, and why.
|
||
|
||
The authority chain, strongest first, for one video file:
|
||
|
||
1. a provider row — Jellyfin matched this exact path and numbered it.
|
||
2. the file's own SxxEyy / 1x04 marker, the series taken from a sibling's
|
||
provider row, else from the nearest ancestor folder that has one, else from
|
||
the release folder's own words.
|
||
3. nothing. An unnumbered file is PARKED under --park with its relative path
|
||
kept: an invented `Season 01` steals an episode number from the file that
|
||
really is S01E01, and a wrong number is worse than no number.
|
||
|
||
Guards, and why each is here:
|
||
|
||
* One series, one folder, one year: the year ANY provider row of a series
|
||
states is shared by all of them, so a box set split across five release
|
||
folders lands inside ONE series folder instead of five.
|
||
* Two files claiming one episode: the bigger keeps the standard name, the other
|
||
parks with the reason. Nothing is overwritten and nothing is deleted — which
|
||
encode survives is Frank's call, and a parked file is one rename from home.
|
||
* A file with no episode evidence that runs long and that the MOVIE provider
|
||
names leaves the TV library for --movies under the movie naming contract.
|
||
Long and unnamed parks; short and unnumbered is bonus material.
|
||
* Series art and the show nfo follow the SERIES, a season image follows its
|
||
SEASON, a subtitle or episode nfo follows the episode's new base — an episode
|
||
moved without its subtitles is a half-move.
|
||
* Hidden files, macOS forks, qBittorrent resume parts and the park pile itself
|
||
are never moved. Skipping the park is what makes re-planning stable.
|
||
|
||
The plan crosses three trees (the library, its park, the movie library), so the
|
||
executor runs with --root at the share that holds them, on its own journal —
|
||
one journal is one undo:
|
||
|
||
python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
|
||
--enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
|
||
--out /mnt/entropy/media-tv-plan.tsv
|
||
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
|
||
--root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
|
||
"""
|
||
import argparse
|
||
import collections
|
||
import importlib.util
|
||
import json
|
||
import os
|
||
import re
|
||
import sys
|
||
import urllib.parse
|
||
import urllib.request
|
||
|
||
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
|
||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
|
||
help="the TV library the plan reorganises")
|
||
ARGS.add_argument("--enum", default="/tmp/tv-enum.tsv",
|
||
help="file census (media-census.py enumerate)")
|
||
ARGS.add_argument("--titles", default="/tmp/jellyfin-tv.tsv",
|
||
help="the provider's numbering (media-census.py jellyfin)")
|
||
ARGS.add_argument("--probed", default="",
|
||
help="ffprobe census for the movie rule; absent = no durations")
|
||
ARGS.add_argument("--tmdb-cache", default="/tmp/tmdb-cache.json",
|
||
help="TMDb answers as an input file, so a plan is reproducible")
|
||
ARGS.add_argument("--tmdb", action="store_true",
|
||
help="look up cache misses (needs TMDB_API_KEY)")
|
||
ARGS.add_argument("--movies", default="/mnt/entropy/Movies",
|
||
help="movie library a mis-filed film moves into")
|
||
ARGS.add_argument("--park", default="",
|
||
help="review pile (default: <root>/_unsorted)")
|
||
ARGS.add_argument("--out", default="/tmp/tv-plan.tsv")
|
||
ARGS.add_argument("--exceptions", default="/tmp/tv-exceptions.txt")
|
||
ARGS = ARGS.parse_args()
|
||
|
||
ROOT = os.path.abspath(ARGS.root)
|
||
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
|
||
MOVIES = os.path.abspath(ARGS.movies)
|
||
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
|
||
_spec = importlib.util.spec_from_file_location("eng", ENGINE)
|
||
eng = importlib.util.module_from_spec(_spec)
|
||
_spec.loader.exec_module(eng)
|
||
|
||
try:
|
||
CACHE = json.load(open(ARGS.tmdb_cache))
|
||
except (OSError, ValueError):
|
||
CACHE = {}
|
||
API_KEY = os.environ.get("TMDB_API_KEY", "")
|
||
|
||
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
|
||
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".rm",
|
||
".rmvb", ".iso", ".img"}
|
||
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
|
||
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
|
||
NFO_EXT = {".nfo", ".xml"}
|
||
# Files that are never anybody's business: hidden, forks, resume state, thumbnails.
|
||
SKIP_NAME = re.compile(r"^(\.|#|dl_state|@eaDir|Thumbs\.db|desktop\.ini)|\.!q$")
|
||
# Show-level metadata keeps its own name at the series root; season-level at the
|
||
# season root; folder.jpg stays with the folder it was written for.
|
||
SERIES_META = re.compile(r"^(tvshow|series|show)(\.nfo|\.xml)?$"
|
||
r"|^(poster|tvposter|banner|clearlogo|clearart|logo|art"
|
||
r"|landscape|fanart|showbackdrop|backdrop)\d*\."
|
||
r"(jpg|jpeg|png|webp|bmp)$", re.I)
|
||
SEASON_META = re.compile(r"^season\.nfo$", re.I)
|
||
FOLDER_IMAGE = re.compile(r"^folder\.(jpg|jpeg|png|webp|bmp)$", re.I)
|
||
SEASON_DIR_RE = re.compile(r"(?i)^\s*(?:season[\s._-]*(\d{1,3})|s(\d{1,2}))\s*$")
|
||
# Release words a folder name carries that are not part of the show's name.
|
||
FOLDER_JUNK = re.compile(r"(?i)^(complete|collection|box|set|batch|pack|series|season"
|
||
r"|multigroup|dvd|remux|uncut|extended|dual|audio|multi|dub"
|
||
r"|dubbed|sub|subbed|internal|repack|proper|webrip)$")
|
||
STOP_WORDS = {"the", "a", "an", "of", "and", "to", "in", "on", "for", "with", "de",
|
||
"la", "le", "il", "des", "der", "das", "y", "e", "el", "les"}
|
||
BAD_CHARS = re.compile(r'[<>:"/|?*]')
|
||
# SxxEyy with an optional second episode, plus the 1x04 form. The second number
|
||
# is guarded so a resolution never reads as an episode: "S01E01-1080p" is ONE
|
||
# episode, and the longest-match backtracking of \d{1,3} is what the (?![\dpP])
|
||
# after it closes off.
|
||
EP_RANGE = re.compile(r"(?i)(?<![a-z0-9])s(?:eason[\s._-]*)?(\d{1,2})[\s._-]*"
|
||
r"e(?:p(?:isode)[\s._-]*)?(\d{1,3})"
|
||
r"(?:[\s._-]*(?:-|to[\s._-]*|ep)[\s._-]*(?:s\d{1,2}[\s._-]*)?e?"
|
||
r"(\d{1,3})(?![\dpP]))?")
|
||
NX_RE = re.compile(r"(?i)(?<![a-z0-9])(\d{1,2})x(\d{2,3})(?![a-z0-9])")
|
||
PART_RE = re.compile(r"(?i)(?:^|[\s._-])(?:part|pt|cd|dvd|disc|disk)[\s._-]?"
|
||
r"([1-9]|i{1,3})(?:$|[\s._-])")
|
||
# Jellyfin reads these subfolder names as bonus material. The planner emits only
|
||
# these spellings, so 40k extras end up in nine folders instead of nine hundred.
|
||
EXTRA_DIRS = [
|
||
("trailers", r"trailer|teaser|promo|advert|commercial|tv[\s._-]spot|preview"),
|
||
("deleted scenes", r"deleted"),
|
||
("behind the scenes", r"behind[\s._-]the[\s._-]scenes|making[\s._-]of|bts"
|
||
r"|b[\s._-]roll|gag[\s._-]reel|blooper|outtake|archival"
|
||
r"|video[\s._-]archive|scene[\s._-]selection|menu|gallery"
|
||
r"|stills?|visual[\s._-]effects|workprint"),
|
||
("interviews", r"interview|cast[\s._-]and[\s._-]crew|audition|commentary"),
|
||
("scene cuts", r"scene[\s._-]cuts?|scenecuts"),
|
||
("featurettes", r"featurettes?"),
|
||
("shorts", r"shorts?"),
|
||
("clips", r"clips?"),
|
||
]
|
||
# Jellyfin's own ExtraType values, folded onto the same folder names.
|
||
EXTRA_KIND = {"trailer": "trailers", "tmbd": "trailers", "intro": "other",
|
||
"movie": "other", "scene": "scene cuts", "clip": "clips",
|
||
"interview": "interviews", "behindthescenes": "behind the scenes",
|
||
"deleted": "deleted scenes", "other": "other", "epilogue": "other",
|
||
"opening": "other", "featurette": "featurettes", "short": "shorts"}
|
||
SHORT_MAX = 45 * 60 # over this, a file with no episode evidence is film-shaped
|
||
|
||
|
||
def rel(p):
|
||
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
|
||
|
||
|
||
def clean(name):
|
||
"""A name the filesystem and both matchers can read."""
|
||
name = BAD_CHARS.sub("-", name).replace("\\", "-").strip(" .")
|
||
return name[:180] or "untitled"
|
||
|
||
|
||
def title_of(text):
|
||
"""The engine's parser, applied to a name that has dots but no extension:
|
||
splitext would read "The.Examples" as stem "The", so a sentinel extension is
|
||
added and the parser sees an ordinary media name."""
|
||
return eng.title_of(text + ".mkv")
|
||
|
||
|
||
def folder_series(name):
|
||
"""The show's own words in a release folder's name: bracketed junk, years,
|
||
season stamps and release vocabulary all fall away."""
|
||
text = re.sub(r"[\[(][^\])]*[\])]", " ", name)
|
||
text = re.sub(r"(?i)\b(19|20)\d\d\b", " ", text)
|
||
out = []
|
||
for w in re.split(r"[.\s_\-\[\](){}+!~,;:&@#/*'\"`´|]+", text):
|
||
c = eng.norm(re.sub(r"[^A-Za-z0-9']", "", w))
|
||
if not c or FOLDER_JUNK.match(c) or eng.JUNK_RE.match(c) or eng.RES_RE.match(c) \
|
||
or eng.folder_is_bonus(c):
|
||
break
|
||
if c.isdigit() and out:
|
||
break
|
||
out.append(w)
|
||
while out and eng.norm(out[-1]) in STOP_WORDS:
|
||
out.pop()
|
||
return clean(" ".join(out)) if out else ""
|
||
|
||
|
||
def markers(name):
|
||
"""(season, [episodes], position) the episode markers a name states; None when
|
||
it states none. The position is where the series' own words stop."""
|
||
stem = os.path.splitext(name)[0]
|
||
m = EP_RANGE.search(stem)
|
||
if m:
|
||
eps = [int(m.group(2))] + ([int(m.group(3))] if m.group(3) else [])
|
||
return int(m.group(1)), eps, m.start()
|
||
m = NX_RE.search(stem)
|
||
if m:
|
||
return int(m.group(1)), [int(m.group(2))], m.start()
|
||
return None
|
||
|
||
|
||
def tmdb(kind, query):
|
||
"""A provider answer for a query: the cache first, TMDb once, then never again."""
|
||
ck = f"{kind}:{query}"
|
||
if ck in CACHE:
|
||
hit = CACHE[ck]
|
||
# A cache an operator edited by hand can carry the year as a number; the
|
||
# comparison below is text, so the wire form is imposed on every hit.
|
||
return None if hit is None else {**hit, "year": str(hit.get("year") or "")}
|
||
if not (ARGS.tmdb and API_KEY):
|
||
return None
|
||
path = "movie" if kind == "movie" else "tv"
|
||
url = ("https://api.themoviedb.org/3/search/" + path + "?query="
|
||
+ urllib.parse.quote(query) + "&include_adult=false")
|
||
req = urllib.request.Request(url, headers={
|
||
# The key rides a header, never the URL: a token in argv is world-readable.
|
||
"Authorization": "Bearer " + API_KEY,
|
||
"Accept": "application/json"})
|
||
try:
|
||
with urllib.request.urlopen(req, timeout=20) as r:
|
||
found = json.load(r).get("results") or []
|
||
except Exception as e: # noqa: BLE001
|
||
print(f"tmdb {kind} lookup failed for {query!r}: {e}", file=sys.stderr)
|
||
return None
|
||
if not found:
|
||
CACHE[ck] = None
|
||
return None
|
||
top = found[0]
|
||
CACHE[ck] = {"name": (top.get("title") or top.get("name") or "").strip(),
|
||
"year": str(top.get("year") or "") or (top.get("first_air_date") or "")[:4],
|
||
"id": top.get("id")}
|
||
return CACHE[ck]
|
||
|
||
|
||
# ------------------------------------------------------------------- the inputs
|
||
rows = []
|
||
for line in open(ARGS.enum):
|
||
size, ext, path = line.rstrip("\n").split("\t")
|
||
p = os.path.abspath(path)
|
||
base = os.path.basename(p)
|
||
if not p.startswith(ROOT + "/") or SKIP_NAME.search(base):
|
||
continue
|
||
if p.startswith(PARK + "/"):
|
||
continue # the park is already reviewed; re-parking is churn
|
||
rows.append({"bytes": int(size), "ext": ext.lower(), "path": p, "base": base})
|
||
held = {r["path"]: r for r in rows}
|
||
|
||
titles = {}
|
||
with open(ARGS.titles) as fh:
|
||
cols = fh.readline().rstrip("\n").split("\t")
|
||
for line in fh:
|
||
f = line.rstrip("\n").split("\t")
|
||
f += [""] * (len(cols) - len(f))
|
||
row = dict(zip(cols, f))
|
||
p = os.path.abspath(row.get("file", ""))
|
||
if p in held:
|
||
titles[p] = row
|
||
|
||
dur = {}
|
||
if ARGS.probed and os.path.exists(ARGS.probed):
|
||
for line in open(ARGS.probed):
|
||
f = line.rstrip("\n").split("\t")
|
||
if len(f) > 2:
|
||
try:
|
||
dur[os.path.abspath(f[1])] = float(f[2] or 0)
|
||
except ValueError:
|
||
pass
|
||
|
||
video = [r for r in rows if r["ext"] in VIDEO_EXT]
|
||
other = [r for r in rows if r["ext"] not in VIDEO_EXT]
|
||
|
||
# What a folder already knows: one provider row names the series for every sibling
|
||
# that has no row of its own — that is how an anime pack gets its show.
|
||
by_dir = collections.defaultdict(list)
|
||
for r in video:
|
||
by_dir[os.path.dirname(r["path"])].append(r)
|
||
dir_series = {}
|
||
for d, rs in by_dir.items():
|
||
names = collections.Counter(titles[r["path"]]["series"] for r in rs
|
||
if r["path"] in titles and titles[r["path"]]["series"])
|
||
if names:
|
||
dir_series[d] = names.most_common(1)[0][0]
|
||
|
||
|
||
# A series folder that sits above a folder of provider rows names its own show, and
|
||
# that is what keeps an already-filed library stable when it is re-planned: extras in
|
||
# "The Examples (2010)/trailers" belong to "The Examples", not to a series invented
|
||
# out of the word "trailers".
|
||
for d, name in sorted(dir_series.items()):
|
||
a = os.path.dirname(d)
|
||
while a and a != ROOT and len(a) > len(ROOT):
|
||
if eng.norm(folder_series(os.path.basename(a))) == eng.norm(name):
|
||
dir_series.setdefault(a, name)
|
||
a = os.path.dirname(a)
|
||
|
||
|
||
def series_of_dir(d):
|
||
"""The series an ancestor folder belongs to, nearest first."""
|
||
while d and d != ROOT and len(d) > len(ROOT):
|
||
if d in dir_series:
|
||
return dir_series[d]
|
||
d = os.path.dirname(d)
|
||
return ""
|
||
|
||
|
||
def head_series(path):
|
||
"""The folder evidence for a file's series: its own folder's words, then the
|
||
top-level release folder's."""
|
||
parts = rel(path).split("/")[:-1]
|
||
for d in reversed(parts):
|
||
name = folder_series(d)
|
||
if name:
|
||
return name
|
||
return ""
|
||
|
||
|
||
def extras_kind(path, extra_type):
|
||
"""The canonical bonus subfolder, from Jellyfin's own extra type first."""
|
||
if extra_type:
|
||
return EXTRA_KIND.get(eng.norm(extra_type).replace(" ", ""), "other")
|
||
for d in rel(path).split("/")[:-1]:
|
||
if eng.folder_is_bonus(d):
|
||
for name, rx in EXTRA_DIRS:
|
||
if re.search(rx, d, re.I):
|
||
return name
|
||
return "extras"
|
||
stem = os.path.splitext(os.path.basename(path))[0]
|
||
for name, rx in EXTRA_DIRS:
|
||
if re.search(rx, stem, re.I):
|
||
return name
|
||
return "extras"
|
||
|
||
|
||
# ----------------------------------------------------------- classify every file
|
||
series = {} # norm(name) -> {name, year, tmdb}
|
||
claimants = [] # (row, name, season, [episodes])
|
||
extras = [] # (row, name, bonus folder)
|
||
films = [] # film-shaped, no episode evidence
|
||
unknown = [] # (row, reason)
|
||
|
||
|
||
def note(name, year="", tmdb_id=""):
|
||
"""Register a series; the first name wins, the first year and id stick."""
|
||
k = eng.norm(name)
|
||
s = series.setdefault(k, {"name": clean(name), "year": "", "tmdb": ""})
|
||
if year and not s["year"]:
|
||
s["year"] = str(year)
|
||
if tmdb_id and not s["tmdb"]:
|
||
s["tmdb"] = str(tmdb_id)
|
||
|
||
|
||
|
||
def as_extra(r, owner, extra_type):
|
||
"""Bonus material needs a series to file under; with none it is parked."""
|
||
if not owner:
|
||
unknown.append((r, "bonus material with no series to own it"))
|
||
return
|
||
note(owner)
|
||
extras.append((r, owner, extras_kind(r["path"], extra_type)))
|
||
|
||
for r in video:
|
||
p, t = r["path"], titles.get(r["path"])
|
||
marked = markers(r["base"])
|
||
if t and t["extra"]:
|
||
as_extra(r, t["series"] or series_of_dir(os.path.dirname(p)) or head_series(p),
|
||
t["extra"])
|
||
continue
|
||
if t and t["series"] and t["season"] and t["episode"]:
|
||
season, eps = int(t["season"]), [int(t["episode"])]
|
||
# One file can hold two episodes; the row gives the first, the name the range.
|
||
if marked and marked[0] == season and len(marked[1]) > 1:
|
||
eps = marked[1]
|
||
claimants.append((r, t["series"], season, eps))
|
||
note(t["series"], t["year"], t["tmdb"])
|
||
continue
|
||
if marked:
|
||
owner = (series_of_dir(os.path.dirname(p)) or head_series(p)
|
||
or title_of(r["base"][:marked[2]])[0])
|
||
if not eng.norm(owner).replace(" ", ""):
|
||
unknown.append((r, "an episode marker with no series name to hang it on"))
|
||
continue
|
||
claimants.append((r, owner, marked[0], marked[1]))
|
||
note(owner)
|
||
continue
|
||
if eng.is_extra(p, title_of(r["base"])[0], dur.get(p, 0.0), ROOT):
|
||
as_extra(r, series_of_dir(os.path.dirname(p)) or head_series(p), "")
|
||
continue
|
||
if dur.get(p, 0.0) >= SHORT_MAX or (p not in dur and title_of(r["base"])[1] != "?"):
|
||
films.append(r)
|
||
continue
|
||
unknown.append((r, "no episode marker, no provider row, nothing long enough to be a film"))
|
||
|
||
# A provider that never states a year still has one: the search is exact, so a
|
||
# miss parks nothing — the folder simply carries no year, which Jellyfin accepts.
|
||
for k, s in series.items():
|
||
if not s["year"]:
|
||
got = tmdb("tv", s["name"])
|
||
if got and eng.norm(got["name"]) == k:
|
||
s["year"] = got["year"]
|
||
|
||
if ARGS.tmdb:
|
||
with open(ARGS.tmdb_cache, "w") as fh:
|
||
json.dump(CACHE, fh, indent=1, sort_keys=True)
|
||
|
||
|
||
def series_dir(name):
|
||
s = series.get(eng.norm(name))
|
||
if not s:
|
||
s = {"name": clean(name), "year": ""}
|
||
label = f"{s['name']} ({s['year']})" if s["year"] else s["name"]
|
||
return f"{ROOT}/{clean(label)}"
|
||
|
||
|
||
# ------------------------------------------------------------------- the plan
|
||
moves, keeps, parked = [], [], []
|
||
reasons = collections.defaultdict(list)
|
||
used = {} # dst -> the file that claimed it
|
||
|
||
|
||
def book(r, dst_dir, name, kind):
|
||
"""Claim one destination for one file, or park the file when it is taken."""
|
||
dst = f"{dst_dir}/{name}"
|
||
if dst == r["path"]:
|
||
keeps.append((r["path"], dst, kind))
|
||
return
|
||
if dst in used:
|
||
park(r, "something else in the plan already files there")
|
||
return
|
||
if dst in held:
|
||
park(r, f"{rel(dst)} already occupies that name")
|
||
return
|
||
used[dst] = r["path"]
|
||
moves.append((r["path"], dst, kind))
|
||
|
||
|
||
def park(r, reason):
|
||
parked.append((r, reason))
|
||
reasons[reason].append(r)
|
||
dst = f"{PARK}/{rel(r['path'])}"
|
||
if dst in used or dst in held:
|
||
dst = f"{PARK}/{rel(r['path'])}.{len(reasons)}"
|
||
used[dst] = r["path"]
|
||
moves.append((r["path"], dst, "park"))
|
||
|
||
|
||
# Episodes first: two files claiming one number is a decision Frank makes, not one
|
||
# the plan sneaks — the bigger copy takes the standard name, the other waits.
|
||
by_ep = collections.defaultdict(list)
|
||
for r, name, season, eps in claimants:
|
||
for e in eps:
|
||
by_ep[(eng.norm(name), season, e)].append(r)
|
||
winners, losers = set(), {}
|
||
for (key, season, e), rs in by_ep.items():
|
||
rs.sort(key=lambda r: (-r["bytes"], r["path"]))
|
||
winners.add(rs[0]["path"])
|
||
for r in rs[1:]:
|
||
losers.setdefault(r["path"],
|
||
f"duplicate of {series[key]['name']} S{season:02d}E{e:02d}"
|
||
" (the bigger copy keeps the standard name)")
|
||
for r, name, season, eps in claimants:
|
||
if r["path"] in losers:
|
||
park(r, losers[r["path"]])
|
||
continue
|
||
s = series[eng.norm(name)]
|
||
label = f" S{season:02d}E" + "-E".join(f"{e:02d}" for e in eps)
|
||
part = PART_RE.search(os.path.splitext(r["base"])[0])
|
||
tail = f" Part {part.group(1).upper()}" if part and len(eps) == 1 else ""
|
||
book(r, f"{series_dir(name)}/Season {season:02d}", s["name"] + label + tail + r["ext"],
|
||
"tv")
|
||
|
||
for r, name, kind in extras:
|
||
book(r, f"{series_dir(name)}/{kind}",
|
||
clean(os.path.splitext(r["base"])[0]) + r["ext"], "extra")
|
||
|
||
# A film inside the TV library leaves for the movie library only when the movie
|
||
# provider answers with a title that matches the name and, when the name states a
|
||
# year, that year too.
|
||
for r in films:
|
||
parsed, year = title_of(r["base"])
|
||
got = tmdb("movie", parsed)
|
||
words = eng.norm(got["name"]).split() if got else []
|
||
if not got or not eng.mentions(parsed.split(), words) \
|
||
or (year != "?" and got["year"] and got["year"] != year):
|
||
park(r, "long file with no episode evidence, and no movie provider names it")
|
||
continue
|
||
label = f"{got['name']} ({got['year']})" if got["year"] else got["name"]
|
||
book(r, f"{MOVIES}/{clean(label)}", clean(label) + r["ext"], "movie")
|
||
|
||
# ------------------------------------------------- sidecars and folder metadata
|
||
moved = {} # src -> dst, for every booked file
|
||
for src, dst, _kind in moves:
|
||
moved[src] = dst
|
||
for src, dst, _kind in keeps:
|
||
moved[src] = dst
|
||
dst_dirs = collections.defaultdict(collections.Counter)
|
||
video_dst = collections.defaultdict(list)
|
||
for src, dst in moved.items():
|
||
if src in {r["path"] for r in video}:
|
||
video_dst[os.path.dirname(src)].append((src, dst))
|
||
dst_dirs[os.path.dirname(src)][os.path.dirname(dst)] += 1
|
||
|
||
|
||
def series_name_at(d):
|
||
"""The series the files of a folder belong to, else the folder's own words."""
|
||
names = [titles[src]["series"] for src, _ in video_dst.get(d, [])
|
||
if src in titles and titles[src]["series"]]
|
||
if names:
|
||
return collections.Counter(names).most_common(1)[0][0]
|
||
return series_of_dir(d) or head_series(d + "/x")
|
||
|
||
|
||
def season_at(d):
|
||
"""The season number a folder carries: its own name, else its files' rows."""
|
||
m = SEASON_DIR_RE.match(os.path.basename(d))
|
||
if m:
|
||
return int(m.group(1) or m.group(2))
|
||
nums = [int(titles[src]["season"]) for src, _ in video_dst.get(d, [])
|
||
if src in titles and titles[src]["season"]]
|
||
return collections.Counter(nums).most_common(1)[0][0] if nums else None
|
||
|
||
|
||
def folder_dst(d):
|
||
"""Where a folder's own images go: the one place its files landed, or the
|
||
series root when they scattered across several seasons."""
|
||
c = dst_dirs.get(d)
|
||
if not c:
|
||
return None
|
||
if len(c) == 1:
|
||
return c.most_common(1)[0][0]
|
||
owner = series_name_at(d)
|
||
return series_dir(owner) if owner else None
|
||
|
||
|
||
def anchor_of(path, stem):
|
||
"""The video file this sidecar names: the longest file name it extends, in its
|
||
own folder first, then the parent's (a Subs/ or Extras/ folder beside them)."""
|
||
for d in (os.path.dirname(path), os.path.dirname(os.path.dirname(path))):
|
||
best = None
|
||
for src, dst in video_dst.get(d, []):
|
||
base = os.path.basename(src)
|
||
for prefix in (os.path.splitext(base)[0], base):
|
||
if stem == prefix or stem.startswith(prefix + "."):
|
||
if best is None or len(prefix) > best[1]:
|
||
best = (src, dst, len(prefix))
|
||
if best:
|
||
return best[0], best[1]
|
||
return None
|
||
|
||
|
||
def sidecar_name(r, src, dst):
|
||
"""The sidecar's own tail — language, flag, the media extension an nfo must
|
||
carry — hung on the video file's new base."""
|
||
stem = os.path.splitext(os.path.basename(src))[0]
|
||
tail = os.path.splitext(r["base"])[0][len(stem):]
|
||
if r["ext"] in NFO_EXT:
|
||
tail = os.path.splitext(os.path.basename(src))[1] + tail
|
||
return clean(os.path.splitext(os.path.basename(dst))[0] + tail + r["ext"])
|
||
|
||
|
||
for r in other:
|
||
p, base = r["path"], r["base"]
|
||
stem = os.path.splitext(base)[0]
|
||
d = os.path.dirname(p)
|
||
if r["ext"] in SUB_EXT or r["ext"] in NFO_EXT or r["ext"] in IMG_EXT:
|
||
a = anchor_of(p, stem)
|
||
if a:
|
||
book(r, os.path.dirname(a[1]), sidecar_name(r, a[0], a[1]),
|
||
"subtitle" if r["ext"] in SUB_EXT else
|
||
("nfo" if r["ext"] in NFO_EXT else "image"))
|
||
continue
|
||
if SERIES_META.match(base):
|
||
owner = series_name_at(d)
|
||
if owner:
|
||
book(r, series_dir(owner), base, "meta")
|
||
continue
|
||
elif SEASON_META.match(base):
|
||
owner, season = series_name_at(d), season_at(d)
|
||
if owner and season is not None:
|
||
book(r, f"{series_dir(owner)}/Season {season:02d}", base, "meta")
|
||
continue
|
||
elif r["ext"] in IMG_EXT or r["ext"] in NFO_EXT:
|
||
dest = folder_dst(d)
|
||
if dest:
|
||
book(r, dest, base, "meta")
|
||
continue
|
||
unknown.append((r, "not media, or media with nothing beside it to belong to"))
|
||
|
||
for r, reason in unknown:
|
||
park(r, reason)
|
||
|
||
|
||
# --------------------------------------------------------------------- output
|
||
def write_plan():
|
||
"""mkdir rows for every destination folder and its whole chain, then the moves.
|
||
Parents first: the executor creates directories in the order they are listed."""
|
||
chain = set()
|
||
for _s, dst, _k in moves:
|
||
d = os.path.dirname(dst)
|
||
while d and d not in (ROOT, MOVIES, PARK, os.path.dirname(d)):
|
||
chain.add(d)
|
||
d = os.path.dirname(d)
|
||
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
|
||
with open(ARGS.out, "w") as fh:
|
||
fh.write("action\tsrc\tdst\tkind\n")
|
||
for d in dirs:
|
||
fh.write(f"mkdir\t\t{d}\tdir\n")
|
||
for src, dst, kind in sorted(moves, key=lambda m: (m[1], m[0])):
|
||
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
|
||
for src, dst, kind in sorted(keeps, key=lambda m: (m[1], m[0])):
|
||
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
|
||
sized = sum(held[s]["bytes"] for s, _d, _k in moves if s in held)
|
||
print(f"plan: {len(moves) - len(parked)} moves, {len(parked)} parked"
|
||
f" ({sized / 2**40:.2f} TiB touched), {len(keeps)} already placed,"
|
||
f" {len(dirs)} folders -> {ARGS.out}")
|
||
|
||
|
||
def write_exceptions():
|
||
no_year = sorted(s["name"] for s in series.values() if not s["year"])
|
||
with open(ARGS.exceptions, "w") as fh:
|
||
fh.write(f"TV relayout exceptions for {ROOT}\n\n")
|
||
fh.write(f"Everything below is parked under {PARK}, keeping the relative path it\n")
|
||
fh.write("has today. Nothing is deleted and nothing is overwritten: a parked file\n")
|
||
fh.write("is one rename from wherever the plan is pointed next.\n\n")
|
||
for reason in sorted(reasons, key=lambda r: -len(reasons[r])):
|
||
rs = reasons[reason]
|
||
fh.write(f"{len(rs):>6} files {reason}\n")
|
||
for r in sorted(rs, key=lambda r: rel(r["path"]))[:400]:
|
||
fh.write(f" {rel(r['path'])}\n")
|
||
if len(rs) > 400:
|
||
fh.write(f" … and {len(rs) - 400} more\n")
|
||
fh.write("\n")
|
||
if no_year:
|
||
fh.write(f"{len(no_year):>6} series carry no year from any provider, so their\n")
|
||
fh.write(" folders are named without one; Jellyfin still matches them.\n")
|
||
fh.write(" " + ", ".join(no_year[:300]) + "\n")
|
||
print(f"exceptions -> {ARGS.exceptions}")
|
||
|
||
|
||
write_plan()
|
||
write_exceptions()
|