feat(media): tvdb names the show — episode titles, id-backed folders, park outside
The TV relayout that landed copied Jellyfin's Series names verbatim, and the
naming census showed what that launders in: one Series item per release folder
(1,107 items for 1,107 top-level dirs), the same show spelled six ways, and
box sets living in 346 separate folders for 821 identities. Jellyfin names a
Series after the folder it scanned, so `24 (2001) Season 1` is a folder echo,
not a show name.
The name authority is now tvdb, reached through the id the server matched:
* media-census.py jellyfin emits two more columns — `title` (the episode row's
own Name) and `tvdb` (the series id, read from BaseItemProviders where
Jellyfin 12 keeps provider ids and from the row's Data blob where older
builds do, so the census does not care which shape the live DB has);
* media-shows.py resolves each series to tvdb — the row's tvdb id, else its
TMDb id through TMDb's external ids, else an exact tvdb name search — and
writes tvdb's name and year. That also collapses the provider's spellings
into ONE identity, which is what turns a box set split over five release
folders into one folder and what makes two files claiming one episode MEET
in the duplicate rule;
* the episode file becomes `<Series> (<year>) - SxxEyy - <Title>`: the title
comes from tvdb's episode list, else the row's own title, else nowhere — no
title means no separator, never an invented one.
Two refusals come with it. A row whose series is the hidden top-level folder the
file itself sits in is NOT evidence: the review pile parked inside the library
was indexed as one show named `_unsorted`, and its rows would have renamed real
shows to it. And inside such a folder a file's own words may MATCH a registered
show but never INVENT one — invention is what filled the pile. `--park`
therefore belongs outside the library, which is also what makes the next plan
re-file the pile instead of skipping it.
Both providers answer offline-reproducibly: every answer is cached in one JSON
file, so a plan can be re-planned without a key. Lookups stay behind --tmdb and
--tvdb, and the tvdb token is read from a mode-600 file, never argv.
tests/media_shows.bats is 23 tests, none of which touch the network — the
fixture Jellyfin DB carries the providers table and the name cache is an input
file. ./tests/run.sh: 753 green.
This commit is contained in:
Frank committed
1 parent
5dbbfcb8a0
commit
c3ab565d9b
4 files changed
+463
-96
No files matched your search
+102
-25
@@ -8,7 +8,14 @@
|
||||
# file name. So every test here states which authority a file had — name only,
|
||||
# provider only, both, neither — and asserts where the PLAN files it, never how
|
||||
# the planner decided. A fixture Jellyfin database stands in for the live one and
|
||||
# the TMDb cache is an input file, so no test touches the network.
|
||||
# the name caches are input files, so no test touches the network.
|
||||
#
|
||||
# The target contract is tvdb-grade naming:
|
||||
# <Series> (<year>)/Season NN/<Series> (<year>) - SxxEyy - <Episode Title>.<ext>
|
||||
# tvdb is the IDENTITY authority: a provider name a server derived from a folder
|
||||
# is not a series name, so a polluted name is undone through the series' tvdb id.
|
||||
# The review pile parks OUTSIDE the library — a parked folder inside it indexes
|
||||
# as one junk show, and that show's rows are then mistaken for evidence.
|
||||
#
|
||||
# The two outcomes that cost a re-run of a 50k-file plan are a file filed as an
|
||||
# episode that is not one (it steals an episode number from its real owner) and a
|
||||
@@ -25,11 +32,11 @@ APPLY="$REPO_ROOT/scripts/media-apply.py"
|
||||
# at PATH SIZE — a file with deterministic content.
|
||||
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
|
||||
|
||||
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] — one row of the authority
|
||||
# export scripts/media-census.py jellyfin writes.
|
||||
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] [TMDB] [TITLE] [TVDB]
|
||||
# — one row of the authority export scripts/media-census.py jellyfin writes.
|
||||
title_row() {
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t\n' "$1" "$2" "$3" "${4:-}" "${5:-}" "${6:-}" \
|
||||
>> "$TITLES"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" \
|
||||
"${5:-}" "${6:-}" "${7:-}" "${8:-}" "${9:-}" >> "$TITLES"
|
||||
}
|
||||
|
||||
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
|
||||
@@ -40,6 +47,7 @@ dst() {
|
||||
setup() {
|
||||
LIB="$BATS_TEST_TMPDIR/TVShows"
|
||||
PARK="$LIB/_unsorted"
|
||||
OUTSIDE="$BATS_TEST_TMPDIR/park"
|
||||
MOVIES="$BATS_TEST_TMPDIR/Movies"
|
||||
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
|
||||
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
|
||||
@@ -77,8 +85,14 @@ setup() {
|
||||
# A season folder whose rows are stale: the database numbers every file of this
|
||||
# pack season 16 episode 20, four seasons' worth of them.
|
||||
at "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" 6000
|
||||
# One episode-shaped file loose at the library root.
|
||||
at "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" 4000
|
||||
# A box-set folder whose Series item the server named AFTER THE FOLDER: the
|
||||
# row's own name is junk, and only a provider id can undo it.
|
||||
BOX="$LIB/Show Box (2016) Season 1 S01 (1080p WEB-DL)"
|
||||
at "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" 6000
|
||||
# The park inside the library: the server indexed the whole pile as ONE show
|
||||
# named after the folder, so its rows state a series that is not a show.
|
||||
at "$PARK/Park Show S01E01.mkv" 5000
|
||||
|
||||
# The enum census lists every file; the planner reads that instead of walking
|
||||
# the tree, so the fixture writes the same three columns.
|
||||
@@ -92,17 +106,21 @@ with open(out, "w") as fh:
|
||||
fh.write(f"{os.stat(p).st_size}\t{os.path.splitext(f)[1]}\t{p}\n")
|
||||
PY
|
||||
|
||||
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\n' > "$TITLES"
|
||||
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\ttitle\ttvdb\n' > "$TITLES"
|
||||
printf '{}\n' > "$CACHE"
|
||||
title_row "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" \
|
||||
"The Examples" 2010 1 1
|
||||
"The Examples" 2010 1 1 "" "" "Pilot"
|
||||
title_row "$ANIME/[grp] Anime Pack - 05.mkv" "Anime Pack" "" 1 5
|
||||
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1
|
||||
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1
|
||||
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2
|
||||
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3
|
||||
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
|
||||
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
|
||||
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2 "" "" "Double Trouble"
|
||||
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3 "" "" "Loose Ends"
|
||||
title_row "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" \
|
||||
"SNL Pack" "" 16 20
|
||||
title_row "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \
|
||||
"Show Box (2016) Season 1" 2016 1 1 "" "2334" "Pilot" "78850"
|
||||
# The row the server wrote for a parked file: the series is the PARK FOLDER.
|
||||
title_row "$PARK/Park Show S01E01.mkv" "_unsorted" "" 1 1
|
||||
}
|
||||
|
||||
plan() {
|
||||
@@ -111,6 +129,12 @@ plan() {
|
||||
--out "$OUT" --exceptions "$EXC"
|
||||
}
|
||||
|
||||
plan_park() { # the review pile outside the library
|
||||
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
|
||||
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
|
||||
--park "$OUTSIDE" --out "$OUT" --exceptions "$EXC"
|
||||
}
|
||||
|
||||
@test "the census exports the provider's episode numbering as a TSV" {
|
||||
DB="$BATS_TEST_TMPDIR/jellyfin.db"
|
||||
python3 - "$DB" "$SEASON" <<'PY'
|
||||
@@ -125,7 +149,7 @@ sid = str(uuid.uuid4())
|
||||
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
|
||||
pre, None, None, 2010, None, 0,
|
||||
'{"ProviderIds":{"Tmdb":"1234"}}'))
|
||||
'{"ProviderIds":{"Tmdb":"1234","Tvdb":"78850"}}'))
|
||||
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(str(uuid.uuid4()), sid, "MediaBrowser.Controller.Entities.TV.Episode",
|
||||
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
|
||||
@@ -136,34 +160,87 @@ PY
|
||||
--out "$BATS_TEST_TMPDIR/exp.tsv"
|
||||
[ "$status" -eq 0 ]
|
||||
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
|
||||
[ "${lines[0]}" = "file series year season episode extra tmdb" ]
|
||||
[ "${lines[0]}" = "file series year season episode extra tmdb title tvdb" ]
|
||||
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
|
||||
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234$"
|
||||
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234 Pilot 78850$"
|
||||
}
|
||||
|
||||
@test "an episode its own name states files as Show SxxEyy under Season NN" {
|
||||
@test "the census reads provider ids from the table newer Jellyfin keeps" {
|
||||
DB="$BATS_TEST_TMPDIR/jellyfin12.db"
|
||||
python3 - "$DB" "$SEASON" <<'PY'
|
||||
import os, sqlite3, sys, uuid
|
||||
db, season = sys.argv[1], sys.argv[2]
|
||||
c = sqlite3.connect(db)
|
||||
pre = "/data/TVShows/" + os.path.basename(season)
|
||||
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
|
||||
ParentIndexNumber, ProductionYear, ExtraType,
|
||||
IsVirtualItem, Data)""")
|
||||
# Jellyfin 12 moved the ids out of the Data blob and into their own table.
|
||||
c.execute("create table BaseItemProviders (ItemId, ProviderId, ProviderValue)")
|
||||
sid = str(uuid.uuid4())
|
||||
eid = str(uuid.uuid4())
|
||||
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
|
||||
pre, None, None, 2010, None, 0, '{}'))
|
||||
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(eid, sid, "MediaBrowser.Controller.Entities.TV.Episode",
|
||||
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
|
||||
None, None, 0, '{}'))
|
||||
c.execute("insert into BaseItemProviders values (?,?,?)", (sid, "Tvdb", "9999"))
|
||||
c.commit()
|
||||
PY
|
||||
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
|
||||
--out "$BATS_TEST_TMPDIR/exp12.tsv"
|
||||
[ "$status" -eq 0 ]
|
||||
tail -1 "$BATS_TEST_TMPDIR/exp12.tsv" \
|
||||
| grep -q " Pilot 9999$"
|
||||
}
|
||||
|
||||
@test "an episode its own name states files as the titled standard name" {
|
||||
plan
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv")" \
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.mkv" ]
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.mkv" ]
|
||||
}
|
||||
|
||||
@test "the tvdb identity names the series folder, not the name a folder left behind" {
|
||||
printf '{"tvdb:series:78850": {"name": "Show Box", "year": "2016"}}\n' > "$CACHE"
|
||||
plan
|
||||
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" \
|
||||
= "$LIB/Show Box (2016)/Season 01/Show Box (2016) - S01E01 - Pilot.mkv" ]
|
||||
}
|
||||
|
||||
@test "a series no authority resolves keeps the provider's name and is reported" {
|
||||
plan
|
||||
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
|
||||
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
|
||||
grep -A9 -i "no tvdb" "$EXC" | grep -q "SNL Pack"
|
||||
}
|
||||
|
||||
@test "a row whose series is the park folder is not evidence of a show" {
|
||||
plan_park
|
||||
run awk -F '\t' -v p="$PARK/" '$1 == "move" && index($3, p) == 1 {print $3}' "$OUT"
|
||||
[ -z "$output" ]
|
||||
run awk -F '\t' '$1 == "move" && $2 ~ /Park Show S01E01/ {print $3}' "$OUT"
|
||||
[[ "$output" == "$OUTSIDE/"* ]]
|
||||
}
|
||||
|
||||
@test "a subtitle follows its episode and keeps its language flag" {
|
||||
plan
|
||||
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt")" \
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.en.srt" ]
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.en.srt" ]
|
||||
}
|
||||
|
||||
@test "a subtitle two folders down inside Subs/<release-name>/ still follows its episode" {
|
||||
plan
|
||||
[ "$(dst "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt")" \
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.jpn.srt" ]
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.jpn.srt" ]
|
||||
}
|
||||
|
||||
@test "a subtitle that carries the container's word in its own name still follows the episode" {
|
||||
plan
|
||||
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt")" \
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples S01E01.forced.srt" ]
|
||||
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.forced.srt" ]
|
||||
}
|
||||
|
||||
@test "a season image keeps its own name; a series nfo keeps the series folder" {
|
||||
@@ -178,10 +255,10 @@ PY
|
||||
[ -z "$(awk -F '\t' '$3 ~ /Season .*teaser/ {print $3}' "$OUT")" ]
|
||||
}
|
||||
|
||||
@test "a file only the provider numbers files at the episode it is matched to" {
|
||||
@test "a file only the provider numbers files at the episode it is matched to, title or no title" {
|
||||
plan
|
||||
[ "$(dst "$ANIME/[grp] Anime Pack - 05.mkv")" \
|
||||
= "$LIB/Anime Pack/Season 01/Anime Pack S01E05.mkv" ]
|
||||
= "$LIB/Anime Pack/Season 01/Anime Pack - S01E05.mkv" ]
|
||||
}
|
||||
|
||||
@test "a bonus folder inside a release pack belongs to the pack's show, not to its own name" {
|
||||
@@ -193,19 +270,19 @@ PY
|
||||
@test "a Season folder outvotes the stale row its files carry" {
|
||||
plan
|
||||
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
|
||||
= "$LIB/SNL Pack/Season 19/SNL Pack S19E01.avi" ]
|
||||
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
|
||||
}
|
||||
|
||||
|
||||
@test "one file holding two episodes keeps both numbers in its name" {
|
||||
plan
|
||||
[ "$(dst "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv")" \
|
||||
= "$LIB/Show Two (2016)/Season 01/Show Two S01E02-E03.mkv" ]
|
||||
= "$LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E02-E03 - Double Trouble.mkv" ]
|
||||
}
|
||||
|
||||
@test "two files claiming one episode: one keeps the name, the other parks" {
|
||||
plan
|
||||
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two S01E01.mkv " "$OUT")
|
||||
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E01 - Two Pilots.mkv " "$OUT")
|
||||
[ "$n" -eq 1 ]
|
||||
run dst "$LIB/Dupes/Show Two S01E01.mkv"
|
||||
[[ "$output" == "$PARK/"* ]]
|
||||
|
||||
Reference in new issue
Block a user