Tvdb's default episode query answers at most 500 rows and states no total.
One Piece (1999), Crayon Shin-chan (1992) and Detective Conan (1996) each
answer exactly 500 while holding over a thousand, so the contradiction rule
added in the previous commit read every long-running show as contradicted:
the pack books more episodes than "the authority holds", and the rule that
exists to stop a wrong anime title would have stripped the right one from
every file of the season.
A capped list now records that it is capped (`tvdb:capped:<id>`, so a warm
cache answers the same question a cold one did) and contradicts nothing;
titles for the episodes the page does hold still ride.
Test: a 500-row page with a 501-episode pack keeps ` - Episode 1` and lists
no contradiction. Red before the change, proved in a copy of the tree with
the cap raised past the fixture.
Three defects in the TVShows planner, all three visible in the applied plan:
- A series folder named a show by words alone. Jellyfin's Tvdb agent and
Plex's scanner both read `[tvdbid-NNNN]`, so the resolved id now rides the
series folder — and only the folder: a file name gets 180 characters and
the episode title needs the room.
- 1,412 files parked because their provider row said `season 0`. Season 0 is
a legal season; Jellyfin's contract is `Season 00` for specials.
- 1,033 files parked because Jellyfin's date matcher writes the AIRDATE YEAR
into the season column. Such a row is now renumbered by the title tvdb
states for the series' id, read backwards off its own episode list. A
title tvdb gives to two episodes names no episode, and that file parks
with the reason it always carried.
A season tvdb's own episode list CONTRADICTS — a release pack books more
episodes there than the authority holds, the anime numbering hazard — keeps
its rename and loses the tvdb title, because the slot its number points at
is a different episode. Those series are listed in the exceptions file,
which is where the operator reads them.
Tests: six through the fixture (five red before the change; the ambiguous
title is a guard that the recovery must never guess). Full bats suite green.
"No row at the destination" was one number, and it conflated three different
things: a sidecar the server never indexes, a file a later pass renamed again,
and a file that never left its source — the last one being actual damage. The
runbook already named the four classes; the tool never computed them, so an
operator reading `missing from the DB: 1851` had no way to tell a booklet.pdf
from a lost album and might have run a rollback over a false alarm.
`missing` rows are now stat'd on both paths and reported as
`on disk at dst, no row names it` / `REAL — file still at src` /
`gone — at neither path`, with offenders listed in that order. A row still
naming `src` stays a separate line: that is the server's own stale pointer, a
phantom, and it costs a re-scan rather than a restore.
Two new tests (REAL, and gone-vs-sidecar) plus two existing layout assertions
loosened from line positions to content. 14/14 in tests/media_verify.bats.
The two Plex arms of media-census.py — `plex` (the movie planner's title
authority) and `audiobooks` (the chapter rows the audiobook plan files by) —
opened the library `mode=ro&immutable=1` while the Jellyfin arms already used
plain `mode=ro`. Same defect as media-verify.py: an uncheckpointed row of a
running scan is invisible, so a just-scanned item is simply absent from the
authority file a plan is built from, and the pass cannot say why.
Both now go through open_plex_ro(), which tries plain ro first and keeps
immutable only as the fallback for a database whose -shm cannot be read. The
new audiobooks test leaves a chapter row committed-but-uncheckpointed and
asserts the export still carries it; red before the change, green after.
A verify run beside a running scan reported rows the scanner had just
written as MISSING, and could fail outright as "database disk image is
malformed". Both libraries live in WAL mode, and the verifier opened them
with immutable=1, which tells sqlite the file never changes: it skips the
write-ahead log, so every committed-but-uncheckpointed row is invisible.
134 MB of jellyfin.db-wal was riding past the verifier's eyes during the
2026-10-31 rescan.
Open plain mode=ro first — sqlite then reads the WAL through its -shm —
and keep immutable=1 only as the fallback for a database whose -shm
cannot be read. The bats fixture leaves a journaled move uncheckpointed
and proves the verifier still counts it landed.
The first tvdb-grade plan handed 12 files to the movie library, and two of them
were wrong in a way the exceptions file could not show: an episode of the
documentary series `Russia 1985-1999: TraumaZone` was renamed to the 1963 Bond
film `From Russia with Love`, and `Gundam 00 Special Edition III - Return The
World` became `Special Edition I - Celestial Being`.
Two guards, each alone insufficient:
- `title_of` salvages a short query from a long release name ("russia", "mobile
suit gundam 00"), and tmdb() took the FIRST search result. The call site then
compared the provider's words against that salvaged query, so any title merely
CONTAINING the query passed. The provider's title must now be a run of words the
FILE itself states: `Special Edition I: Celestial Being` is not in a filename
that says `Special Edition III - Return The World`.
- tmdb() read the year from `year`/`first_air_date`, which the MOVIE search does
not return — it returns `release_date`. Every film answer therefore arrived
year-less and the year guard at the call site could never fire. That is how a
1963 answer survived a filename stating 1991.
Tests: a provider answer the file does not name parks, and nothing lands in the
movie library; an answer from another year is refused. The year-from-the-wire half
is proven by the live re-plan, which now dates every movie answer.
Two destination defects the first tvdb-grade plan exposed in its own output —
both visible by reading the plan's folder names, neither caught by any gate.
1. Jellyfin sanitizes a hidden folder's display name: `_unsorted` comes back as
`unsorted`. The park-echo guard compared the row's series against the folder
with the underscore, so 1,091 pile rows survived as a real series and the plan
renamed 617 files out of the pile into a VISIBLE folder named `unsorted` —
inventing the junk show the guard exists to prevent. The comparison now strips
leading underscores from both sides.
2. TVDb hands back some series with the year already on the name (`Acapulco
(2021)`). label_of appended the year anyway, producing `Acapulco (2021)
(2021)` — a folder no matcher and no human reaches for twice. A name that
already ends in `(YYYY)` is left alone.
Both are covered by fixtures: a pile row whose series is the sanitized spelling
must park, never file under the pile's name; a provider name carrying its year
produces one year in the path and the plan contains no `(YYYY) (YYYY)` anywhere.
The first tvdb-grade TV plan built 49,771 moves with not ONE TMDb answer: every
search and every external_ids call came back 401. The credential on this box is
the v3 key (~/.config/secrets/tmdb, `api_key=`), and TMDb takes a v3 key only as
the `api_key` query parameter — a v4 request token is the only thing the Bearer
header accepts. Measured on the same credential in one pass: the query form
resolves `New Girl (2011)`, the Bearer form raises HTTP 401.
tmdb_request() now picks the transport from the shape it was handed (three
dot-separated segments = a JWT → Bearer, anything else → query parameter), and
the old header comment's reasoning is corrected: a token in argv is
world-readable in /proc/<pid>/cmdline, but this URL is built inside the process
and never reaches argv, so the query form leaks nothing.
The failure mode that let it look like a finished plan is fixed with it. The
401s were stderr-only, so the summary printed a confident move count while every
series folder still carried the metadata provider's folder-echo name — the exact
junk this contract exists to undo. Provider health is now part of the summary
line and of the exceptions file, and a test covers it offline with
`env -u TMDB_API_KEY`: a plan that asked and could not ask says NOT
provider-complete in both places, and still writes the moves it can justify,
because the answers that did arrive are worth their rename.
The TV relayout that landed copied Jellyfin's Series names verbatim, and the
naming census showed what that launders in: one Series item per release folder
(1,107 items for 1,107 top-level dirs), the same show spelled six ways, and
box sets living in 346 separate folders for 821 identities. Jellyfin names a
Series after the folder it scanned, so `24 (2001) Season 1` is a folder echo,
not a show name.
The name authority is now tvdb, reached through the id the server matched:
* media-census.py jellyfin emits two more columns — `title` (the episode row's
own Name) and `tvdb` (the series id, read from BaseItemProviders where
Jellyfin 12 keeps provider ids and from the row's Data blob where older
builds do, so the census does not care which shape the live DB has);
* media-shows.py resolves each series to tvdb — the row's tvdb id, else its
TMDb id through TMDb's external ids, else an exact tvdb name search — and
writes tvdb's name and year. That also collapses the provider's spellings
into ONE identity, which is what turns a box set split over five release
folders into one folder and what makes two files claiming one episode MEET
in the duplicate rule;
* the episode file becomes `<Series> (<year>) - SxxEyy - <Title>`: the title
comes from tvdb's episode list, else the row's own title, else nowhere — no
title means no separator, never an invented one.
Two refusals come with it. A row whose series is the hidden top-level folder the
file itself sits in is NOT evidence: the review pile parked inside the library
was indexed as one show named `_unsorted`, and its rows would have renamed real
shows to it. And inside such a folder a file's own words may MATCH a registered
show but never INVENT one — invention is what filled the pile. `--park`
therefore belongs outside the library, which is also what makes the next plan
re-file the pile instead of skipping it.
Both providers answer offline-reproducibly: every answer is cached in one JSON
file, so a plan can be re-planned without a key. Lookups stay behind --tmdb and
--tvdb, and the tvdb token is read from a mode-600 file, never argv.
tests/media_shows.bats is 23 tests, none of which touch the network — the
fixture Jellyfin DB carries the providers table and the name cache is an input
file. ./tests/run.sh: 753 green.
Every apply and rollback printed "next: scripts/media-view.sh reap", but on
Dankserver there is nothing to reap: the metadata view has never been installed
(`media-view.sh status` -> NOT MOUNTED, `/var/lib/media-meta` absent) and the
live quadlets still bind the array read-only. All three relayout arms tonight
issued that order — 242,084 moves — and it pointed at a directory that does not
exist. Where no overlay is mounted the servers bind the array itself, so the
sidecars moved with the media.
`--metadata-view` (default /var/lib/media-meta/view) names the overlay and the
note now rides only when that path is a mountpoint. Red first: two new cases in
tests/media_relayout.bats (apply and rollback); the shared helper names a real
mountpoint so the pre-existing presence assertion keeps its meaning. 37 ok.
A scan answers 204 whether or not it found anything, so the fleet's documented
proof — every move row has a row at its dst and zero rows still name its src —
was a hand-written sqlite query every time anyone needed it. That is the step a
half-finished relayout is judged on, so it is now a command with an exit code.
It encodes the two doors that bite: Jellyfin stores the container-side path
(/data stands for the share) while Plex stores the host path verbatim and hides
deleted scans behind deleted_at on two tables. The report breaks the missing rows
down by extension, because a .cue or .log has no row to land in and must not read
as a lost album; --ignore-ext keeps those out of the verdict.
Plex's artist row for 272 chapters is a chapter number the ripper put in the
artist tag (0.4, 31, 05), which would have filed them under a shelf called 0.4.
A narrator is a person, so it must contain a letter; the same test is deliberately
not applied to a book title, which may honestly be 1984.
The first real plan filed 1,106 chapters from 14 unrelated top-level folders
onto one shelf, because all 1,106 carry the album tag [Unknown Album] and the
same artist tag. Plex's album title is the authority for the shelf only when it
names a book; a placeholder names nothing, and unrelated books share it. Those
chapters now park with the path they have today, timestamp-glued variants
included; Various Artists stays a legitimate multi-cast narrator.
The audiobook tree is the one library Jellyfin does not read at all and Plex
reads completely: 58,975 chapters matched into Artist/Album/Track, where the
album is the BOOK and the book's single artist row is the NARRATOR. That is the
authority the new planner files by, never the collection folder a pack made.
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>
Two laws are the opposite of the music tool's, both forced by the rows: files
are never renumbered (72% of the books carry chapters with no usable index, and
playback order comes from the embedded track tag either way, so an invented
number can only lie), and the shelf is the narrator, not the author (every book
has exactly one artist row; the author survives in 10% of album titles and in no
column Plex stores, so one rule beats a shelf half authors and half narrators).
media-census.py gains the audiobooks export: the chapters Plex matched under a
root, host paths straight through, nothing outside the root and nothing Plex
deleted. 22 tests in tests/media_audiobooks.bats pin the export's join and the
plan's output.
The music library is rips and discography packs whose filenames say nothing the
matcher needs. The authority is Jellyfin's own audio rows: one per track, with
the album and album-artist ids it carries in BaseItemProviders (the item JSON
holds an empty ProviderIds for albums in this build).
Grouping is what the tags assert, not where files sit: (album-artist tag, album,
year), case-folded. A compilation is one album whose tracks name many performers,
so it files once under Various Artists; two album-artist tags on one album name
are two albums. The folder spelling is the dominant one, because a name taken
from whichever row sorted first depends on row order and re-names albums that are
already filed.
An occupied path is never a destination: the executor refuses occupied targets,
so a plan that aims at one fails after the renames meant to clear it. A file
standing on its own target, or already carrying a collision suffix, is a keep -
otherwise the plan shuffles instead of being a fixed point.
A dry run of the TV planner against the real 58k-file library produced a root
series called "01 NCOP" out of a bonus subfolder, copied Jellyfin's stale
numbering (every file under SNL 16-20/Season 16..19 carries season 16 episode
20; one Gundam row says season 2004), and lost subtitles a scene pack had
buried in Subs/<release-name>/.
- Folders now RESOLVE a series and never invent one: a folder's words match a
registered name as a token run ("(CBB) Tengen Toppa Gurren Lagann - The
Complete Series (BD 1080p)" names Tengen Toppa Gurren Lagann), and folder
words alone name nothing, so an unknown pack's bonus folder parks instead of
opening a new show at the library root.
- Numbering priority: the marker the file's own name states, then the Season NN
folder plus the number the name leads with inside it, then the provider row.
A row that still insists on season 1989 parks its file rather than creating
Season 1989.
- The provider row keeps identity: the registered name and its year survive a
file whose own marker numbers it, and the pack index a release leads with
("4a. Seed DESTINY") is folded out of the folder name.
- A sidecar searches every ancestor folder for the video it names, not just the
parent, so Subs/<release-name>/<release>.jpn.srt follows its episode.
- The shared bonus-folder test no longer reads an index as a title: name_words
drops "extras" as release junk, so "01. Extras" reduced to "01" and matched
the "01" of "NCOP 01", cancelling the bonus evidence. Only words can name a
film.
The first run against the real library died in the anchor lookup: a subtitle
whose own name carries the media extension matches BOTH prefix forms of its
video, and the "keep the longest" comparison read the destination slot of the
tuple instead of the length slot. The fixture only ever matched one prefix, so
the seam was green over a crash.
The tail the sidecar keeps is now stripped of that container word: Jellyfin and
Plex both read "Show S01E01.forced.srt", not "Show S01E01.mkv.forced.srt", and
an nfo still carries the media extension because that one is constructed, not
parsed.
The Movies relayout planned from Plex; the TV library has no Plex rows and
58k files whose names lie about which episode they hold. Jellyfin has already
matched and numbered 47,676 of them, so the planner treats that numbering as
the authority and the file name as the fallback: an episode number the name
states, or a provider row, or nothing - and "nothing" parks the file under
_unsorted with a reason instead of inventing a number.
- media-census.py jellyfin: one TSV row per matched episode, its container
path mapped back to the host, series name/year/TMDb id from the Series row
(the episode row carries none of them). Both TSV writers now end lines with
\n so a shell consumer sees the last field clean.
- media-shows.py: the plan. Series folders take the provider's own spelling;
one file claiming two episodes keeps both numbers; two files claiming one
episode - the bigger keeps the standard name, the other parks; bonus
material files under the series' extras folder, never as an episode; a long
file inside the TV library leaves for Movies only when TMDb answers with a
matching title, otherwise it parks. Re-running the plan on a library it
already filed moves nothing.
- tests/media_shows.bats: 15 behaviors, including the executor preflight and
the re-plan stability law.
Jellyfin 12 ships TMDB, TVDB, StudioImages and MusicBrainz as core plugins:
no DLL, only a configuration file under the data root, so the keys have to
reach that XML from somewhere that is not git. They live in the fleet's
per-service env-file convention -- /home/dank/Containers/jellyfin/provider-keys
(TMDB_API_KEY=, TVDB_API_KEY=, optional TVDB_PIN=, mode 600) -- and
scripts/jellyfin-providers.sh plan|install|verify reads only that file, prints
no key byte, and refuses a keys file that is not mode 600 or that has no line
for the provider about to be written. install is idempotent: it never touches
a key the UI already saved, backs up every XML it edits into
~/.AGENTTRASH/<date>-jellyfin-providers/ first, and only installs the TVDB
plugin when the pinned official zip carries the plugin guid.
The deployer runs install BEFORE it restarts jellyfin, because the plugin and
its keys are read at start-up and a restart-first deploy boots the old state.
A missing keys file warns instead of failing: a host without the keys still
deploys, degraded rather than broken. No plugin configuration XML is tracked.
The container has had one owner since the Quadlet cutover
(dankserver/jellyfin/quadlet/jellyfin.container). The block was a second owner,
and wrong on every path: jellyfin/jellyfin:latest instead of the pinned ghcr
image, /opt/jellyfin/* data roots that no longer exist, and a read-only
/mnt/entropy:/data bind that the metadata view has since replaced.
The planner no longer counts jellyfin as a compose service, so editing the
inventory cannot restart a media server; tests/deployment_scoping.bats pins that
contract for both jellyfin and plex.
The merged door was /mnt/media, which read like a third volume and split the host
state across two trees. It is now /var/lib/media-meta/view, so the array layer,
the SSD upper layer, the kernel workdir, the reap trash and the merged tree are
one directory — and the mount table gains no new /mnt entry beside the array.
systemd names a mount unit after its ESCAPED path, so the repo copy keeps the
readable name media-meta-view.mount while `install` writes the host file as
var-lib-media\x2dmeta-view.mount; `plan` prints both and the page records them, so
nobody hunts for a unit that "does not exist". install also creates the mount
point itself, because systemd will not.
The seam now additionally pins that the unit's Where= is exactly the path both
containers bind — guard and bind cannot drift apart. The Quadlet Lifecycle
phantom-path example is corrected too: it named an invented /mnt/media/*, while
the real phantom is the legacy /opt/jellyfin/* that dankserver/compose.yaml still
carries for a service whose roots moved to /home/dank/Containers/jellyfin.
A relayout renames on the raw array; the sidecars the media servers wrote live in
the NVMe upper layer of the view and stay at the old coordinates, which is how a
ghost folder (artwork and an .nfo, no video) is born. apply and rollback now
print the follow-up — scripts/media-view.sh reap — at the moment the operator
needs it, and the existing apply seam asserts the hint is in the output.
Both media servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo /
episode.nfo) only when it sits next to the media file, and the library lives on
the RAID6 array, deliberately mounted read-only for every container and slow to
seek for a thousand tiny XML reads. So neither server binds the array any more:
both bind /mnt/media, an OverlayFS merge of the array (lowerdir, never written)
with /var/lib/media-meta on the NVMe (upperdir, where every write lands).
Jellyfin binds the view rw at /data and is the fleet's only metadata writer;
Plex binds it ro at /mnt/entropy, the container path its own database already
recorded, so only the host side of that bind moved and its library paths keep
resolving.
scripts/media-view.sh plans, installs, verifies, reports and reaps the view. It
refuses to mount over an unmounted array, refuses an upper layer on rotational
storage, and refuses a non-root install. deploy-remote.sh raises the view before
it restarts either media server, both units carry
ConditionPathIsMountPoint=/mnt/media so a missing view skips the start instead of
letting a scan clear the library database, and the planner routes the installer
and dankserver/host/** to the jellyfin+plex pair.
tests/media_overlay.bats (11 seams) pins the resolved shape, both refusals, the
two binds and their guard, the deployer's ordering, the planner mapping, and the
reap behaviour against a fixture (ghosts listed, then moved to the SSD trash --
files that exist only above are sidecars, not ghosts, so only directories are
candidates).
Two fixes found by one live sweep, shipped together because the second is
what makes the first finishable.
1. scripts/media-tv.py — the other half of media-plan.py's refusal to file
TV. Groups episode-shaped files by the series name before the marker
(media-titles.ep_of, imported not restated), writes a plan in the
executor's own format: <library>/<Series> (<year>)/Season <NN>/<own
name>.<ext>. Refuses a group below --min, an episode naming no season,
and an episode the destination library already holds at the same byte
count — the movie share carries a second copy of all 124 Monk episodes
the TV library has had for years; a duplicate is Frank's call, never a
tool's copy. Applied by media-apply.py with its own journal and
--root /mnt/entropy.
2. scripts/media-apply.py — gone meant "rmdir intended", so a folder whose
rmdir failed was blacklisted forever and no later sweep could finish it
(live movie sweep: 4517 targeted, 146 permanently stranded). gone now
means "actually gone": a failed intent is retried by the next sweep.
tests: 9 new tests in media_relayout.bats, each red against the previous
revision — 8 TV + 1 sweep retry behind a chmod 500 parent. ./tests/run.sh
633 green. No live move run yet.
The live sweep removed 2271 husks and left 146 standing, every one a release
folder blocked by a junk subfolder (Screens/, Sample/, a BDMV leaf) that no
file ever moved out of. Eligibility was per-folder, so the blocker was never a
target and its parent could never be removed.
Proof now carries downward: inside a folder the journal shows emptied, every
empty directory qualifies. Nothing under such a folder can hold data, and every
removal is still journalled as rmdir and undone by rollback.
The new test is red against the previous revision, green now. The live sweep
touched no file: 8193 before, 8193 after.
Keyed on the whole episode title, every episode of a box set became a group of
one and never reached the five-episode size that calls something a series, so
the live run filed 124 Monk episodes as 124 movie folders, each named after its
own episode. ep_of now returns the series — the name before SxxEyy / 1x04 /
Part N — and the planner groups and names the quarantine pile by that series,
so one show is one pile: 'Monk - S05E03 - Mr. Monk and the Big Game' is series
monk. The threshold keeps its job: a lone film named Part 2 stays a film.
Re-planning the live census with this moves nothing differently except the 124
Monk files, which now join the TV pile: 168 episode files across 3 series.
An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.
* media-census.py the read-only inputs: a stat pass, an ffprobe header pass
(nice'd, so it is safe beside a live endpoint), and the
Plex metadata cache read that outranks any filename guess
* media-plan.py the Jellyfin-format planner: one folder per movie, the
video named exactly like the folder, extras in the
canonical subfolders, copies of one film as versions. It
opens no media file, so a whole-library plan costs seconds,
and it ends in gates — bad-names, occupied, double-claim,
left-unmoved, unplanned — that must all read zero before a
human points the executor at 24 TB
* media-apply.py the executor: rename(2) only, the intent fsynced to a
write-ahead journal BEFORE the move it authorises, so an
interrupted run is finishable and every run is undoable.
Refusals are all-or-nothing and it never overwrites a file
Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):
* a Plex year of 0 means "indexed, never matched" — trusting it produced
folders "Title (0)": items with no year, no poster, no identity
* a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
tree neither matcher reads as a movie
* a folder inside a movie's folder is bonus territory even when its name is
misspelled ("Exrtas"), unless the file's own name carries another year
* matching words split on everything but a letter or digit — the earlier
tokenizer left "(1960)/extras" glued, so a film could not be found by the
folder holding it and one box set's featurettes drifted into a neighbour
* a name with no words in it is bonus material, never a movie folder named 1
Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
A by-digest pull leaves the image carrying every digest it was pulled under, so
reading RepoDigests[0] reported a healthy pinned image as wrong and refused a
window that was ready to run. Membership, not position. Also records the measured
step optimum: 106 tok/s at an 8,192-token step against 76 at 16,384 on a 32,768
calibration window, so the wide step has a ceiling, not a plateau.
Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid
scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.
The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.
Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
Frank's rule: the served iq4r this arm competes with was calibrated on a
10M-token set, so the arm is calibrated at least as deeply. The floor was
lowered to 1.5M on the vendor's '10M beats 1M by under 1%' note; that
measures the vendor's own plate, not ours — the arm is judged against a
10M-calibrated reference, so cutting the calibration 6x to save wall clock
buys a worse position on the only comparison that matters.
One round of the re-chunked corpus is ~9.31M tokens, so a 12M floor seals
in round 1 plus part of round 2. A round that seals nothing is normal (the
floor is pooled) and round_verdict treats it as continue, never failure.
Also fixes tests/qwen38_campaign.bats' window dry run: its fixture lands on
/tmp (a 16 GiB tmpfs) where the 60 GiB seal preflight can never pass, so the
test now uses the documented Q38_CALIB_MIN_FREE_MIB override instead of
asserting environment luck.
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.
LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.
The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.
Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.
0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.
The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).
Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.
The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.
Four gates, each with the measurement that motivated it:
- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
@sha256:665775ed…), overridable only by naming --image deliberately. Proven by
the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
starts the service. `podman rm -f` does not kill the engine: an orphan made
production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
target configurable, so the tap lives on /mnt/entropy instead of the root fs.
Measured from the engine's own counters on the 2026-10-29 drain: prefill 71 tok/s steady state, a
180 s window showing 7 prefill steps of the capped 2,048 tokens and 3 preemptions. One 7,168-token
document is therefore ~100 s, round 1 (2,367 docs, 10.03M tokens) ~39 h, and the 12M-position
floor ~47 h of drained production — not the ~18 h the brief assumed.
The tap is the only phase whose cost is read volume per document: at the inherited 2,048-token step
a document is four steps, and each step re-streams every layer's routed expert union from the
container (43.4 GiB a rank live on disk). One wide step reads each layer's union once.
The 'one step, one slot count' audit law binds kldref and score because their KL numbers are only
comparable at an identical shape. Calib produces layer Hessians and no number any report compares,
so Q38_CALIB_MNBT / Q38_CALIB_SEQS are its own, defaulting to the inherited value — nothing moves
unless a probe asks. The KL modes ignore both.
Tests: 589/0 — the knob reaches calib's argv, does not reach kldref's, and a non-numeric value is
refused by name.
RADIANCE_CALIB_TOKENS=12000000 is a pooled floor and calib-7168.jsonl carries 10.03M tokens, so
round 1 of a corpus push seals nothing by construction. The gate PR #84 added failed exactly
there, one round too early, while the drain was already running.
Now round_verdict() separates the cases: 'shrink' (the tap lost bytes) fails the run, 'flat' says
so and continues, 'grew' continues, 'done' stops. A run that never sees the engine's completion
line still exits non-zero on max rounds, and the rig marker remains the lock that stops a push
into anything that is not the rig.
Test: round_verdict's four verdicts, behaviorally.
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.
- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
and that container is running. With the driver's existing tap-progress gate, a push can no
longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
instead of falling through the health wait, bank the 336 GiB reference container into trash
only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
section that names the live campaign instead of the retired swift15 era.
Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.
Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
Frank
dank
dank