Compare commits

..
Author SHA1 Message Date
Frank 82a2183d40 docs(media): a film handoff needs the file's own words and year
The provider's first hit for a salvaged query is a near miss, and the page said
only that a provider must name the file. The bullet now states the two checks and
the two real misfiles they exist to stop.
2026-10-31 07:39:45 -07:00
Frank c70f9b3f6d fix(media): a film leaves the TV library only for a film it actually names
The first tvdb-grade plan handed 12 files to the movie library, and two of them
were wrong in a way the exceptions file could not show: an episode of the
documentary series `Russia 1985-1999: TraumaZone` was renamed to the 1963 Bond
film `From Russia with Love`, and `Gundam 00 Special Edition III - Return The
World` became `Special Edition I - Celestial Being`.

Two guards, each alone insufficient:

- `title_of` salvages a short query from a long release name ("russia", "mobile
  suit gundam 00"), and tmdb() took the FIRST search result. The call site then
  compared the provider's words against that salvaged query, so any title merely
  CONTAINING the query passed. The provider's title must now be a run of words the
  FILE itself states: `Special Edition I: Celestial Being` is not in a filename
  that says `Special Edition III - Return The World`.
- tmdb() read the year from `year`/`first_air_date`, which the MOVIE search does
  not return — it returns `release_date`. Every film answer therefore arrived
  year-less and the year guard at the call site could never fire. That is how a
  1963 answer survived a filename stating 1991.

Tests: a provider answer the file does not name parks, and nothing lands in the
movie library; an answer from another year is refused. The year-from-the-wire half
is proven by the live re-plan, which now dates every movie answer.
2026-10-31 07:38:54 -07:00
Frank 32b8901bff docs(media): the pile is matched with its underscore stripped
The server drops the leading underscore from the hidden folder's display name, so
the park-echo rule that compares spelled names lets the review pile survive as a
visible series. The law now says which spelling is compared; the mechanism stays
with the guard's docstring.
2026-10-31 07:28:22 -07:00
Frank 6d03bbc8e7 fix(media): the review pile is not a series, and a folder carries the year once
Two destination defects the first tvdb-grade plan exposed in its own output —
both visible by reading the plan's folder names, neither caught by any gate.

1. Jellyfin sanitizes a hidden folder's display name: `_unsorted` comes back as
   `unsorted`. The park-echo guard compared the row's series against the folder
   with the underscore, so 1,091 pile rows survived as a real series and the plan
   renamed 617 files out of the pile into a VISIBLE folder named `unsorted` —
   inventing the junk show the guard exists to prevent. The comparison now strips
   leading underscores from both sides.

2. TVDb hands back some series with the year already on the name (`Acapulco
   (2021)`). label_of appended the year anyway, producing `Acapulco (2021)
   (2021)` — a folder no matcher and no human reaches for twice. A name that
   already ends in `(YYYY)` is left alone.

Both are covered by fixtures: a pile row whose series is the sanitized spelling
must park, never file under the pile's name; a provider name carrying its year
produces one year in the path and the plan contains no `(YYYY) (YYYY)` anywhere.
2026-10-31 07:27:47 -07:00
Frank 32b58dd787 fix(media): TMDb's v3 key travels as a query parameter; Bearer is a JWT only
The first tvdb-grade TV plan built 49,771 moves with not ONE TMDb answer: every
search and every external_ids call came back 401. The credential on this box is
the v3 key (~/.config/secrets/tmdb, `api_key=`), and TMDb takes a v3 key only as
the `api_key` query parameter — a v4 request token is the only thing the Bearer
header accepts. Measured on the same credential in one pass: the query form
resolves `New Girl (2011)`, the Bearer form raises HTTP 401.

tmdb_request() now picks the transport from the shape it was handed (three
dot-separated segments = a JWT → Bearer, anything else → query parameter), and
the old header comment's reasoning is corrected: a token in argv is
world-readable in /proc/<pid>/cmdline, but this URL is built inside the process
and never reaches argv, so the query form leaks nothing.

The failure mode that let it look like a finished plan is fixed with it. The
401s were stderr-only, so the summary printed a confident move count while every
series folder still carried the metadata provider's folder-echo name — the exact
junk this contract exists to undo. Provider health is now part of the summary
line and of the exceptions file, and a test covers it offline with
`env -u TMDB_API_KEY`: a plan that asked and could not ask says NOT
provider-complete in both places, and still writes the moves it can justify,
because the answers that did arrive are worth their rename.
2026-10-31 07:16:46 -07:00
Frank 0a365274b6 docs(agent): a PR comment posts to the issues endpoint, and an empty jq body is silent
Measured while commenting on PR 102: POST /pulls/102/comments answers 405 to a
plain body (it is the review-comment store); the conversation thread is
POST /issues/<PR-number>/comments, which answers 201 and returns an html_url on
the PR. Gitea maintains one issue index, so the PR number is the same number.

The trap that hid it: building the body with jq --rawfile b /dev/stdin inside a
command substitution feeds jq nothing (the substitution does not inherit the
heredoc), Gitea answers 422, and jq -r .id on the error prints nothing — a
silent no-op that looked like a posted comment. The body goes in a file.
2026-10-31 07:11:13 -07:00
Frank e531e90278 docs(media): a scan request belongs to the server's own page
Media Relayout.md hit the vault's 500-line gate the moment the tvdb contract
landed, and the fat in it was not the new text: the Plex and Jellyfin scan
blocks (35 lines) sat inside the relayout procedure even though they are the
way you talk to those servers after ANY change, and two Laws bullets repeated
the same facts a third time. Both blocks now live on [[Plex]] and
[[Jellyfin]]; the relayout page keeps the two laws an operator needs at that
moment — 204 is acceptance and anything else queued nothing, and one accepted
refresh is the whole procedure, because a second moves `Scan Media Library` to
`Cancelling` and every later 204 is discarded.

Same pass, the dead numbers the vault's own law says to destroy: an applied
music pass's date, an applied audiobook pass's move/park counts, and the 146
sweep stragglers of one specific first sweep. The laws they taught stay.

490 lines, `bats tests/docs_completeness.bats` 6/6 (length violation gone,
link integrity clean).
2026-10-31 07:10:28 -07:00
Frank c3ab565d9b feat(media): tvdb names the show — episode titles, id-backed folders, park outside
The TV relayout that landed copied Jellyfin's Series names verbatim, and the
naming census showed what that launders in: one Series item per release folder
(1,107 items for 1,107 top-level dirs), the same show spelled six ways, and
box sets living in 346 separate folders for 821 identities. Jellyfin names a
Series after the folder it scanned, so `24 (2001) Season 1` is a folder echo,
not a show name.

The name authority is now tvdb, reached through the id the server matched:

  * media-census.py jellyfin emits two more columns — `title` (the episode row's
    own Name) and `tvdb` (the series id, read from BaseItemProviders where
    Jellyfin 12 keeps provider ids and from the row's Data blob where older
    builds do, so the census does not care which shape the live DB has);
  * media-shows.py resolves each series to tvdb — the row's tvdb id, else its
    TMDb id through TMDb's external ids, else an exact tvdb name search — and
    writes tvdb's name and year. That also collapses the provider's spellings
    into ONE identity, which is what turns a box set split over five release
    folders into one folder and what makes two files claiming one episode MEET
    in the duplicate rule;
  * the episode file becomes `<Series> (<year>) - SxxEyy - <Title>`: the title
    comes from tvdb's episode list, else the row's own title, else nowhere — no
    title means no separator, never an invented one.

Two refusals come with it. A row whose series is the hidden top-level folder the
file itself sits in is NOT evidence: the review pile parked inside the library
was indexed as one show named `_unsorted`, and its rows would have renamed real
shows to it. And inside such a folder a file's own words may MATCH a registered
show but never INVENT one — invention is what filled the pile. `--park`
therefore belongs outside the library, which is also what makes the next plan
re-file the pile instead of skipping it.

Both providers answer offline-reproducibly: every answer is cached in one JSON
file, so a plan can be re-planned without a key. Lookups stay behind --tmdb and
--tvdb, and the tvdb token is read from a mode-600 file, never argv.

tests/media_shows.bats is 23 tests, none of which touch the network — the
fixture Jellyfin DB carries the providers table and the name cache is an input
file. ./tests/run.sh: 753 green.
2026-10-31 07:01:50 -07:00
Frank 5dbbfcb8a0 docs(media): an accepted refresh can still be dropped by a wedged task runner
Tonight's sequence, measured: the post-sweep refresh answered 204 while
`Scan Media Library` sat in `Cancelling`, and Jellyfin's stale-path counts froze
for an hour at 0 % CPU — the request was accepted and discarded. Restarting the
user unit cleared the runner; one POST after that reported State `Running` at
130 % CPU and the counts began to move. So the status code is not the evidence,
and the restart is `systemctl --user restart jellyfin.service`, never
`deploy.sh`, which would run the never-tested metadata-view cutover.
2026-10-31 04:13:12 -07:00
Frank 02a568fb97 docs(media): the Jellyfin scan 401 was curl-config quoting, not the server
Corrects the lesson written an hour ago, which said a non-204 was transient and
should simply be re-issued. It is reproducible and mechanical: `printf` with
`\"` inside the single-quoted format string drops the backslash, curl then ends
the quoted value at the first bare quote, and Jellyfin receives
`Authorization: MediaBrowser Token=` with an empty key. Both forms were run
against the host in one pass — `\\"` answers 204, `\"` answers 401 — so the
block's double backslashes are load-bearing, and the handoff script that deviated
left a 58k-move library unscanned for 40 minutes.

Also the trap that diagnosis walked into: a second accepted refresh while a scan
is running moves `Scan Media Library` to `Cancelling` and the pass starts over.
One accepted request is the procedure; wait it out.
2026-10-31 03:41:22 -07:00
Frank 1e1527d101 docs(media): the SSD metadata view has never been deployed on Dankserver
The vault described the overlay as the live shape. It is not: `media-view.sh
status` answers NOT MOUNTED, /var/lib/media-meta does not exist, and the
installed quadlets under ~/.config/containers/systemd still carry the pre-view
binds (/mnt/entropy:/data:ro for Jellyfin, /mnt/entropy:/mnt/entropy:ro for
Plex) while git binds the view with ConditionPathIsMountPoint. The deploy gate
(deploy-remote.sh:ensure_media_view) has therefore never run: the first
`deploy.sh dankserver --service jellyfin|plex` is an untested cutover of both
media servers, and Jellyfin's read-only sidecar error is today's expected state,
not a fault.

Same commit, the scan-acceptance law from tonight's TV handoff: it logged
`jellyfin scan: 401`, so Jellyfin never rescanned a 58k-move library until the
request was re-issued with the identical key and header form (204). Anything
that is not 204 means no scan was queued — re-issue, do not diagnose.
2026-10-31 03:13:53 -07:00
Frank 9eb85ea1f9 fix(media): order the sidecar reap only where the view is mounted
Every apply and rollback printed "next: scripts/media-view.sh reap", but on
Dankserver there is nothing to reap: the metadata view has never been installed
(`media-view.sh status` -> NOT MOUNTED, `/var/lib/media-meta` absent) and the
live quadlets still bind the array read-only. All three relayout arms tonight
issued that order — 242,084 moves — and it pointed at a directory that does not
exist. Where no overlay is mounted the servers bind the array itself, so the
sidecars moved with the media.

`--metadata-view` (default /var/lib/media-meta/view) names the overlay and the
note now rides only when that path is a mountpoint. Red first: two new cases in
tests/media_relayout.bats (apply and rollback); the shared helper names a real
mountpoint so the pre-existing presence assertion keeps its meaning. 37 ok.
2026-10-31 03:13:53 -07:00
dank 96944f0b17 feat(media): media-verify.py replays a journal against the servers' own tables
A scan answers 204 whether or not it found anything, so the fleet's documented
proof — every move row has a row at its dst and zero rows still name its src —
was a hand-written sqlite query every time anyone needed it. That is the step a
half-finished relayout is judged on, so it is now a command with an exit code.

It encodes the two doors that bite: Jellyfin stores the container-side path
(/data stands for the share) while Plex stores the host path verbatim and hides
deleted scans behind deleted_at on two tables. The report breaks the missing rows
down by extension, because a .cue or .log has no row to land in and must not read
as a lost album; --ignore-ext keeps those out of the verdict.
2026-10-31 01:02:48 -07:00
dank 33f97cd3b3 docs(media): the TV library arm gets its own journal, and its plan lives on the array
Two relayouts writing one journal cannot be rolled back apart: the mover
(Movies -> TVShows) and the library re-filer both pointed at
media-relayout-tv.jsonl. The re-filer now writes media-relayout-shows.jsonl,
and its plan/exceptions move off /tmp, which a reboot eats and which is not a
record.
2026-10-31 00:55:32 -07:00
dank 254fb41fd0 docs(media): the music relayout is mid-flight, and the array takes one writer
The page said the music plan "was applied" while 62k of its 131k journal rows
were still being written; a runbook line that gets believed is worse than none.
Also lands the single-writer law the audiobook handoff exists to honour, and the
durable plan path (/tmp is not a record).
2026-10-31 00:40:30 -07:00
dank 3613f69593 Merge pull request 'feat(media): the audiobook relayout, filed by the book Plex matched' (#101) from feat/media-audiobook-relayout into main 2026-10-31 00:38:02 -07:00
dank 6bf70c6a64 fix(media): a shelf needs a name, so a mistagged chapter number parks
Plex's artist row for 272 chapters is a chapter number the ripper put in the
artist tag (0.4, 31, 05), which would have filed them under a shelf called 0.4.
A narrator is a person, so it must contain a letter; the same test is deliberately
not applied to a book title, which may honestly be 1984.
2026-10-31 00:34:40 -07:00
dank f1b8b60817 fix(media): a tag-scan placeholder is not a book identity
The first real plan filed 1,106 chapters from 14 unrelated top-level folders
onto one shelf, because all 1,106 carry the album tag [Unknown Album] and the
same artist tag. Plex's album title is the authority for the shelf only when it
names a book; a placeholder names nothing, and unrelated books share it. Those
chapters now park with the path they have today, timestamp-glued variants
included; Various Artists stays a legitimate multi-cast narrator.
2026-10-31 00:31:30 -07:00
dank 4ae2bde9ab docs(media): fit the audiobook section inside the OKF caps, refresh the index
The linter caps a page at 500 lines and a description at 120 chars: the section
lost four lines of prose (no law went with them) and the description lost its
restatement of the format names. gen-vault-index.py regenerated the locator row.
2026-10-31 00:25:58 -07:00
dank ceb5617a42 docs(media): the audiobook arm, and the music plan's applied state
One page owns the relayout facts: the new section carries the Plex-only reader,
the shape, the four commands, and the two laws a future operator would otherwise
re-litigate (never renumber a chapter; the shelf is the narrator). The music
section said its plan was NOT applied — it was applied on 2026-10-31, and a stale
line in a runbook gets believed, so it now names the journal and the rollback.
2026-10-31 00:24:50 -07:00
dank a7ceda95fd feat(media): the audiobook relayout, filed by the book Plex matched
The audiobook tree is the one library Jellyfin does not read at all and Plex
reads completely: 58,975 chapters matched into Artist/Album/Track, where the
album is the BOOK and the book's single artist row is the NARRATOR. That is the
authority the new planner files by, never the collection folder a pack made.

  Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>

Two laws are the opposite of the music tool's, both forced by the rows: files
are never renumbered (72% of the books carry chapters with no usable index, and
playback order comes from the embedded track tag either way, so an invented
number can only lie), and the shelf is the narrator, not the author (every book
has exactly one artist row; the author survives in 10% of album titles and in no
column Plex stores, so one rule beats a shelf half authors and half narrators).

media-census.py gains the audiobooks export: the chapters Plex matched under a
root, host paths straight through, nothing outside the root and nothing Plex
deleted. 22 tests in tests/media_audiobooks.bats pin the export's join and the
plan's output.
2026-10-31 00:24:45 -07:00
dank a32361ee5e Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-31 00:10:58 -07:00
dank ce6b3b1d3b docs(media): how to prove a relayout landed, read from both servers' tables
A scan answers 204 whether or not it found anything, so proof is the journal
replayed against Jellyfin's BaseItems.Path and Plex's media_parts.file: every
move has a row at its dst, no row names its src. Records the four wrong doors:
no sqlite3 binary on the hosts or in either container, Jellyfin's single
BaseItems table (no MediaItems), its container-side /data path prefix, and
Plex's HTTPD closing on ?path= / on /search matching titles not paths.
2026-10-31 00:10:50 -07:00
dank 382e3eb0b8 Merge pull request 'feat(media): the music library's own relayout, filed by its tags' (#100) from feat/jellyfin-provider-keys into main 2026-10-31 00:04:41 -07:00
Frank 54a393edab Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-30 23:58:34 -07:00
Frank a5459b0b1e docs(media): the music library's own relayout, with the laws it earned 2026-10-30 23:58:34 -07:00
Frank b03931f7ce feat(media): a music relayout that files by the tag row Jellyfin matched
The music library is rips and discography packs whose filenames say nothing the
matcher needs. The authority is Jellyfin's own audio rows: one per track, with
the album and album-artist ids it carries in BaseItemProviders (the item JSON
holds an empty ProviderIds for albums in this build).

Grouping is what the tags assert, not where files sit: (album-artist tag, album,
year), case-folded. A compilation is one album whose tracks name many performers,
so it files once under Various Artists; two album-artist tags on one album name
are two albums. The folder spelling is the dominant one, because a name taken
from whichever row sorted first depends on row order and re-names albums that are
already filed.

An occupied path is never a destination: the executor refuses occupied targets,
so a plan that aims at one fails after the renames meant to clear it. A file
standing on its own target, or already carrying a collision suffix, is a keep -
otherwise the plan shuffles instead of being a fixed point.
2026-10-30 23:58:34 -07:00
dank 82f6ec55c2 Merge pull request 'feat(media): plan a TV relayout from the numbering Jellyfin already has' (#99) from feat/media-tv-relayout into main 2026-10-30 23:38:36 -07:00
Frank 536949c18d docs(media): the TV library's own relayout, with the evidence chain it believes
One page owns relayout: the census commands, the order the planner believes
things in, the two refusals a dry run on the real library proved necessary, the
apply and rollback path, and the standing wart that a provider name can itself
be scene junk ("Black Clover 001", 92 episodes) — renaming that is Jellyfin's
job, not the mover's.
2026-10-30 23:35:14 -07:00
Frank 337afac91e fix(media): a TV plan that follows the authorities it was given
A dry run of the TV planner against the real 58k-file library produced a root
series called "01 NCOP" out of a bonus subfolder, copied Jellyfin's stale
numbering (every file under SNL 16-20/Season 16..19 carries season 16 episode
20; one Gundam row says season 2004), and lost subtitles a scene pack had
buried in Subs/<release-name>/.

- Folders now RESOLVE a series and never invent one: a folder's words match a
  registered name as a token run ("(CBB) Tengen Toppa Gurren Lagann - The
  Complete Series (BD 1080p)" names Tengen Toppa Gurren Lagann), and folder
  words alone name nothing, so an unknown pack's bonus folder parks instead of
  opening a new show at the library root.
- Numbering priority: the marker the file's own name states, then the Season NN
  folder plus the number the name leads with inside it, then the provider row.
  A row that still insists on season 1989 parks its file rather than creating
  Season 1989.
- The provider row keeps identity: the registered name and its year survive a
  file whose own marker numbers it, and the pack index a release leads with
  ("4a. Seed DESTINY") is folded out of the folder name.
- A sidecar searches every ancestor folder for the video it names, not just the
  parent, so Subs/<release-name>/<release>.jpn.srt follows its episode.
- The shared bonus-folder test no longer reads an index as a title: name_words
  drops "extras" as release junk, so "01. Extras" reduced to "01" and matched
  the "01" of "NCOP 01", cancelling the bonus evidence. Only words can name a
  film.
2026-10-30 23:29:34 -07:00
Frank c0c77590b1 fix(media): a sidecar named "episode.mkv.srt" anchors, and loses the container word
The first run against the real library died in the anchor lookup: a subtitle
whose own name carries the media extension matches BOTH prefix forms of its
video, and the "keep the longest" comparison read the destination slot of the
tuple instead of the length slot. The fixture only ever matched one prefix, so
the seam was green over a crash.

The tail the sidecar keeps is now stripped of that container word: Jellyfin and
Plex both read "Show S01E01.forced.srt", not "Show S01E01.mkv.forced.srt", and
an nfo still carries the media extension because that one is constructed, not
parsed.
2026-10-30 23:17:55 -07:00
Frank 441b908937 feat(media): plan a TV relayout from the numbering Jellyfin already has
The Movies relayout planned from Plex; the TV library has no Plex rows and
58k files whose names lie about which episode they hold. Jellyfin has already
matched and numbered 47,676 of them, so the planner treats that numbering as
the authority and the file name as the fallback: an episode number the name
states, or a provider row, or nothing - and "nothing" parks the file under
_unsorted with a reason instead of inventing a number.

  - media-census.py jellyfin: one TSV row per matched episode, its container
    path mapped back to the host, series name/year/TMDb id from the Series row
    (the episode row carries none of them). Both TSV writers now end lines with
    \n so a shell consumer sees the last field clean.
  - media-shows.py: the plan. Series folders take the provider's own spelling;
    one file claiming two episodes keeps both numbers; two files claiming one
    episode - the bigger keeps the standard name, the other parks; bonus
    material files under the series' extras folder, never as an episode; a long
    file inside the TV library leaves for Movies only when TMDb answers with a
    matching title, otherwise it parks. Re-running the plan on a library it
    already filed moves nothing.
  - tests/media_shows.bats: 15 behaviors, including the executor preflight and
    the re-plan stability law.
2026-10-30 23:15:32 -07:00
dank 1a1dbac268 Merge pull request 'Jellyfin metadata providers: keys ride a host file the deployer installs' (#98) from feat/jellyfin-provider-keys into main 2026-10-30 22:48:21 -07:00
Frank 949d3cb6f4 docs(jellyfin): one page owns the provider keys and where they live
The page carries the deploy-time shape: which four providers are core plugins
in Jellyfin 12, which one is not, the exact XML each key lives in, the host
file the keys come from and its 600 rule, the three-way pin on the TVDB
artifact with its rollback, and what the deployer does when the keys file is
absent. OMDb is recorded as deliberately unconfigured -- keyed by email,
1000 req/day, and a ToS that forbids storing what it returns. index.md's row
carries the same locator.
2026-10-30 22:47:49 -07:00
Frank 3ce8d62e70 feat(jellyfin): the provider keys ride a host file the deployer installs
Jellyfin 12 ships TMDB, TVDB, StudioImages and MusicBrainz as core plugins:
no DLL, only a configuration file under the data root, so the keys have to
reach that XML from somewhere that is not git. They live in the fleet's
per-service env-file convention -- /home/dank/Containers/jellyfin/provider-keys
(TMDB_API_KEY=, TVDB_API_KEY=, optional TVDB_PIN=, mode 600) -- and
scripts/jellyfin-providers.sh plan|install|verify reads only that file, prints
no key byte, and refuses a keys file that is not mode 600 or that has no line
for the provider about to be written. install is idempotent: it never touches
a key the UI already saved, backs up every XML it edits into
~/.AGENTTRASH/<date>-jellyfin-providers/ first, and only installs the TVDB
plugin when the pinned official zip carries the plugin guid.

The deployer runs install BEFORE it restarts jellyfin, because the plugin and
its keys are read at start-up and a restart-first deploy boots the old state.
A missing keys file warns instead of failing: a host without the keys still
deploys, degraded rather than broken. No plugin configuration XML is tracked.
2026-10-30 22:47:39 -07:00
dank 38737ab84e Merge pull request 'Media metadata view: Jellyfin writes, Plex reads, the array stays read-only' (#97) from feat/media-metadata-view into main 2026-10-30 22:18:27 -07:00
Frank 4437519430 chore(media): retire the legacy Jellyfin block from dankserver/compose.yaml
The container has had one owner since the Quadlet cutover
(dankserver/jellyfin/quadlet/jellyfin.container). The block was a second owner,
and wrong on every path: jellyfin/jellyfin:latest instead of the pinned ghcr
image, /opt/jellyfin/* data roots that no longer exist, and a read-only
/mnt/entropy:/data bind that the metadata view has since replaced.

The planner no longer counts jellyfin as a compose service, so editing the
inventory cannot restart a media server; tests/deployment_scoping.bats pins that
contract for both jellyfin and plex.
2026-10-30 22:17:04 -07:00
Frank 3b29e173c2 refactor(media): mount the view inside /var/lib/media-meta instead of under /mnt
The merged door was /mnt/media, which read like a third volume and split the host
state across two trees. It is now /var/lib/media-meta/view, so the array layer,
the SSD upper layer, the kernel workdir, the reap trash and the merged tree are
one directory — and the mount table gains no new /mnt entry beside the array.

systemd names a mount unit after its ESCAPED path, so the repo copy keeps the
readable name media-meta-view.mount while `install` writes the host file as
var-lib-media\x2dmeta-view.mount; `plan` prints both and the page records them, so
nobody hunts for a unit that "does not exist". install also creates the mount
point itself, because systemd will not.

The seam now additionally pins that the unit's Where= is exactly the path both
containers bind — guard and bind cannot drift apart. The Quadlet Lifecycle
phantom-path example is corrected too: it named an invented /mnt/media/*, while
the real phantom is the legacy /opt/jellyfin/* that dankserver/compose.yaml still
carries for a service whose roots moved to /home/dank/Containers/jellyfin.
2026-10-30 22:05:31 -07:00
Frank 9706f5d2de docs(media): the view moves sidecars into Plex's reach, it does not enable its NFO agent 2026-10-30 21:42:30 -07:00
Frank 206dc3baf1 docs(media): one page for the metadata view, and the three pages it re-points
New owner page Dankserver/Media Metadata View.md: the shape, the two binds and
why Plex keeps the /mnt/entropy container path, the commands, the laws (SSD-only
upper layer, never mount over an unmounted array, never delete through the view
because a whiteout only hides the array's copy, reap after any relayout, inotify
does not cross the merge), the failure signatures, and the rollback.

Jellyfin.md: /data is the view, so Local metadata works; the
"Read-only file system : '/data/…/movie.nfo'" line that used to be expected noise
is now the signature of a missing view. Plex.md: same bind, one fact changed.
Media Relayout.md: two new laws for the metadata that does not follow a rename.
AGENTS.md: runbook-index row plus a summary row. Root index regenerated.
2026-10-30 21:41:56 -07:00
Frank f5a3badb56 feat(media): the relayout executor names the reap its moves leave behind
A relayout renames on the raw array; the sidecars the media servers wrote live in
the NVMe upper layer of the view and stay at the old coordinates, which is how a
ghost folder (artwork and an .nfo, no video) is born. apply and rollback now
print the follow-up — scripts/media-view.sh reap — at the moment the operator
needs it, and the existing apply seam asserts the hint is in the output.
2026-10-30 21:41:56 -07:00
Frank 3e4b4f24f7 feat(media): share one metadata set between Jellyfin and Plex over an SSD overlay
Both media servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo /
episode.nfo) only when it sits next to the media file, and the library lives on
the RAID6 array, deliberately mounted read-only for every container and slow to
seek for a thousand tiny XML reads. So neither server binds the array any more:
both bind /mnt/media, an OverlayFS merge of the array (lowerdir, never written)
with /var/lib/media-meta on the NVMe (upperdir, where every write lands).

Jellyfin binds the view rw at /data and is the fleet's only metadata writer;
Plex binds it ro at /mnt/entropy, the container path its own database already
recorded, so only the host side of that bind moved and its library paths keep
resolving.

scripts/media-view.sh plans, installs, verifies, reports and reaps the view. It
refuses to mount over an unmounted array, refuses an upper layer on rotational
storage, and refuses a non-root install. deploy-remote.sh raises the view before
it restarts either media server, both units carry
ConditionPathIsMountPoint=/mnt/media so a missing view skips the start instead of
letting a scan clear the library database, and the planner routes the installer
and dankserver/host/** to the jellyfin+plex pair.

tests/media_overlay.bats (11 seams) pins the resolved shape, both refusals, the
two binds and their guard, the deployer's ordering, the planner mapping, and the
reap behaviour against a fixture (ghosts listed, then moved to the SSD trash --
files that exist only above are sidecars, not ghosts, so only directories are
candidates).
2026-10-30 21:41:51 -07:00
Frank 86f5903e74 docs(media): how to file a below-threshold episode by hand
Six single-episode files stayed in the movie library when the threshold refused
them; the hand path is the same executor on a hand-written plan and the same
journal. The trap recorded here is the series year -- never inherit it from the
title whose bonus folder an episode happens to be parked in.
2026-10-30 20:45:24 -07:00
Frank bd4ee78811 docs(media): clear the two OKF violations the last push shipped
The seam ran through `grep -c`, which returned success on a non-zero count and
let a red suite through: description over the 120-char cap, and a timestamp
that check-okf reads against the local date, so a UTC stamp a day ahead of
Seattle is 'in the future'. index.md regenerated for the new description.
docs_completeness 6/6, exit status checked this time.
2026-10-30 20:40:27 -07:00
Frank b7b39cdc92 docs(media): Jellyfin 12.4 needs the MediaBrowser header; the scan key is on the box
X-Emby-Token is refused for every key on 12.4 -- which is why this page
carried the false claim that no credential on the box could trigger a scan and
that the two Seerr keys were dead. The documented snippet now authenticates
with Authorization: MediaBrowser Token=..., reads 204 from /Library/Refresh,
and points at ~/.config/secrets/jellyfin (mirrored as JELLYFIN_API_KEY in the
ignored dankserver/.env). Law: prove a 401 against the ApiKeys table before
blaming the key.
2026-10-30 20:39:51 -07:00
Frank 09532e5216 docs(media): a deletion is staged on the item's own filesystem
Trashing 31 GiB of duplicate Monk to ~/.AGENTTRASH would have been a
cross-device copy of the thing being deleted, into 187 GiB of free space, on a
volume at 100% -- and rename(2) refuses it regardless. The stage belongs beside
the item.
2026-10-30 20:37:27 -07:00
Frank e1786e7cf2 docs(media): the Monk duplicate is verified; compare libraries with the series engine
The destination-hold refusal's evidence, stated exactly: 124 episodes both
sides, same episode list, every byte count identical, 31.5 GiB each way,
head+tail blake2b identical on a 6-episode sample.

Plus the law this came out of: a path-substring filter for a series name also
returns other shows (monk matches Monkey Dust), and whichever file a walk
reaches first standing in for an episode turns two libraries into a fake
duplicate — the same sloppiness aimed the other way deletes a unique file.
ep_of on both sides, and a hash before anything is trashed.
2026-10-30 20:33:41 -07:00
dank 409a8d8b0c Merge pull request 'feat(media): a TV mover that will not file a show the library already has' (#96) from feat/media-tv-mover into main 2026-10-30 20:31:38 -07:00
Frank e83162a2ac feat(media): a TV mover that will not file a show the library already has
Two fixes found by one live sweep, shipped together because the second is
what makes the first finishable.

1. scripts/media-tv.py — the other half of media-plan.py's refusal to file
   TV. Groups episode-shaped files by the series name before the marker
   (media-titles.ep_of, imported not restated), writes a plan in the
   executor's own format: <library>/<Series> (<year>)/Season <NN>/<own
   name>.<ext>. Refuses a group below --min, an episode naming no season,
   and an episode the destination library already holds at the same byte
   count — the movie share carries a second copy of all 124 Monk episodes
   the TV library has had for years; a duplicate is Frank's call, never a
   tool's copy. Applied by media-apply.py with its own journal and
   --root /mnt/entropy.

2. scripts/media-apply.py — gone meant "rmdir intended", so a folder whose
   rmdir failed was blacklisted forever and no later sweep could finish it
   (live movie sweep: 4517 targeted, 146 permanently stranded). gone now
   means "actually gone": a failed intent is retried by the next sweep.

tests: 9 new tests in media_relayout.bats, each red against the previous
revision — 8 TV + 1 sweep retry behind a chmod 500 parent. ./tests/run.sh
633 green. No live move run yet.
2026-10-30 20:30:24 -07:00
dank 5cd9a0c4c2 Merge pull request 'fix(media): the series is the name before the marker; a sweep carries its own empty subfolders' (#95) from fix/media-series-key into main 2026-10-30 20:20:24 -07:00
Frank a803aa046a fix(media): a sweep carries the emptied folder's own empty subfolders
The live sweep removed 2271 husks and left 146 standing, every one a release
folder blocked by a junk subfolder (Screens/, Sample/, a BDMV leaf) that no
file ever moved out of. Eligibility was per-folder, so the blocker was never a
target and its parent could never be removed.

Proof now carries downward: inside a folder the journal shows emptied, every
empty directory qualifies. Nothing under such a folder can hold data, and every
removal is still journalled as rmdir and undone by rollback.

The new test is red against the previous revision, green now. The live sweep
touched no file: 8193 before, 8193 after.
2026-10-30 20:18:54 -07:00
Frank 73f77a13c3 fix(media): a series is the name before the episode marker
Keyed on the whole episode title, every episode of a box set became a group of
one and never reached the five-episode size that calls something a series, so
the live run filed 124 Monk episodes as 124 movie folders, each named after its
own episode. ep_of now returns the series — the name before SxxEyy / 1x04 /
Part N — and the planner groups and names the quarantine pile by that series,
so one show is one pile: 'Monk - S05E03 - Mr. Monk and the Big Game' is series
monk. The threshold keeps its job: a lone film named Part 2 stays a film.

Re-planning the live census with this moves nothing differently except the 124
Monk files, which now join the TV pile: 168 episode files across 3 series.
2026-10-30 20:12:05 -07:00
dank dd76fcf5d4 Merge pull request 'feat(media): sweep removes the folders a completed run emptied' (#94) from feat/media-sweep into main 2026-10-30 20:04:24 -07:00
Frank 1cb28095a7 feat(media): sweep removes the folders a completed run emptied
An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
2026-10-30 20:03:16 -07:00
dank e1760e3316 Merge pull request 'feat(media): the movie-library relayout toolchain — census, plan, executor' (#93) from feat/media-relayout into main 2026-10-30 19:38:03 -07:00
dank 272e06daf6 feat(media): the movie-library relayout toolchain — census, plan, executor
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
2026-10-30 19:33:44 -07:00
dank 4db985b90d Merge pull request 'fix(qwen38): the preflight asks whether the pin is ONE of an image's digests' (#92) from fix/quant-pipeline-hardening into main 2026-10-30 16:30:07 -07:00
Frank 44d72b9e78 fix(qwen38): the preflight asks whether the pin is ONE of an image's digests
A by-digest pull leaves the image carrying every digest it was pulled under, so
reading RepoDigests[0] reported a healthy pinned image as wrong and refused a
window that was ready to run. Membership, not position. Also records the measured
step optimum: 106 tok/s at an 8,192-token step against 76 at 16,384 on a 32,768
calibration window, so the wide step has a ceiling, not a plateau.
2026-10-30 16:30:03 -07:00
dank 8073d0aa9d Merge pull request 'fix(qwen38): one identity preflight owns every fact a calibration window needs' (#91) from fix/quant-pipeline-hardening into main
Merge pull request #91
2026-10-30 16:26:41 -07:00
Frank 5c678b804c fix(qwen38): one identity preflight owns every fact a calibration window needs
Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid

scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
2026-10-30 16:26:07 -07:00
dank 81e0995f7d Merge pull request 'fix(qwen38): the calibration window must prove the rig owns the socket' (#90) from fix/quant-pipeline-hardening into main 2026-10-30 16:08:29 -07:00
Frank 167e6d6de3 fix(qwen38): the calibration window must prove the rig owns the socket
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.

The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.

Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
2026-10-30 15:54:57 -07:00
dank 608b0058e3 Merge pull request 'campaign(qwen38): the calibration floor is 12M tokens' (#89) from fix/quant-pipeline-hardening into main 2026-10-30 15:47:25 -07:00
Frank 4043798053 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:47:15 -07:00
Frank 521537a34c Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:11 -07:00
Frank 19fdffe8bd docs(agents): a start-only quadlet handler no-ops an image bump
The law today's authentik bump paid for: start on an active unit is a no-op, so
the new digest installs and the old container keeps serving behind three green
gates. Restart applies a pin.
2026-10-30 15:34:10 -07:00
Frank 54a4a4180a Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:03 -07:00
Frank f7aec7f154 fix(authentik): restart the units so an image bump actually applies
The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
2026-10-30 15:34:02 -07:00
Frank 299078eaba campaign(qwen38): the calibration floor is 12M tokens
Frank's rule: the served iq4r this arm competes with was calibrated on a
10M-token set, so the arm is calibrated at least as deeply. The floor was
lowered to 1.5M on the vendor's '10M beats 1M by under 1%' note; that
measures the vendor's own plate, not ours — the arm is judged against a
10M-calibrated reference, so cutting the calibration 6x to save wall clock
buys a worse position on the only comparison that matters.

One round of the re-chunked corpus is ~9.31M tokens, so a 12M floor seals
in round 1 plus part of round 2. A round that seals nothing is normal (the
floor is pooled) and round_verdict treats it as continue, never failure.

Also fixes tests/qwen38_campaign.bats' window dry run: its fixture lands on
/tmp (a 16 GiB tmpfs) where the 60 GiB seal preflight can never pass, so the
test now uses the documented Q38_CALIB_MIN_FREE_MIB override instead of
asserting environment luck.
2026-10-30 15:33:52 -07:00
dank 7b2b5b66fe Merge pull request 'fix(qwen38): the calibration tap runs on latest radiance, not the production pin' (#87) from fix/quant-pipeline-hardening into main 2026-10-30 15:30:21 -07:00
Frank e1d4d59773 docs(dragoon,zealot): record both runtime pins as they now run
frank-dragoon: engine 0.18.20 (image e35133b2…), the 18-model NPU catalog, the
re-cut qwen3-embedding-0.6b device programs, and the wire surface the new engine
actually answers (GET /tokenize is 405, /props is 200, /completion and /ping
stay 404). Eras gains the 2026-10-30 row.

frank-zealot: fork build 12118 (0.7.0-dev, 024474448) with the payload-load
evidence (202 pq2_0 tensors, 202 Hadamard-folded weights), plus the law a
rebuild pays for: cmake --install strips the rpath and installs bin/+lib/, so
the configure needs CMAKE_INSTALL_RPATH and LLAMA_BIN names …/bin/llama-server.
2026-10-30 15:27:21 -07:00
Frank 5787c86954 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:27:21 -07:00
Frank acbb539b97 feat(frank-zealot): PrismML fork b10777 -> build 12118 (0.7.0-dev, 024474448)
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.

LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.

The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.

Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
2026-10-30 15:25:24 -07:00
Frank 49a83ef1d5 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:25:24 -07:00
Frank d012654fc9 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:23:06 -07:00
dank b45771e87e Merge pull request 'docs(radiance): reference the engine — options, plugins, conversion, tools, operations' (#88) from docs/radiance-engine-reference into main 2026-10-30 15:17:58 -07:00
Frank 011ceeb344 fix(npu fetch): never resume into a file from another revision
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
2026-10-30 15:13:42 -07:00
Frank 88a9bc02be Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:42 -07:00
Frank 6ac544a7b6 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:23 -07:00
Frank 33396d2daa feat(frank-dragoon): halogen-flash-server 0.16.2 -> 0.18.20, NPU lane re-pinned
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.

0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
2026-10-30 15:13:19 -07:00
Frank bdc0613739 docs(radiance): correct the tokeniser corpus name and statsdump auth note 2026-10-30 15:03:31 -07:00
Frank 3e32cb9cd1 campaign(qwen38): the tap producer is latest radiance, at a vendor-backed floor
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.

The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).

Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.

The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
2026-10-30 15:02:58 -07:00
Frank 6e80cd44f0 Merge remote-tracking branch 'origin/main' into wt/20261030-143311 2026-10-30 15:01:40 -07:00
Frank 3a04e34414 docs(radiance): reference the engine's options, plugins and conversion
Nine pages under LLM/ document the upstream radiance engine at 8da043c:
- engine shape, plugin model, RAD2 container, ABI and environment
- every serve flag grouped with defaults and the law behind each knob
- the HTTP surface, request fields and refusal classes
- rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring
- the (arch id, quant) family table and the recipe of record per model
- kernel/quantiser plugin authoring, encodings, build and checks
- architecture plugin authoring: declare, step, probe, lane and media rules
- rad-info/radbench/radtune and the shell harnesses
- the container contract, host prerequisites, readiness and restart triage

Also completes the dangling Related line on the frank-archon endpoint page.
2026-10-30 15:01:37 -07:00
Frank 9085a80bd1 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:01:07 -07:00
Frank 58184008dd chore(images): refresh every registry pin to current upstream digest
Container images re-resolved against their registries today; each pin is the
manifest-list digest the tag answers now, so no container is served from a
build that upstream has already replaced.

Version-tag moves: authentik 2026.8.3 -> 2026.8.5 (server/worker/outpost and
the ldap image, which now carries the tag instead of a bare digest so the pair
can never drift apart), seerr v3.4.1 -> v3.5.1, calibre 9.15.0 -> 9.17.0,
calibre-web 0.6.27-ls402 -> ls407, plex ls329 -> ls330, tautulli
v2.18.1-ls245 -> v2.18.2-ls250.

Digest-only moves inside the same stream: postgres 16-alpine, mariadb 11,
redis 7-alpine, valkey 9-alpine, grafana, alertmanager, gluetun, searxng,
open-webui, jellyfin, romm, chromedp/headless-shell, lightpanda/browser.

Rollback is the previous digest on the same Image= line.
2026-10-30 15:00:52 -07:00
Frank b4a1e6069e fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.

Four gates, each with the measurement that motivated it:

- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
  @sha256:665775ed…), overridable only by naming --image deliberately. Proven by
  the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
  One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
  filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
  starts the service. `podman rm -f` does not kill the engine: an orphan made
  production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
  restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
  target configurable, so the tap lives on /mnt/entropy instead of the root fs.
2026-10-30 14:18:03 -07:00
dank 9f997f0841 Merge pull request 'perf(quant): the calibration boot's step and slots are its own knobs — probe at the measured 71 tok/s' (#86) from fix/qwen38-calib-shape into main 2026-10-29 16:50:50 -07:00
Frank f6298e3501 perf(quant): the calibration boot's step and slots are its own knobs (probe: 71 tok/s, 47 h tap)
Measured from the engine's own counters on the 2026-10-29 drain: prefill 71 tok/s steady state, a
180 s window showing 7 prefill steps of the capped 2,048 tokens and 3 preemptions. One 7,168-token
document is therefore ~100 s, round 1 (2,367 docs, 10.03M tokens) ~39 h, and the 12M-position
floor ~47 h of drained production — not the ~18 h the brief assumed.

The tap is the only phase whose cost is read volume per document: at the inherited 2,048-token step
a document is four steps, and each step re-streams every layer's routed expert union from the
container (43.4 GiB a rank live on disk). One wide step reads each layer's union once.

The 'one step, one slot count' audit law binds kldref and score because their KL numbers are only
comparable at an identical shape. Calib produces layer Hessians and no number any report compares,
so Q38_CALIB_MNBT / Q38_CALIB_SEQS are its own, defaulting to the inherited value — nothing moves
unless a probe asks. The KL modes ignore both.

Tests: 589/0 — the knob reaches calib's argv, does not reach kldref's, and a non-numeric value is
refused by name.
2026-10-29 16:50:50 -07:00
dank f14efdce1a Merge pull request 'fix(quant): a flat calibration round is not a failure — the tap's floor is pooled' (#85) from fix/qwen38-flat-round into main 2026-10-29 16:17:52 -07:00
Frank a6fb0209e8 fix(quant): a flat calibration round is not a failure — the tap's floor is pooled
RADIANCE_CALIB_TOKENS=12000000 is a pooled floor and calib-7168.jsonl carries 10.03M tokens, so
round 1 of a corpus push seals nothing by construction. The gate PR #84 added failed exactly
there, one round too early, while the drain was already running.

Now round_verdict() separates the cases: 'shrink' (the tap lost bytes) fails the run, 'flat' says
so and continues, 'grew' continues, 'done' stops. A run that never sees the engine's completion
line still exits non-zero on max rounds, and the rig marker remains the lock that stops a push
into anything that is not the rig.

Test: round_verdict's four verdicts, behaviorally.
2026-10-29 16:17:52 -07:00
dank 92a412e99d Merge pull request 'fix(quant): a tap is only a tap when its rig is running — marker gate, gated calib window, tail-and-slices gate' (#84) from fix/qwen38-tap-marker into main 2026-10-29 16:14:34 -07:00
Frank c238e32d6f fix(quant): a tap is only a tap when its rig is running
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.

- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
  GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
  kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
  non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
  and that container is running. With the driver's existing tap-progress gate, a push can no
  longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
  instead of falling through the health wait, bank the 336 GiB reference container into trash
  only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
  172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
  read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
  section that names the live campaign instead of the retired swift15 era.

Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
2026-10-29 16:14:16 -07:00
dank da5eb968f9 Merge pull request 'fix(quant): harden the campaign pipeline, then refactor its corpus tools onto one module' (#83) from fix/quant-pipeline-hardening into main 2026-10-29 15:29:33 -07:00
dank b3052e9243 Merge pull request 'fix(frank-archon): fold the pinned host pool back to 49,152; keep the 2,048 step' (#82) from feat/qwen38-w5nl64 into main 2026-10-29 15:29:29 -07:00
Frank 3e5a37a21f refactor(quant): one corpus plumbing module, and the builders out of main()
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.

Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
2026-10-29 15:26:08 -07:00
Frank 1e4fa14772 fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.
2026-10-29 15:14:43 -07:00
Frank e6fba5d97e fix(frank-archon): fold the pinned host pool back to 49,152; keep the 2,048 step
PR #81's pool half measured inert the day it deployed: the 66,560 boot printed the
same elastic 26.73 GiB / experts 20.72 / kv cache 5.64 / vram_weights 24.30 as the
49,152 boot and opened the pool with 0 B used, and the qwen38-w5nl64 rig moved
vram_kv 269.99 -> 271.25 MiB (window still 7,436 tokens: its 95.37 GiB n-gram table
claim decides, not the pool). The step half stays. Tests: 576 ok / 0 fail.
2026-10-29 15:10:33 -07:00
Frank ca1475543d docs(frank-archon): the 2026-10-29 boot ledger - the 66,560 pool gives the same cache as 49,152
The container opened the raised pool with 0 B used and printed the same elastic 26.73 GiB / experts 20.72 GiB / kv cache 5.64 GiB / vram_weights 24.30 GiB as the 49,152 boot, so the era's superseded ledger line is destroyed and the evidence table gains the ceiling-case row beside the 45,568 one it already carried. Tests: ./tests/run.sh 576 ok / 0 fail.
2026-10-29 13:48:34 -07:00
dank f83c241668 Merge pull request 'feat(frank-archon): raise the pinned host pool to 66,560; the campaign inherits MNBT 2,048' (#81) from feat/frank-archon-pool-step into main 2026-10-29 13:14:27 -07:00
dank b4d7f6e7fe Merge pull request 'fix(qwen38): calib rides the shared step; the wide one cost the container its cache' (#80) from feat/qwen38-w5nl64 into main 2026-10-29 11:18:36 -07:00
91 changed files with 12984 additions and 458 deletions

No files matched your search

+19 -8
View File
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
2. Bench against the service itself. No scratch ports on an LLM host — one
process owns the card. Use the runtime's own harness (`llama-bench`; a
live llama.cpp server also reports honest `timings` per request, including
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
`ignore_eos` run on the raw `/completion` path there answers **500 "The
model produced output that does not match the expected Content-only
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -439,16 +439,16 @@ disagrees with it is the bug.
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
loader (the payload carries the fork's custom PQ2_0 tensor type plus
`scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests) + the rank-1
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
— and build 10777 is also the gate that lets it load: older forks refuse it
with `unsupported prism.hadamard.version: 2`.
— and the fork BUILD is also the gate that lets it load: anything older than
b10777 refuses it with `unsupported prism.hadamard.version: 2`.
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -741,6 +741,13 @@ linked runbooks.
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
says identity mode costs nothing, so the XDNA2 NPU returned for free.
[[dankclaw-iommu-translation-off]]
- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
`systemctl --user start` on an already-active unit does nothing, so installing
a new digest under a running service keeps the OLD container serving while the
deploy, its health gate and the Playwright suite all pass — the authentik
handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
outpost, the one handler that restarted). A pin change is applied by
`restart`, never `start`.
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -800,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -841,5 +850,7 @@ history.
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
@@ -23,7 +23,7 @@ Network=dankclaw_default
# Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
# the tag's nested manifest list, so pin the single-platform digest).
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
Environment=AUTHENTIK_INSECURE=false
Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}
@@ -20,7 +20,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-outpost
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=authentik-postgres
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
Environment=POSTGRES_DB=authentik
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
[Container]
ContainerName=authentik-server
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=server
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -16,7 +16,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-worker
Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -15,7 +15,7 @@ After=network.target
[Container]
ContainerName=gitea-postgres
Network=gitea.network
Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
EnvironmentFile=/home/dank/Containers/gitea/env
Environment=POSTGRES_USER=gitea
Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
[Container]
ContainerName=chrome-devtools-protocol
Network=host
Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
Entrypoint=/headless-shell/headless-shell
Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
ShmSize=1073741824
@@ -23,7 +23,7 @@ After=gluetun.service network.target
[Container]
ContainerName=marreta_browser
Network=gluetun.container
Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
[Service]
TimeoutStartSec=300
@@ -33,7 +33,7 @@ Wants=network-online.target
[Container]
ContainerName=alertmanager
Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
Network=dankclaw_default
User=0:0
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
[Container]
ContainerName=grafana
Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
Network=dankclaw_default
User=0:0
PublishPort=127.0.0.1:3002:3000
@@ -27,7 +27,7 @@ After=network.target
[Container]
ContainerName=open-webui
Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
Environment=ANONYMIZED_TELEMETRY=False
+1 -1
View File
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
[Container]
ContainerName=gluetun
Network=searxng.network
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
AddCapability=NET_ADMIN
AddDevice=/dev/net/tun:/dev/net/tun
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
ContainerName=searxng-valkey
Network=searxng.network
NetworkAlias=valkey
Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
Volume=/opt/searxng/valkey:/data
[Service]
+1 -1
View File
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
[Container]
ContainerName=searxng
Network=gluetun.container
Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
Volume=/opt/searxng/config:/etc/searxng
+1 -1
View File
@@ -29,7 +29,7 @@ Wants=network-online.target
[Container]
ContainerName=seerr
Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
Network=dankclaw_default
PublishPort=127.0.0.1:5055:5055
Volume=/home/dank/Containers/seerr/data:/app/config
@@ -43,7 +43,7 @@ After=network.target
[Container]
ContainerName=calibre-server
Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
Network=pasta
EnvironmentFile=/home/dank/Containers/calibre-server/env
Environment=PUID=1000
@@ -16,7 +16,7 @@ After=network.target
[Container]
ContainerName=calibre-web
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
Network=pasta
Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
Environment=LANG=en_US.UTF-8
+6 -28
View File
@@ -94,34 +94,12 @@ services:
timeout: 10s
retries: 3
# ═══════════════════════════════════════════════
# Jellyfin — media server (movies, TV, music)
# ═══════════════════════════════════════════════
jellyfin:
image: jellyfin/jellyfin:latest
container_name: jellyfin
restart: unless-stopped
ports:
- "8096:8096/tcp"
- "8920:8920/tcp"
- "7359:7359/udp"
- "1900:1900/udp"
environment:
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
volumes:
- /opt/jellyfin/config:/config
- /opt/jellyfin/cache:/cache
- /mnt/entropy:/data:ro
devices:
- /dev/dri:/dev/dri
group_add:
- "992"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
interval: 30s
timeout: 10s
retries: 3
# Jellyfin left this file when it became a Quadlet: the container is owned by
# dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
# /home/dank/Containers/jellyfin, and its library arrives through the SSD
# overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
# was here named /opt/jellyfin/* and a read-only array bind — a second owner for
# the container name `jellyfin`, wrong on every path. One owner per container.
# ═══════════════════════════════════════════════
# Calibre-Web — e-book web frontend
@@ -0,0 +1,30 @@
# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
#
# Installed by `scripts/media-view.sh install` — the repo copy is the source of
# truth (fleet law: never edit host state by hand).
#
# An OverlayFS merge that shows the media library as one tree while splitting
# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
# written), upperdir is a directory on the NVMe that receives every write. The
# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
# sidecars and artwork are SSD bytes — which is what makes them fast to browse
# and what keeps Plex's NFO agent off the spinners — while the media itself stays
# on the array untouched.
#
# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
# empty directory and the view would present an empty library as truth.
[Unit]
Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
RequiresMountsFor=/mnt/entropy
[Mount]
What=overlay
Where=/var/lib/media-meta/view
Type=overlay
Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
[Install]
WantedBy=local-fs.target
+10 -4
View File
@@ -4,8 +4,9 @@
# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
#
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
# anywhere writes to the entropy volume. Every
# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
# while the sidecars this server writes land on the NVMe. Every
# creation-time environment variable of the compose-era container is kept as
# an Environment= line (zero drift); none of them are secrets, so this unit
# needs no EnvironmentFile.
@@ -16,7 +17,7 @@ After=network.target
[Container]
ContainerName=jellyfin
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
Network=dankserver_appnet
Environment=DEBIAN_FRONTEND=noninteractive
Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
PublishPort=1900:1900/udp
Volume=/home/dank/Containers/jellyfin/config:/config
Volume=/home/dank/Containers/jellyfin/cache:/cache
Volume=/mnt/entropy:/data:ro
# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
# fail-closed guard — starting against a plain, unmounted view directory would let
# a scan conclude the library is empty.
Volume=/var/lib/media-meta/view:/data
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
AddDevice=/dev/dri
GroupAdd=keep-groups
HealthCmd=curl -f http://localhost:8096/health
+7 -2
View File
@@ -4,14 +4,19 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
ContainerName=plex
Network=dankserver_appnet:alias=plex
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
Volume=/var/lib/plexmediaserver:/config
Volume=/mnt/entropy:/mnt/entropy:ro
# The container path stays /mnt/entropy on purpose: every path Plex recorded in
# its own database is written against it, so swapping the HOST source of this bind
# to the overlay view re-points Plex at the sidecars without touching its DB.
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
Volume=/var/lib/media-meta/view:/mnt/entropy:ro
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
PublishPort=32400:32400
[Service]
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=romm-mariadb
Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
Network=dankserver_appnet:alias=mariadb
EnvironmentFile=/home/dank/Containers/romm/env
Environment=MARIADB_DATABASE=romm
+1 -1
View File
@@ -12,7 +12,7 @@ After=network.target
[Container]
ContainerName=romm-redis
Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
Network=dankserver_appnet:alias=redis
Volume=/home/dank/Containers/romm/redis:/data
HealthCmd=redis-cli ping
+1 -1
View File
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
[Container]
ContainerName=romm
Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
Network=dankserver_appnet
EnvironmentFile=/home/dank/Containers/romm/env
Environment=DB_HOST=mariadb
@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
ContainerName=tautulli
Network=dankserver_appnet:alias=tautulli
Environment=PUID=0
+149 -18
View File
@@ -22,6 +22,17 @@ Mechanics notes:
prompts do not bias the Hessians (the corpus is drawn round-robin over
source classes, so loss is class-neutral as long as failures are rare;
a systematic 4xx is loud in the log and must be fixed, not ignored).
- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
the run, and a run that never sees the engine's completion line exits non-zero.
The first version failed on a flat round, which aborted exactly one round too
early (2026-10-29).
- before any prompt, the tap must carry the rig's own marker
(`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
container, and that container must be running. Measured 2026-10-29:
with neither check, four rounds of 10M tokens went into restored
production after the rig's boot had died, and recorded nothing.
Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
[--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
import argparse
import concurrent.futures
import json
import os
import subprocess
import sys
import time
import urllib.request
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import qwen38_common as qc # noqa: E402 - sibling module, same directory
DONE_MARK = "calibration wrote"
@@ -49,9 +67,9 @@ def tap_done(container):
return DONE_MARK in (out.stdout + out.stderr)
def push_one(url, prompt):
def push_one(url, prompt, model):
body = json.dumps({
"model": "frank-archon",
"model": model,
"prompt": prompt,
"max_tokens": 1,
"temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
return False
def main():
def push_round(url, prompts, model, parallel):
"""Push the whole corpus once; returns how many requests were accepted."""
ok = 0
with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
for good in ex.map(lambda p: push_one(url, p, model), prompts):
ok += 1 if good else 0
return ok
def tap_bytes(path):
"""Bytes the tap has sealed so far; 0 when the directory is absent.
A tap whose directory is wrong, full or read-only stops writing while the
engine keeps answering 200 — the driver then re-pushes the whole corpus for
every round it is allowed and the loss is invisible until the hours are gone.
"""
total = 0
try:
for root, _, files in os.walk(path):
for f in files:
try:
total += os.path.getsize(os.path.join(root, f))
except OSError:
pass
except OSError:
return 0
return total
def round_verdict(before, after, done):
"""What one round's tap bytes mean.
'done' the engine logged that it sealed the tap — the run is over
'shrink' the tap LOST bytes: reset, pruned or replaced under us
'grew' sealed bytes this round
'flat' sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
pooled floor, and a round that does not reach it has nothing to
write yet (the campaign's corpus is 10.03M tokens against a 12M
floor, so round 1 is always flat). Failing on it aborted the run
exactly one round too early.
"""
if done:
return "done"
if after < before:
return "shrink"
if after == before:
return "flat"
return "grew"
def require_rig(a):
"""Refuse unless the rig that opened this tap is up and serving right now.
Three facts, in order: the runner wrote its marker into the tap dir, the
marker names the container this pusher targets, and that container is
running. A marker without a live container is a boot that died; a live
container without a marker is some other engine on the port — on 2026-10-29
it was restored production, which took four rounds of 10M tokens before the
old driver admitted it had recorded nothing.
"""
marker = os.path.join(a.tap_dir, a.tap_marker)
if not os.path.isfile(marker):
qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
"run-qwen38-bf16.sh calib, so a push here would feed whatever "
"answers the port and record nothing")
try:
lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
except OSError as exc:
qc.refuse(f"cannot read the rig marker {marker}: {exc}")
meta = dict(line.split("=", 1) for line in lines if "=" in line)
who = meta.get("container", "")
if who != a.container:
qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
f"{a.container!r} — wrong tap, or a stale marker")
try:
p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
capture_output=True, text=True, timeout=30)
except Exception as exc:
qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
if p.returncode != 0 or p.stdout.strip() != "true":
qc.refuse(f"the rig container {a.container} is not running — whatever is "
"answering the port, it is not the calibration rig")
def parse_args():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=7770)
ap.add_argument("--parallel", type=int, default=64)
ap.add_argument("--container", default="arbench")
ap.add_argument("--model", default="frank-archon")
ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
ap.add_argument("--tap-marker", default=".rig-boot",
help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
ap.add_argument("--max-rounds", type=int, default=4)
a = ap.parse_args()
url = f"http://127.0.0.1:{a.port}/v1/completions"
return ap.parse_args()
def load_prompts(a) -> list[str]:
if not os.path.isfile(a.corpus):
qc.refuse(f"no corpus at {a.corpus}")
prompts = []
with open(a.corpus) as f:
for line in f:
line = line.strip()
if line:
prompts.append(json.loads(line)["prompt"])
print(f"corpus: {len(prompts)} prompts", flush=True)
for lineno, rec in qc.iter_jsonl(a.corpus):
if not isinstance(rec.get("prompt"), str):
qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
prompts.append(rec["prompt"])
if not prompts:
qc.refuse(f"the corpus at {a.corpus} holds no prompts")
return prompts
def main():
a = parse_args()
url = f"http://127.0.0.1:{a.port}/v1/completions"
prompts = load_prompts(a)
require_rig(a)
chars = sum(len(p) for p in prompts)
print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
flush=True)
for rnd in range(1, a.max_rounds + 1):
if tap_done(a.container):
break
before = tap_bytes(a.tap_dir)
t0 = time.time()
ok = 0
with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
for good in ex.map(lambda p: push_one(url, p), prompts):
ok += 1 if good else 0
dt = time.time() - t0
ok = push_round(url, prompts, a.model, a.parallel)
dt = max(time.time() - t0, 1e-9)
print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
flush=True)
f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
if ok == 0:
print("FAIL: not one prompt was accepted; fix the engine or the argv "
"before spending another round", flush=True)
sys.exit(1)
if tap_done(a.container):
break
after = tap_bytes(a.tap_dir)
verdict = round_verdict(before, after, False)
if verdict == "shrink":
print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
f"({before} B before, {after} B after) — it was reset or replaced "
"under us", flush=True)
sys.exit(1)
if verdict == "flat":
print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
f"{before} B) — continuing", flush=True)
else:
print("max rounds exhausted, tap not done", flush=True)
sys.exit(1)
+1 -1
View File
@@ -21,7 +21,7 @@
# deployed knob has since dropped to 262144.
set -euo pipefail
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
CUR=/home/dank/models/halogen-flashnext-v2
NEW=/home/dank/models/halogen-v2-abliterated
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
+18 -3
View File
@@ -47,7 +47,12 @@
set -u -o pipefail
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
# the same image the tap producer boots: a tap must be converted by the line that
# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
# it. Production stays on its own pin — this is the campaign's toolchain.
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
@@ -71,10 +76,12 @@ PLAN_ONLY=0
KEEP_RECIPE=0
while [ "$#" -gt 0 ]; do
case "$1" in
--arm) ARMS+=("$2"); shift 2 ;;
--arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
ARMS+=("$2"); shift 2 ;;
--dry-run) DRY=1; shift ;;
--plan-only) PLAN_ONLY=1; shift ;;
--target-count) TARGET_COUNT="$2"; shift 2 ;;
--target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
TARGET_COUNT="$2"; shift 2 ;;
--keep-recipe) KEEP_RECIPE=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
@@ -124,6 +131,13 @@ fi
for a in "${ARMS[@]}"; do
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
# A killed run leaves its container behind (--rm only fires when podman itself
# exits), and the next arm then dies on "name already in use" AFTER the plan
# phase; refuse here, where the fix is one `podman rm -f`.
podman container exists "qwen38-convert-$a" && {
echo "container qwen38-convert-$a still exists (a killed run?);" \
"'podman rm -f qwen38-convert-$a' and re-run" >&2
exit 2; }
done
# --- the protected plane, spliced into a per-run copy of the template --------------------------
@@ -203,6 +217,7 @@ for arm in "${ARMS[@]}"; do
rc=0; run_convert "$arm" || rc=$?
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
[ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
# count. An arm that came out materially smaller lost a plane somewhere.
+16 -1
View File
@@ -155,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
# Quadlet unit now — the last compose services were the hermes trio, whose units
# have owned their containers since 2026-09-12 and whose compose blocks were
# dead weight that could only create a duplicate container.
# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
# has owned the container since the cutover, so a compose edit must not restart it.
DANKSERVER_COMPOSE_SERVICES = {
"mariadb",
"redis",
"romm",
"copyparty",
"jellyfin",
"calibre-web",
}
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
@@ -421,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
"tautulli": {"tautulli"},
}
_add(services, *quadlet_services.get(parts[1], set()))
if len(parts) >= 2 and parts[1] == "host":
# dankserver/host/systemd/media-meta-view.mount is the overlay view the
# media servers bind (scripts/media-view.sh installs it on the host),
# so editing it restarts jellyfin and plex and nothing else.
_add(services, "jellyfin", "plex")
if len(parts) >= 2 and parts[1] == "observability":
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
@@ -466,6 +472,15 @@ def services_for_path(path: str, target: str) -> set[str]:
# that service (target_services then keeps it only on the host being
# planned). Editing the collector restarts the running exporter.
_add(services, "gpu-exporter")
if name == "media-view.sh":
# The installer is what creates, mounts and reaps the view both media
# servers read, so a change to it is a restart of that pair.
_add(services, "jellyfin", "plex")
if name == "jellyfin-providers.sh":
# The deployer runs this before jellyfin restarts, so a change to
# the installer is a change to the provider state Jellyfin loads at
# start-up: it selects jellyfin and nothing else.
_add(services, "jellyfin")
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-zealot only. The ik_llama.cpp
+46 -4
View File
@@ -1202,7 +1202,42 @@ deploy_romm_stack() {
systemctl --user restart "$service.service"
}
ensure_media_view() {
# Both media servers read their library through the SSD-backed overlay view
# (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
# array below stays read-only and every sidecar Jellyfin writes lands on the
# NVMe.
# Raise it BEFORE either container starts — a start against a bare view
# directory would let a scan clear the library database. The units also carry
# ConditionPathIsMountPoint, so a missing view skips the start rather than
# damaging anything. A relayout must be followed by `media-view.sh reap`
# (see the Media Relayout page), which this is not.
run_privileged bash scripts/media-view.sh install
bash scripts/media-view.sh verify
}
ensure_provider_keys() {
# Jellyfin's metadata-provider keys live in the container-owned plugin
# configuration under the data root, never in git; scripts/jellyfin-providers.sh
# is what places them, reading the per-service host env file (the fleet
# convention: grafana, caddy, pihole, copyparty, romm, seerr) that holds
# TMDB_API_KEY= and TVDB_API_KEY=, mode 600.
# Missing keys mean a keyless provider — a degraded library, not a broken
# deploy — so this warns rather than failing the restart. But when the keys
# ARE placed it must happen before the restart: the plugin and its keys are
# read at start-up, so a restart first would boot the old state.
local keys=/home/dank/Containers/jellyfin/provider-keys
if [ -f "$keys" ]; then
bash scripts/jellyfin-providers.sh install
else
echo "WARN: $keys is absent - provider keys NOT placed"
echo "WARN: create it (TMDB_API_KEY=, TVDB_API_KEY=, chmod 600) and re-deploy jellyfin"
fi
}
deploy_jellyfin() {
ensure_media_view
ensure_provider_keys
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload
if ! systemctl --user is-active -q jellyfin.service; then
@@ -1290,6 +1325,7 @@ deploy_copyparty() {
}
deploy_plex() {
ensure_media_view
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload
if ! systemctl --user is-active -q plex.service; then
@@ -1362,11 +1398,17 @@ deploy_authentik() {
# generated". Boot persistence comes from [Install] WantedBy=default.target
# inside each .container file, which the generator honors at user-manager
# start (linger must stay on for the deploy user).
systemctl --user start authentik-postgres.service
# RESTART, not start: `start` on an already-active unit is a no-op, so an
# image bump would install the new digest and keep serving the old
# container while every gate stayed green (measured 2026-10-30: server and
# worker stayed on the previous release beside an outpost that moved,
# because only the outpost handler restarts). The generator has already
# re-rendered the unit from the .container, so a restart is what applies it.
systemctl --user restart authentik-postgres.service
wait_container_healthy authentik-postgres 120 || return 1
systemctl --user start authentik-server.service
systemctl --user restart authentik-server.service
wait_container_healthy authentik-server 300 || return 1
systemctl --user start authentik-worker.service
systemctl --user restart authentik-worker.service
}
deploy_authentik_outpost() {
@@ -1482,7 +1524,7 @@ deploy_anubis() {
# artifact would leave a restart-looping unit and a dark endpoint. The
# port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released.
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
# The payload dir is its own twin because the NPU lane's containment gate below
# reads it; every artifact path under it stays a full literal as usual.
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
+33 -16
View File
@@ -8,8 +8,10 @@
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
# whose files do not match.
#
# The manifest below IS that catalog block (dumped from the pinned 0.16.2
# image on 2026-10-19), so the files land in the layout the entrypoint
# The manifest below IS that catalog block (dumped from the pinned 0.18.20
# image on 2026-10-30; the 0.16.2 block this replaced named different device
# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
@@ -57,21 +59,25 @@ manifest() {
case "$1" in
qwen3-embedding-0.6b)
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
FILES=(
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
"devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
"devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
"devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
"devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
"devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
"devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
"devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
"devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
"devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
"devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
"devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
"devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
"devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
"devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
"devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
"devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
"devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
"devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
"devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
"devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
"devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
"devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
"devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
"devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
"devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
"devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
"devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
"devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
) ;;
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
# 62 GiB fetch hit this on 2026-10-18).
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
# Resume ONLY into a genuinely partial file. A staged file of a
# DIFFERENT size is an artifact of another revision (a version
# bump re-cuts the device programs), and `-C -` against it
# appends the tail to the old bytes: the result is the right
# BYTE COUNT with the wrong content, which reads as a corrupt
# program, not as a fetch fault (measured 2026-10-30 on
# consts.hnpb). Fetch those from zero.
resume=()
if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
resume=(-C -)
fi
curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+12 -10
View File
@@ -74,12 +74,14 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
# x8 link when the router picks it. The ledger's coverage line is the referee, not
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
# 2026-10-29: 49,152 -> 66,560 (tp x 65 GiB, inside the 246 GiB the box has). The
# campaign rig's bf16 container at the shared step reported "kv cache 269.99 MiB a
# card: 7436 tokens", so 682 of the 1,294 calibration documents were refused at that
# pool; the pool, not the step, is what widens it (scripts/run-qwen38-bf16.sh). The
# new cache line belongs to the next boot ledger, not to this comment.
FRANK_ARCHON_HOST_POOL_MIB=66560
# 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
# endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
# vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
# the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
# step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
# by this pool, so the refused calibration documents were fixed by cutting them to the
# window (scripts/rechunk-kld-corpus.py), not by widening the pool.
FRANK_ARCHON_HOST_POOL_MIB=49152
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
# floor is built from it.
FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -142,10 +144,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
# itself before refusing by name, so recovery is a boot, not a visit.
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
# at the 66,560 pool with 4 slots that computes to 196,664 MiB (161,848 at the old
# 49,152 pool).
FRANK_ARCHON_TTM_MIN_MIB=197000
FRANK_ARCHON_TTM_TARGET_MIB=224000
# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
# 66,560 pool the 2026-10-29 A/B rejected).
FRANK_ARCHON_TTM_MIN_MIB=162000
FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
# this unquoted on purpose, so keep multi-flag values as one string.
FRANK_ARCHON_EXTRA_ARGS=
+5 -2
View File
@@ -10,8 +10,11 @@
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
# engine moves with the payload, never ahead of it.
# Image digest-pinned (a tag pull silently re-points; the digest is what was
# qualified — 0.16.2 is the version the model card names as its quickstart pin).
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
# pin) — the vendor has moved two minor versions and the payload lock is what
# the deploy's health gate plus the boot ledger decide; rollback is the
# previous digest, 0c61bf84…, on this same line.
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
# Container identity kept from the 27B era: the unit's ExecStop names it.
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
# Predecessor identity the serve script clears if it is still holding the GPUs.
+5 -4
View File
@@ -1,12 +1,12 @@
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
# the refusal-direction adapter applied at load.
#
# ONE OWNER: shape, payload identity, eras, laws and every measured number
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
# knobs and only the reason a reader editing THIS line cannot recover
# elsewhere. Where the two disagree, this file wins on values.
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
# metadata dequantizes wrong, silently.
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
# disk as the rollback path).
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
# every --spec-draft-*); half a set has segfaulted at load.
SPEC_TYPE=draft-mtp
+391
View File
@@ -0,0 +1,391 @@
#!/usr/bin/env bash
# Metadata-provider keys for Jellyfin on Dankserver, placed from the service's
# host env file with the plugin artifact pinned by content hash.
#
# Jellyfin 12 ships TMDb, OMDb, StudioImages and MusicBrainz as CORE plugins:
# they have no DLL, only a configuration file under the data root, and that
# file is created by the running container. So the fleet's provider state lives
# in two places the repo does not track —
#
# plugins/configurations/Jellyfin.Plugin.Tmdb.xml <TmdbApiKey /> (empty = keyless)
# plugins/configurations/Jellyfin.Plugin.Tvdb.xml <ApiKey /> (needs the plugin)
#
# and both are owned by the container user. This tool is the only thing that
# writes them: it is declarative, idempotent, and it never prints a key, so it
# is safe to run while someone watches.
#
# Keys never enter git and never enter argv (a value in argv is world-readable
# through /proc/<pid>/cmdline). They live in the per-service host env file that
# is the fleet's convention for container credentials (grafana, caddy, pihole,
# copyparty, romm, seerr), mode 600:
#
# /home/dank/Containers/jellyfin/provider-keys
# TMDB_API_KEY=<key>
# TVDB_API_KEY=<key>
# TVDB_PIN=<pin> # optional: only a subscriber-shaped TVDB key
# # needs one; a licensed key does not
#
# TVDB itself is not core in 12 (the metadata provider was cut in 10.9), so it
# is installed from the official catalog, pinned three ways: sha256 of the
# artifact, the plugin's own guid, and its version. A matching sha proves only
# that the bytes are the bytes fetched — the guid check is what says it is the
# official plugin, the version check is what says it is the one this file was
# reviewed against.
#
# Subcommands
# plan print the resolved shape and every pending change, touch nothing
# install fetch+verify the plugin, then write the keys (backing up first)
# verify prove the plugin and both keys are actually in place
#
# Nothing here restarts Jellyfin: the deployer owns that
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
# start-up, so a key written while the server runs is not live until then.
set -euo pipefail
# Jellyfin's data root on Dankserver (the rootless quadlet's config volume).
JF_ROOT="${JF_ROOT:-/home/dank/Containers/jellyfin/config}"
PROV_KEYS="${PROV_KEYS:-/home/dank/Containers/jellyfin/provider-keys}"
TRASH_DIR="${TRASH_DIR:-$HOME/.AGENTTRASH/$(date +%Y%m%d)-jellyfin-providers}"
PLUGINS="$JF_ROOT/plugins"
CONF="$PLUGINS/configurations"
TMDB_XML="$CONF/Jellyfin.Plugin.Tmdb.xml"
TVDB_XML="$CONF/Jellyfin.Plugin.Tvdb.xml"
# repo.jellyfin.org catalog entry for TheTVDB (owner: jellyfin), version 24.0.0.0,
# targetAbi 12.0.0.0 — published 2026-09-09, verified against Jellyfin 12.4.0.
TVDB_PLUGIN_URL="${TVDB_PLUGIN_URL:-https://repo.jellyfin.org/files/plugin/thetvdb/thetvdb_24.0.0.0.zip}"
TVDB_PLUGIN_SHA256="${TVDB_PLUGIN_SHA256:-6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319}"
TVDB_PLUGIN_GUID="${TVDB_PLUGIN_GUID:-a677c0da-fac5-4cde-941a-7134223f14c8}"
TVDB_PLUGIN_VERSION="${TVDB_PLUGIN_VERSION:-24.0.0.0}"
PLUGIN_DIR="thetvdb_${TVDB_PLUGIN_VERSION}"
TVDB_DLL="Jellyfin.Plugin.Tvdb.dll"
die() {
printf 'jellyfin-providers: %s\n' "$*" >&2
exit 1
}
usage() {
printf 'usage: %s plan|install|verify\n' "$0" >&2
exit 2
}
# check_keys — refuse anything that is not a readable private file. An open
# mode is an error, not a warning: a provider key readable by every user on the
# box is the failure this check exists to catch.
check_keys() {
[ -f "$PROV_KEYS" ] ||
die "$PROV_KEYS not found (create it with TMDB_API_KEY= and TVDB_API_KEY=, then chmod 600)"
local mode
mode="$(stat -c '%a' "$PROV_KEYS")"
case "$mode" in
600 | 400) ;;
*) die "$PROV_KEYS is mode $mode; provider keys must be 600" ;;
esac
}
# keys_value <VAR> — the value of one variable in the env file, or nothing.
keys_value() {
awk -v k="$1" -F= '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$PROV_KEYS"
}
# keys_len <VAR> — the length, and a refusal naming the variable when absent.
# The value itself is never printed.
keys_len() {
local v
v="$(keys_value "$1")"
[ -n "$v" ] || die "$PROV_KEYS has no $1= line"
printf '%s' "${#v}"
}
# xml_value <file> <element> — the element's text, empty when it is absent or
# self-closing.
xml_value() {
python3 - "$1" "$2" <<'PYEOF'
import re, sys
src = open(sys.argv[1], encoding="utf-8").read()
m = re.search(r"<%s(?:\s*/>|>(.*?)</%s>)" % (sys.argv[2], sys.argv[2]), src, re.S)
print(m.group(1).strip() if (m and m.group(1)) else "")
PYEOF
}
# xml_secret <file> <element> <var> — set <element> to that variable from the
# env file. The python process reads the env file itself, so the value never
# appears in argv.
xml_secret() {
python3 - "$1" "$2" "$3" "$PROV_KEYS" <<'PYEOF'
import os, re, sys, tempfile
from xml.sax.saxutils import escape
xml_path, elem, var, keys_path = sys.argv[1:5]
val = ""
for line in open(keys_path, encoding="utf-8"):
key, sep, rest = line.partition("=")
if sep and key.strip() == var:
val = rest.strip()
break
if not val:
raise SystemExit(f"{keys_path} has no {var}= line")
src = open(xml_path, encoding="utf-8").read()
pat = re.compile(r"([ \t]*)<%s(?:\s*/>|>.*?</%s>)" % (elem, elem), re.S)
text = escape(val)
if pat.search(src):
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
else:
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
if out == src:
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
d, base = os.path.split(xml_path)
fd, tmp = tempfile.mkstemp(prefix=base + ".", dir=d)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(out)
os.chmod(tmp, os.stat(xml_path).st_mode & 0o777)
os.replace(tmp, xml_path)
PYEOF
}
# plugin_meta <zip> — "guid version" out of the artifact's own meta.json.
plugin_meta() {
python3 - "$1" <<'PYEOF'
import json, sys, zipfile
meta = json.loads(zipfile.ZipFile(sys.argv[1]).read("meta.json").decode("utf-8"))
print(meta.get("guid", ""), meta.get("version", ""))
PYEOF
}
check_root() {
# Never fabricate a data root: creating plugins/ beside a typo'd path would
# install a plugin Jellyfin never loads and hide the operator's mistake.
[ -d "$CONF" ] || die "no Jellyfin configuration directory at $CONF (is JF_ROOT right?)"
}
check_keys_report() {
check_keys
# A refusal inside a command substitution used as a printf argument would
# be swallowed, so the guard runs as its own statement.
keys_len TMDB_API_KEY >/dev/null
keys_len TVDB_API_KEY >/dev/null
printf '= keys read from %s (mode %s)\n' "$PROV_KEYS" "$(stat -c '%a' "$PROV_KEYS")"
}
# The plugin's state, as one line in the shared vocabulary.
plugin_state() {
if [ -d "$PLUGINS/$PLUGIN_DIR" ]; then
printf '= %s already installed\n' "$PLUGIN_DIR"
return 0
fi
local other
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
if [ -n "$other" ]; then
printf '+ install %s (a different version is present: %s)\n' "$PLUGIN_DIR" "$other"
return 0
fi
printf '+ install %s from %s\n' "$PLUGIN_DIR" "${TVDB_PLUGIN_URL#*://}"
}
tmdb_state() {
if [ ! -f "$TMDB_XML" ]; then
printf '! %s is missing — save the TMDb plugin settings once in the UI, then re-run\n' "$TMDB_XML" >&2
return 1
fi
local now want
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
want="$(keys_value TMDB_API_KEY)"
if [ -n "$now" ] && [ "$now" = "$want" ]; then
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
elif [ -n "$now" ]; then
printf '+ replace the TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
else
printf '+ set TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
fi
}
tvdb_state() {
local now
if [ -f "$TVDB_XML" ]; then
now="$(xml_value "$TVDB_XML" ApiKey)"
if [ -n "$now" ]; then
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
else
printf '+ fill the empty ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
fi
else
printf '+ create Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
fi
}
write_plugin() {
local tmp guid version other
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
[ -z "$other" ] ||
die "another TheTVDB version is already installed ($other); move it to the trash first — two copies of one plugin means two loads"
tmp="$(mktemp "${TMPDIR:-/tmp}/thetvdb.XXXXXX.zip")"
# The pins are checked before anything is unpacked, so a bad artifact never
# reaches the plugins directory.
curl -fsSL --max-time 300 -o "$tmp" "$TVDB_PLUGIN_URL" ||
{ rm -f "$tmp"; die "could not fetch $TVDB_PLUGIN_URL"; }
local got
got="$(sha256sum "$tmp" | cut -d' ' -f1)"
if [ "$got" != "$TVDB_PLUGIN_SHA256" ]; then
rm -f "$tmp"
die "artifact sha256 is $got, pinned $TVDB_PLUGIN_SHA256 — refusing to install"
fi
read -r guid version <<<"$(plugin_meta "$tmp")"
if [ "$guid" != "$TVDB_PLUGIN_GUID" ]; then
rm -f "$tmp"
die "artifact guid is $guid, expected the official $TVDB_PLUGIN_GUID — refusing to install"
fi
if [ "$version" != "$TVDB_PLUGIN_VERSION" ]; then
rm -f "$tmp"
die "artifact version is $version, pinned $TVDB_PLUGIN_VERSION — refusing to install"
fi
mkdir -p "$PLUGINS/$PLUGIN_DIR"
python3 -m zipfile -e "$tmp" "$PLUGINS/$PLUGIN_DIR/"
rm -f "$tmp"
[ -f "$PLUGINS/$PLUGIN_DIR/$TVDB_DLL" ] ||
die "$PLUGIN_DIR has no $TVDB_DLL — the artifact does not match this plugin"
printf '+ installed %s\n' "$PLUGIN_DIR"
}
backup_to_trash() {
local f="$1"
mkdir -p "$TRASH_DIR"
local dest="$TRASH_DIR/$(basename "$f").$(date +%H%M%S)"
cp -p "$f" "$dest"
printf '+ previous copy kept at %s\n' "$dest"
}
cmd_write_keys() {
local now want
KEYS_CHANGED=0
# TMDb: the running container owns this file and every setting in it. Only
# the key element is touched, so country, language and adult settings
# survive the edit.
[ -f "$TMDB_XML" ] ||
die "$TMDB_XML is missing — save the TMDb plugin settings once in the UI, then re-run"
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
keys_len TMDB_API_KEY >/dev/null
want="$(keys_value TMDB_API_KEY)"
if [ "$now" = "$want" ]; then
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
else
backup_to_trash "$TMDB_XML"
xml_secret "$TMDB_XML" TmdbApiKey TMDB_API_KEY
printf '+ set TmdbApiKey (%s chars)\n' "${#want}"
KEYS_CHANGED=1
fi
# TVDB: never overwrite a config the operator saved through the UI.
if [ -f "$TVDB_XML" ]; then
now="$(xml_value "$TVDB_XML" ApiKey)"
if [ -n "$now" ]; then
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
else
keys_len TVDB_API_KEY >/dev/null
backup_to_trash "$TVDB_XML"
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
if [ -n "$(keys_value TVDB_PIN)" ] && [ -z "$(xml_value "$TVDB_XML" Pin)" ]; then
xml_secret "$TVDB_XML" Pin TVDB_PIN
fi
printf '+ filled the ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
KEYS_CHANGED=1
fi
else
keys_len TVDB_API_KEY >/dev/null
cat >"$TVDB_XML" <<XML
<?xml version="1.0" encoding="utf-8"?>
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ApiKey />
</Configuration>
XML
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
if [ -n "$(keys_value TVDB_PIN)" ]; then
xml_secret "$TVDB_XML" Pin TVDB_PIN
fi
printf '+ created Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
KEYS_CHANGED=1
fi
}
cmd_plan() {
check_root
check_keys_report
printf 'root %s\n' "$JF_ROOT"
plugin_state
tmdb_state
tvdb_state
}
cmd_install() {
check_root
check_keys_report
KEYS_CHANGED=0
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
write_plugin
KEYS_CHANGED=1
else
printf '= %s already installed\n' "$PLUGIN_DIR"
fi
cmd_write_keys
if [ "$KEYS_CHANGED" -eq 1 ]; then
# Nothing is restarted here on purpose: this host may be playing media
# right now, and both the plugin and its keys are read at start-up.
printf 'next: scripts/deploy.sh dankserver --service jellyfin (the plugin and keys load at start-up)\n'
else
printf '= no change\n'
fi
}
cmd_verify() {
check_root
local bad=0
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
printf 'MISSING plugin directory %s (thetvdb not installed)\n' "$PLUGIN_DIR" >&2
bad=1
else
local dlls abi
dlls="$(find "$PLUGINS/$PLUGIN_DIR" -maxdepth 1 -name '*.dll' | wc -l)"
[ "$dlls" -ge 1 ] || {
printf 'MISSING %s inside %s\n' "$TVDB_DLL" "$PLUGIN_DIR" >&2
bad=1
}
abi="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("targetAbi","?"))' \
"$PLUGINS/$PLUGIN_DIR/meta.json" 2>/dev/null || echo '?')"
printf 'plugin %s: %s dll(s), targetAbi %s\n' "$PLUGIN_DIR" "$dlls" "$abi"
fi
local pair elem n
for pair in "$TMDB_XML|TmdbApiKey" "$TVDB_XML|ApiKey"; do
elem="${pair##*|}"
if [ ! -f "${pair%%|*}" ]; then
printf 'MISSING %s\n' "${pair%%|*}" >&2
bad=1
continue
fi
if ! python3 -c 'import sys, xml.dom.minidom; xml.dom.minidom.parse(sys.argv[1])' "${pair%%|*}"; then
printf 'UNPARSEABLE %s\n' "${pair%%|*}" >&2
bad=1
continue
fi
n="$(xml_value "${pair%%|*}" "$elem")"
if [ -z "$n" ]; then
printf 'MISSING key: %s has no %s value\n' "$(basename "${pair%%|*}")" "$elem" >&2
bad=1
else
printf '%s %s chars\n' "$elem" "${#n}"
fi
done
[ "$bad" -eq 0 ] || return 1
printf 'providers configured\n'
}
cmd="${1:-}"
case "$cmd" in
plan) cmd_plan ;;
install) cmd_install ;;
verify) cmd_verify ;;
*) usage ;;
esac
+114 -50
View File
@@ -1,44 +1,53 @@
#!/usr/bin/env bash
# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
# acceptance gate: three KLD reports against ONE reference recording, one verdict.
# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
# recording.
#
# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
# kldref, in the same drained window that records the calibration grams). Every scored
# container is measured against that one recording, so the KL scale runs from 0 = the
# checkpoint itself downward in faithfulness. Three containers ride it:
# checkpoint itself downward in faithfulness.
#
# fp8.json the published FP8-native release — the achievable end of the interval
# iq4r.json the payload frank-archon serves — the bar to beat
# cand.json the arm under test — the subject
# TWO MODES, because the disk decides how many anchors can exist at once:
#
# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
# is half of the distance between the 4-bit payload and the near-lossless release, not half
# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
# is gone, so its 172.8 GiB fits.
# gate IQ4R.json CAND.json candidate vs the served payload
# gate FP8.json IQ4R.json CAND.json the same, inside the absolute interval
#
# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
# 1. KL(fp8) < KL(iq4r) the anchors are ordered. If the near-lossless
# release does NOT sit closer to the checkpoint than
# the 4-bit payload, the corpus, the tokenizer or the
# boot shape is wrong and no arm number means anything.
# 2. KL(fp8) < KL(cand) < KL(iq4r) the candidate lies strictly inside the interval —
# outside it, something is mis-measured, not better
# 3. KL(cand) <= KL(iq4r) - R x gap it closes at least R (--recover, default 0.5) of the
# gap = KL(iq4r) - KL(fp8): at least half the way from
# what is served toward the release (agreed 2026-10-27)
# 4. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
# release, not half of an invented distance to the checkpoint. Without it the gate still
# answers the campaign's question — does the candidate beat what is served — and says so.
#
# The law, on the summary of one slice (--slice all | a corpus source):
#
# candidate vs served (2 reports)
# 1. mean KL(cand) < mean KL(iq4r) the served payload is beaten on the corpus
# 2. p99.9(cand) <= p99.9(iq4r) and the tail is no worse — a mean can hide a few
# positions taking the whole hit
# 3. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
# 4. every source slice passes 1-3 a win bought by regressing one distribution
# (code, prose, tools) is not a win
#
# inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
# recovery fraction, as agreed 2026-10-27) plus:
# 5. p99.9(cand) <= p99.9(iq4r) the tail law
# 6. every source slice passes the slice law
#
# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
# same step size, --deterministic, no speculation) that makes the three numbers comparable.
# same step size, --deterministic, no speculation) that makes the numbers comparable.
# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
#
# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
# kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
set -uo pipefail
[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
if [ "$#" -lt 2 ]; then
echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
echo " kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
exit 2
fi
python3 - "$@" <<'PY'
import json, sys
@@ -52,13 +61,15 @@ while i < len(a):
else:
files.append(a[i])
i += 1
if len(files) != 3:
sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
if len(files) not in (2, 3):
sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
" kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
o = dict(zip(opts[0::2], opts[1::2]))
slice_ = o.get("--slice", "all")
recover = float(o.get("--recover", 0.5))
if not 0.0 < recover <= 1.0:
sys.exit(f"--recover must be in (0,1], got {recover}")
interval = len(files) == 3
def die(msg):
@@ -66,11 +77,15 @@ def die(msg):
raise SystemExit(2)
def read(path):
def load(path):
try:
d = json.load(open(path))
return json.load(open(path))
except Exception as e:
die(f"cannot read the KLD report {path}: {e}")
def summary(path, d):
"""The slice this gate grades: 'all', or one corpus source."""
if slice_ == "all":
g = d.get("all")
else:
@@ -80,23 +95,25 @@ def read(path):
f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
if g is None or "kld" not in g or "top1_agreement" not in g:
die(f"{path} has no {slice_!r} summary — wrong report version?")
return d, g
return g
docs = {label: load(path) for label, path in
zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
rows = []
for label, path in zip(("fp8", "iq4r", "cand"), files):
d, g = read(path)
rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
g["top1_agreement"], g["positions"]))
for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
g = summary(path, docs[label])
rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
print(f"reference: BF16 container; slice: {slice_}")
print(f"reference: BF16 container; slice: {slice_}; "
+ ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8} positions")
for label, path, model, kl, p999, top1, n in rows:
print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}% {n}")
r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
gap = kl_iq4r - kl_fp8
r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
kl_cand, p_cand, t_cand = r["cand"]
kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
ok = True
@@ -106,17 +123,64 @@ def law(n, text, passed):
print(f" [{'PASS' if passed else 'FAIL'}] {n}. {text}")
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
gap > 0)
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
kl_fp8 < kl_cand < kl_iq4r)
need = kl_iq4r - recover * gap
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
if gap > 0 else "gap is not positive; cannot score recovery",
gap > 0 and kl_cand <= need)
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
def tail_law(n):
law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
p_cand <= p_iq4r)
# --- the slice law: every corpus source must pass the metric laws -------------
def slice_law(n):
"""A win that regresses one source class is not a win: check them all."""
mine = docs["cand"].get("by_source") or {}
theirs = docs["iq4r"].get("by_source") or {}
if not mine or not theirs:
law(n, "no per-source summaries to check", True)
return
missing = sorted(set(theirs) - set(mine))
extra = sorted(set(mine) - set(theirs))
if missing or extra:
law(n, "same source slices in both reports"
+ (f" (missing from cand: {', '.join(missing)})" if missing else "")
+ (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
return
bad = []
for src in sorted(theirs):
c, q = mine[src], theirs[src]
worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
worse_top1 = c["top1_agreement"] < q["top1_agreement"]
if worse_mean or worse_tail or worse_top1:
why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
("top-1", worse_top1)) if w)
bad.append(f"{src} ({why})")
law(n, "every source slice passes (mean, p99.9, top-1)"
+ (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
if interval:
kl_fp8 = r["fp8"][0]
gap = kl_iq4r - kl_fp8
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
gap > 0)
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
kl_fp8 < kl_cand < kl_iq4r)
need = kl_iq4r - recover * gap
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
if gap > 0 else "gap is not positive; cannot score recovery",
gap > 0 and kl_cand <= need)
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
tail_law(5)
slice_law(6)
else:
law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
kl_cand < kl_iq4r)
tail_law(2)
law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
slice_law(4)
print("KLD GATE " + ("PASS" if ok else "FAIL"))
sys.exit(0 if ok else 1)
PY
+175 -112
View File
@@ -34,14 +34,25 @@ import json
import os
import random
import re
import sys
from jinja2 import Environment, FileSystemLoader
from tokenizers import Tokenizer
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import qwen38_common as qc # noqa: E402 - sibling module, same directory
CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
".yaml", ".yml", ".bats", ".service", ".container", ".toml")
JSONL_EXTS = (".jsonl", ".json")
DOC_DIR = "serverAppConfigs-docs"
FRAGMENT_TOKENS = 2400 # what one fragment may contribute
CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
POOL_CYCLES = 40 # how many times a class pool may repeat before giving up
KLD_GUARD = 5000 # rendered conversations tried before declaring the pool short
FLEET_CMDS = [
"systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
return out[:cap]
def main() -> None:
def parse_args() -> argparse.Namespace:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True)
ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
ap.add_argument("--hf-code-parquet", default="")
ap.add_argument("--hf-prose-parquet", default="")
ap.add_argument("--seed", type=int, default=20261006)
a = ap.parse_args()
return ap.parse_args()
rng = random.Random(a.seed)
tokenizer = Tokenizer.from_file(a.tokenizer_json)
env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
def load_template(template: str) -> tuple[Environment, str]:
"""The chat template as the engine will render it, plus its short name."""
env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
trim_blocks=True, lstrip_blocks=True)
env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
ValueError(m))
tname = os.path.basename(a.template)
return env, os.path.basename(template)
def classify(repo: str) -> dict[str, list[str]]:
"""Walk the checkout into the three source classes the mix draws from."""
code, prose, jsonl = [], [], []
for root, dirs, files in os.walk(a.repo):
for root, dirs, files in os.walk(repo):
dirs[:] = [d for d in dirs if not d.startswith(".")]
for f in sorted(files):
p = os.path.relpath(os.path.join(root, f), a.repo)
p = os.path.relpath(os.path.join(root, f), repo)
if DOC_DIR in p and f.endswith(".md"):
prose.append(p)
elif p.startswith(DOC_DIR):
@@ -180,137 +195,185 @@ def main() -> None:
code.append(p)
elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
jsonl.append(p)
# file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
pools: dict[str, tuple[list, list]] = {}
for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
[p for p in lst if fnv(p) % 5 != 0])
return {"code": code, "prose": prose, "tools": jsonl}
def text_of(p: str) -> str:
path = os.path.join(a.repo, p)
raw = open(path, encoding="utf-8", errors="replace").read()
if p.endswith(JSONL_EXTS):
lines = [json.loads(l)["content"] for l in raw.splitlines()
if l.strip() and "content" in l]
raw = "\n".join(lines)
toks = tokenizer.encode(raw, add_special_tokens=False).ids
return tokenizer.decode(toks[:2400], skip_special_tokens=True)
def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
"""File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
return {cls: ([p for p in lst if fnv(p) % 5 == 0],
[p for p in lst if fnv(p) % 5 != 0])
for cls, lst in classes.items()}
def jsonl_content(raw: str) -> str:
"""The content fields of a content-keyed JSONL, or "" for anything else.
A .json in this repo is not necessarily a content-keyed JSONL: the substring
test passes on any line that merely mentions "content", and a list-shaped
document raises TypeError — both used to crash the whole build. Skip what
does not fit instead.
"""
lines = []
for line in raw.splitlines():
if not line.strip() or "content" not in line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(rec, dict) and isinstance(rec.get("content"), str):
lines.append(rec["content"])
return "\n".join(lines)
def fragment_text(repo: str, rel: str, tok) -> str:
"""One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
raw = open(os.path.join(repo, rel), encoding="utf-8",
errors="replace").read()
if rel.endswith(JSONL_EXTS):
raw = jsonl_content(raw)
ids = tok.encode(raw, add_special_tokens=False).ids
return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
def build_frags(a: argparse.Namespace, tok,
rng: random.Random) -> dict[str, dict[str, list[str]]]:
"""Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
pools = split_pools(classify(a.repo))
cmds = harvest_cmds(a.repo, rng)
# fragment pools per class, corpus-split first
frags: dict[str, dict[str, list[str]]] = {}
for cls in pools:
kld_files, cal_files = pools[cls]
fr = {}
src = {"kld": kld_files, "cal": cal_files}
for cls, (kld_files, cal_files) in pools.items():
if cls == "tools":
fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
"cal": [tool_call(rng, cmds) for _ in range(40000)]}
else:
fr["kld"] = [text_of(p) for p in src["kld"]]
fr["cal"] = [text_of(p) for p in src["cal"]]
frags[cls] = fr
frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
"cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
# per-class fragment caps: prose rows are short (~66 tok) so need many;
# codeparrot files are long (~2.6k tok) so a small cap suffices
hfcap = {"code": 40000, "prose": 60000, "tools": 0}
for cls, spec in (("code", a.hf_code_parquet),
("prose", a.hf_prose_parquet)):
for path in [p for p in spec.split(",") if p]:
frags[cls]["cal"].extend(
load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
for cls in frags:
rng.shuffle(frags[cls]["cal"])
return frags
def toks(s: str) -> int:
return len(tokenizer.encode(s, add_special_tokens=False).ids)
def emit_calib() -> tuple[int, list[dict]]:
targets = {"code": .45, "prose": .30, "tools": .25}
rows: list[dict] = []
for cls, frac in targets.items():
want = int(a.calib_tokens * frac)
pool = frags[cls]["cal"]
def pack_class(cls: str, pool: list[str], want: int, tok,
chunk_tokens: int) -> list[dict]:
"""Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
if not pool:
return []
rows: list[dict] = []
# Count every fragment once: the loop cycles the pool up to POOL_CYCLES
# times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
# build's wall time.
counts = [qc.n_tokens(tok, f) for f in pool]
got = 0
i = 0
buf: list[str] = []
nbuf = 0
while got < want:
j = i % len(pool)
frag = pool[j]
i += 1
n = counts[j]
if nbuf + n > chunk_tokens and buf:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
buf, nbuf = [], 0
if got >= want:
break
buf.append(frag)
nbuf += n
if i > len(pool) * POOL_CYCLES:
break # pool exhausted with repeats; stop
if buf and got < want:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
print(f"calib {cls}: {got} tokens, {i} fragments")
return rows
def emit_calib(a: argparse.Namespace, frags, tok,
rng: random.Random) -> tuple[int, list[dict]]:
"""calib.jsonl rows: the class mix, shuffled, prompts only."""
rows: list[dict] = []
for cls, frac in CLASS_MIX.items():
rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
tok, a.chunk_tokens))
rng.shuffle(rows)
return len(rows), [{"prompt": r["prompt"]} for r in rows]
def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
env: Environment, tname: str) -> tuple[int, list[dict], int]:
"""kld.jsonl rows: multi-turn conversations, each with a scored tail."""
template = env.get_template(tname)
rows: list[dict] = []
total = scored_total = 0
classes = ["code", "prose", "tools"]
cursor = {c: 0 for c in classes}
guard = 0
while total < a.kld_tokens and guard < KLD_GUARD:
guard += 1
msgs, srcs = [], []
for t in range(rng.choice([2, 3, 4])):
cls = classes[(len(rows) + t) % 3]
pool = frags[cls]["kld"]
if not pool:
continue
got, i, buf, nbuf = 0, 0, [], 0
while got < want:
frag = pool[i % len(pool)]
i += 1
n = toks(frag)
if nbuf + n > a.chunk_tokens and buf:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
buf, nbuf = [], 0
if got >= want:
break
buf.append(frag)
nbuf += n
if i > len(pool) * 40:
break # pool exhausted with repeats; stop
if buf and got < want:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
print(f"calib {cls}: {got} tokens, {i} fragments")
rng.shuffle(rows)
return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
for r in rows]
frag = pool[cursor[cls] % len(pool)]
cursor[cls] += 1
msgs.append({"role": "user",
"content": "Summarize and explain this "
f"engineering material:\n{frag}"})
msgs.append({"role": "assistant",
"content": frag[:4000]})
srcs.append(cls)
rendered = template.render(messages=msgs, add_generation_prompt=False)
n = qc.n_tokens(tok, rendered)
if n < a.kld_min_prompt:
continue
rows.append({"prompt": rendered,
"score_from": n - a.score_tail,
"source": "+".join(sorted(set(srcs)))})
total += n
scored_total += a.score_tail
return len(rows), rows, scored_total
def emit_kld() -> tuple[int, list[dict]]:
rows, total = [], 0
classes = ["code", "prose", "tools"]
i = {c: 0 for c in classes}
guard = 0
while total < a.kld_tokens and guard < 5000:
guard += 1
msgs, srcs = [], []
for t in range(rng.choice([2, 3, 4])):
cls = classes[(len(rows) + t) % 3]
pool = frags[cls]["kld"]
if not pool:
continue
frag = pool[i[cls] % len(pool)]
i[cls] += 2
msgs.append({"role": "user",
"content": "Summarize and explain this "
f"engineering material:\n{frag}"})
msgs.append({"role": "assistant",
"content": frag[:4000]})
srcs.append(cls)
rendered = env.get_template(tname).render(
messages=msgs, add_generation_prompt=False)
n = toks(rendered)
if n < a.kld_min_prompt:
continue
rows.append({"prompt": rendered,
"score_from": n - a.score_tail,
"source": "+".join(sorted(set(srcs)))})
total += n
return len(rows), rows
os.makedirs(a.out_dir, exist_ok=True)
nc, rows = emit_calib()
with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
nk, krows = emit_kld()
with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
for r in krows:
f.write(json.dumps(r) + "\n")
scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
- r["score_from"] for r in krows)
manifest = {
def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
scored: int) -> dict:
return {
"seed": a.seed, "template": tname,
"tokenizer": os.path.basename(a.tokenizer_json),
"calib_prompts": nc, "kld_prompts": nk,
"kld_scored_positions": scored,
"kld_score_tail": a.score_tail,
"disjoint_file_pools": "local KLD material only; HF code feeds calib",
"calib_mix": {"code": .45, "prose": .30, "tools": .25},
"calib_mix": dict(CLASS_MIX),
"hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
}
with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
json.dump(manifest, f, indent=1)
print(json.dumps(manifest, indent=1))
def main() -> None:
a = parse_args()
rng = random.Random(a.seed)
tok = qc.load_tokenizer(a.tokenizer_json)
env, tname = load_template(a.template)
frags = build_frags(a, tok, rng)
os.makedirs(a.out_dir, exist_ok=True)
nc, calib_rows = emit_calib(a, frags, tok, rng)
qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
man = manifest(a, tname, nc, nk, scored)
qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
print(json.dumps(man, indent=1))
if __name__ == "__main__":
+454
View File
@@ -0,0 +1,454 @@
#!/usr/bin/env python3
"""media-apply.py — the only thing that moves movie-library files.
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
an interrupted run has destroyed nothing. Every run is resumable and every run
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
BEFORE the rename it authorises. That ordering is the whole design: a crash
leaves either an intent whose move happened or an intent whose move never ran,
and both are recognised on the next run. A file that moved with no journal line
cannot have been moved by this tool, and saying so is better than guessing.
Four subcommands:
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
no --apply => report only and write NOTHING, not even folders
rollback --journal J [--root ROOT] [--apply]
back to the paths the census saw; folders this run created are
removed only when empty, never anything else
verify --plan PLAN --journal J [--root ROOT]
every planned file is at its target with the journalled byte count
sweep --journal J [--root ROOT] [--apply]
removes the folders THIS RUN emptied — the journal has to show a
file that moved out of one, so a folder someone made by hand can
never be removed by this tool, and rollback puts them back
Refusals (exit 2, before the first rename, with nothing touched): a target
already occupied by a foreign file, a source that has gone, a duplicate target,
a path outside --root, a journal that was already rolled back, a move that would
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
--force relaxes only the guards that are opinion (journal already rolled back,
paths outside --root); it never allows overwriting a file.
--metadata-view names the overlay the media servers read their library through
(default /var/lib/media-meta/view). The sidecar follow-up printed after an apply
or rollback is conditioned on that path being a mountpoint: where no view is
mounted the servers bind the array itself, the sidecars moved with the media,
and a reap order would point at a directory that is not there.
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
inside it: a log inside the library is a file Jellyfin will index.
"""
import argparse
import datetime
import json
import os
import sys
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
class Refuse(Exception):
"""A gate the operator has to see; every refusal leaves the library alone."""
def journal_default(root):
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
class Journal:
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
def __init__(self, path, writing):
self.path, self.writing = path, writing
self.records = []
if os.path.exists(path):
for line in open(path):
line = line.strip()
if line:
self.records.append(json.loads(line))
self._fh = None
def already(self, dst, src):
"""An intent for this exact (src, dst) was written by an earlier run."""
return any(r.get("op") == "move" and r.get("dst") == dst
and r.get("src") == src for r in self.records)
@property
def rolled_back(self):
return any(r.get("op") == "rollback" for r in self.records)
def record(self, **fields):
self.records.append(fields)
if not self.writing:
return
if self._fh is None:
self._fh = open(self.path, "a")
self._fh.write(json.dumps(fields) + "\n")
self._fh.flush()
os.fsync(self._fh.fileno())
def close(self):
if self._fh:
self._fh.close()
def read_plan(path, root, force):
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
mkdirs, keeps, moves = [], [], []
with open(path) as fh:
header = fh.readline()
if not header.startswith("action\t"):
raise Refuse(f"{path} is not a plan (first column is "
f"{header.split(chr(9))[0]!r}, expected 'action') —"
" regenerate it with scripts/media-plan.py")
for line in fh:
f = line.rstrip("\n").split("\t")
if f[0] == "mkdir":
mkdirs.append(f[2])
elif f[0] == "keep":
keeps.append(f[1])
elif f[0] == "move":
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
if not force:
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
if p and not (p == root or p.startswith(root + os.sep)):
raise Refuse(f"{p} is outside --root {root}"
" (a plan built for another library?)")
return mkdirs, keeps, moves
# The media servers may read the library through the SSD overlay view
# (scripts/media-view.sh). A relayout moves bytes on the raw array, so sidecars
# the servers wrote through that view stay at the old coordinates — naming the
# follow-up here is the point of use for that fact, but only where the view
# exists: where it is not mounted the servers bind the array itself, the
# sidecars moved with the media, and a reap order points at a directory that
# is not there. See the Media Metadata View page for the host's state.
METADATA_VIEW = "/var/lib/media-meta/view"
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
" these moves — next: scripts/media-view.sh reap")
def metadata_note(args):
"""Print the sidecar follow-up only when the overlay is really mounted."""
if os.path.ismount(args.metadata_view):
print(METADATA_HINT)
def do_apply(args, root, journal, mkdirs, keeps, moves):
"""Preflight everything, then folders, then renames. Never a partial start."""
problems = []
for p in keeps:
if not os.path.exists(p):
problems.append(f"missing: keep row source is gone: {p}")
seen = {}
for src, dst, _kind in moves:
if src == dst:
problems.append(f"no-op: move row whose target is its source: {src}")
if dst in seen:
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
seen[dst] = src
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue # this intent already ran
if not os.path.exists(src):
if os.path.exists(dst):
problems.append(f"missing: {src} is gone and {dst} carries no journal"
" line — it moved outside this tool, refusing to guess")
else:
problems.append(f"missing: {src} is gone and {dst} is not there either")
continue
if os.path.exists(dst):
problems.append(f"occupied: target already holds another file: {dst}")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved")
for p in sorted(problems)[:20]:
print(" " + p)
if len(problems) > 20:
print(f" ... and {len(problems) - 20} more")
return 2
if not args.apply:
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
" — nothing written (pass --apply to run)")
for src, dst, _k in moves[:5]:
print(f" would move {src} -> {dst}")
return 0
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
dirs_new = 0
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
if d in made or os.path.isdir(d):
continue
journal.record(op="mkdir", dst=d)
os.makedirs(d, exist_ok=True)
dirs_new += 1
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
# library this size is not an option anyone would choose.
cross = []
for src, dst, _k in moves:
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue
parent = os.path.dirname(dst)
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
cross.append(dst)
if cross:
print(f"refusing: {len(cross)} moves would cross a filesystem"
f" (first: {cross[0]}) — this tool only renames")
return 2
moved = done = 0
for src, dst, _kind in moves:
if journal.already(dst, src):
if os.path.exists(dst) and not os.path.exists(src):
done += 1
continue
if os.path.exists(src) and not os.path.exists(dst):
os.rename(src, dst) # the intent ran, the move did not
moved += 1
continue
st = os.stat(src)
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
os.rename(src, dst)
moved += 1
print(f"applied: {moved} moved, {done} already done in an earlier run,"
f" {dirs_new} folders created, {len(keeps)} files stayed put")
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
f" --journal {journal.path} --root {root} --apply")
metadata_note(args)
return 0
def do_verify(root, journal, moves, keeps):
"""Every planned file sits at its target carrying the journalled bytes."""
journalled = {(r["src"], r["dst"]): r for r in journal.records
if r.get("op") == "move"}
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
if r.get("op") == "move"]
bad, missing, bytes_total = [], [], 0
for src, dst, _kind in todo:
rec = journalled.get((src, dst))
if not os.path.exists(dst):
missing.append(dst)
continue
size = os.path.getsize(dst)
bytes_total += size
if rec and rec.get("bytes") not in (None, size):
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
if os.path.exists(src) and src != dst:
bad.append(f"{src}: still in place as well as at {dst}")
missing += [p for p in keeps if not os.path.exists(p)]
if bad or missing:
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
for line in (missing + bad)[:20]:
print(" " + line)
return 2
empty = [r["dst"] for r in journal.records
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
and not os.listdir(r["dst"])]
print(f"verified: {len(todo)} files at their targets"
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
if empty:
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
return 0
def sweep_targets(root, journal):
"""The folders this run emptied, deepest first — never one it did not touch.
Eligibility is evidence, not appearance. A folder is a target only when the
journal shows a file moved out of it or out of something beneath it, it holds
nothing but empty directories now, it is not a folder this run created, and no
journalled destination lives under it. A symlink anywhere inside disqualifies
the whole subtree: this tool never follows a link, and deleting the folder
that holds one would delete someone's mount point.
Proof carries downward: once a folder qualifies, every empty directory inside
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
a move source, so without that rule it holds its emptied parent on disk
forever — the parent can never be removed while the child stands.
"""
moves = [r for r in journal.records if r.get("op") == "move"]
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
# An rmdir INTENT that failed left its folder standing, and the next sweep
# must be able to finish it: only a folder that is really gone is out of
# scope. Counting every intent as done made 146 live survivors permanent.
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
and not os.path.exists(r["path"])}
dsts = [r["dst"] for r in moves]
marked = set()
for r in moves:
d = os.path.dirname(r["src"])
while d.startswith(root + os.sep):
marked.add(d)
d = os.path.dirname(d)
def removable(d):
if d in made or d in gone or os.path.islink(d):
return False
if any(p == d or p.startswith(d + os.sep) for p in dsts):
return False
try:
entries = os.listdir(d)
except OSError:
return False
for e in entries:
c = os.path.join(d, e)
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
return False
return True
roots = [d for d in marked if removable(d)]
targets = set(roots)
for d in roots:
for dirpath, dirnames, _ in os.walk(d):
for c in dirnames:
p = os.path.join(dirpath, c)
if removable(p):
targets.add(p)
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
def do_sweep(args, root, journal):
"""Remove the empty husks an apply left standing at the library root."""
moves = [r for r in journal.records if r.get("op") == "move"]
if not moves:
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
" does not guess at emptiness")
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
if not (p == root or p.startswith(root + os.sep))})
if outside and not args.force:
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
f" (first: {outside[0]}) — this journal belongs to another"
" library")
targets = sweep_targets(root, journal)
if not targets:
print("swept: 0 folders — nothing this run emptied is still standing empty")
return 0
if not args.apply:
print(f"dry run: {len(targets)} folders this run emptied would be removed"
" (pass --apply to run)")
for d in targets[:5]:
print(f" would remove {d}")
return 0
removed = 0
for d in targets:
journal.record(op="rmdir", path=d)
try:
os.rmdir(d)
except OSError as e:
print(f"left in place: {d} ({e})")
continue
removed += 1
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
print(f"journal: {journal.path} — rollback recreates them with their files")
return 0
def do_rollback(args, root, journal):
"""Undo every journalled move, newest first, after a full preflight."""
moves = [r for r in journal.records if r.get("op") == "move"]
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
problems = []
for r in reversed(moves):
src, dst = r["src"], r["dst"]
if not args.force and not (src.startswith(root + os.sep)
or dst.startswith(root + os.sep)):
problems.append(f"outside --root: {dst}")
continue
if os.path.exists(dst) and os.path.exists(src):
problems.append(f"occupied: both paths exist, refusing to choose:"
f" {src} | {dst}")
elif os.path.exists(dst):
pass
elif os.path.exists(src):
continue # already undone
else:
problems.append(f"missing: {dst} is gone and {src} never came back")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved back")
for p in problems[:20]:
print(" " + p)
return 2
if not args.apply:
print(f"dry run: would move {len(moves)} files back to their census paths"
f" and recreate {len(set(swept))} swept folders (pass --apply)")
return 0
undone = 0
recreated = 0
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
if not os.path.isdir(d):
os.makedirs(d, exist_ok=True)
recreated += 1
for r in reversed(moves):
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
os.rename(r["dst"], r["src"])
undone += 1
dirs_gone = 0
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
if d == root or not os.path.isdir(d) or os.listdir(d):
continue
try:
os.rmdir(d)
dirs_gone += 1
except OSError as e:
print(f"left in place: {d} ({e})")
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
print(f"rolled back: {undone} files back where the census found them,"
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
metadata_note(args)
return 0
def main():
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
ap.add_argument("--plan")
ap.add_argument("--root", default=DEFAULT_ROOT)
ap.add_argument("--journal")
ap.add_argument("--apply", action="store_true", help="actually change the library")
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
ap.add_argument("--metadata-view", default=METADATA_VIEW,
help="overlay the media servers read through; the sidecar"
" note prints only when this path is a mountpoint")
args = ap.parse_args()
root = os.path.abspath(args.root)
if not os.path.isdir(root):
raise Refuse(f"no library at {root}")
journal_path = args.journal or journal_default(root)
journal = Journal(journal_path, args.apply)
try:
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
raise Refuse("this journal has already been rolled back"
" (--force to replay it)")
if args.cmd in ("apply", "verify") and not args.plan:
raise Refuse(f"{args.cmd} needs --plan")
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
if args.plan else ([], [], []))
if args.cmd == "apply":
return do_apply(args, root, journal, mkdirs, keeps, moves)
if args.cmd == "verify":
return do_verify(root, journal, moves, keeps)
if args.cmd == "sweep":
return do_sweep(args, root, journal)
return do_rollback(args, root, journal)
finally:
journal.close()
if __name__ == "__main__":
try:
sys.exit(main())
except Refuse as r:
print(f"refusing: {r}")
sys.exit(2)
+425
View File
@@ -0,0 +1,425 @@
#!/usr/bin/env python3
r"""media-audiobooks.py — a plan that re-files the audiobook library into one folder per book.
media-audiobooks.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Plex reads this library and Jellyfin does not: its music scanner has no audiobook
concept, and the tree is indexed entirely by Plex, which matched 58,975 of the
62,879 files here into its Artist/Album/Track shape. So the authority is
`scripts/media-census.py audiobooks` — the BOOK (album row), its NARRATOR (that
book's single artist row) and its CHAPTERS (track rows) — and never the folder a
release pack made. The plan is:
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>.<ext>
... the same folder, " (2)" on the name, when two chapters
of one book carry the same name
What that buys is the one property both media servers' book scanners assume: a
folder holds exactly one book. Today 212 top-level folders are a mix of
author-named shelves and collection packs, and the books inside the packs are
split across `… [Part 1]`, `… [Part 2]` folders three or four deep.
Three laws here are the opposite of what a music shelf does, and each is forced
by what the rows actually hold:
* A chapter is NEVER renumbered. 7,556 of the 10,485 books contain at least one
chapter whose index Plex does not know, so a number written into a name would
be invented — and it would buy nothing, because both servers order playback by
the embedded track tag, not by the file name. A wrong number in a name is worse
than none: it is the number a human later trusts. A file already carrying its
own number keeps it, untouched.
* The shelf is the NARRATOR, not the author. Every book has exactly one artist row
and it is the narrator; the author survives only inside 10% of the album titles
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving the
author would mean parsing titles for one book in ten and inventing it for the
rest. One rule that always resolves beats a shelf that is half author and half
narrator; the author stays visible in the book folder's own name, which is the
title Plex shows.
* Nothing is guessed at. A chapter row with no book or no narrator parks — a
tag-scan placeholder (`[Unknown Album]`) names neither, since unrelated books
share it and filing on one merges them, and so does a chapter number the ripper
mistagged as the artist (`0.4`). An audio file Plex never matched parks, a
video container parks, and a loose file in a folder holding two books parks.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What a chapter file can be. .m4b is the audiobook container; the rest are the
# audio a pack ships when it was ripped as music.
AUDIO_EXT = {".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav", ".flac",
".wma", ".aiff", ".aif", ".m4v", ".aac", ".mka"}
# Never an audiobook, whatever the library it sits in says: Plex reads these as
# video, and renaming one into an audio extension would claim it is a chapter.
VIDEO_EXT = {".mkv", ".mp4", ".webm", ".avi", ".mov", ".mpg", ".mpeg", ".ts",
".m2ts", ".vob", ".ifo", ".wmv", ".iso", ".img", ".bdmv", ".ogm",
".divx"}
# Tag-scan placeholders. These strings are not identities: 1,106 chapters from
# 14 unrelated top-level folders carry the album `[Unknown Album]`, so filing on
# them would merge books that have nothing to do with each other — the exact
# shape this relayout exists to make impossible. Such a chapter parks with the
# path it has today and the fix belongs in its tags. `Various Artists` is NOT a
# placeholder: that is what a multi-cast read carries, and the one book here
# with 1,189 chapters under it is genuinely one book.
PLACEHOLDER_BOOK = {"[unknown album]", "unknown album", "no title", "untitled",
"[untitled]", "unknown title", "n/a", "-"}
PLACEHOLDER_NARRATOR = {"[unknown artist]", "unknown artist", "unknown author",
"<unknown>", "[unknown]", "unknown"}
def identity(value, placeholders, needs_letter=False):
"""The tag value Plex matched, when it names something; '' when it does not.
A trailing parenthesized group is stripped for the test only, because rippers
glue timestamps to the placeholder (`Unknown album (03/03/2015 21:39:55)`).
`needs_letter` is for a NARRATOR alone: a shelf is a person's name, and 272
chapters here carry a chapter number in the artist tag (`0.4`, `31`). The same
test must never judge a BOOK title, which may honestly be `1984`.
"""
v = (value or "").strip()
bare = re.sub(r"\s*\([^()]*\)\s*$", "", v).strip().casefold()
if bare in placeholders or (needs_letter and not any(c.isalpha() for c in v)):
return ""
return v
# The characters Plex documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the chapter row Plex matched
PLACED = {} # chapter path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
AUDIO_BY_DIR = defaultdict(set) # source dir -> chapter files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding chapters
FOLDERS_SEEN = {} # (dir, depth) -> book folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its chapters may sit
BOOKS = {} # (narrator, book, year) -> the book group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the rows mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing unprintable,
no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 60k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ------------------------------------------------------------------------ books
def group_books():
"""Every matched chapter into one group per (narrator, book, year).
The key is the identity Plex asserts in its own database, not the folder the
file sits in: one book arrives as four part folders inside a collection pack,
and two copies of one book in one pack are one book to Plex, whose album item
IS that narrator, title and year. The year is part of the key so a 1965
recording and a 2020 recording that share a title stay two books.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above Part 1/Part 2 sees both
narrator = identity(r["narrator"], PLACEHOLDER_NARRATOR, needs_letter=True)
book = identity(r["book"], PLACEHOLDER_BOOK)
if not book:
park(path, "a chapter row that names no book — Plex's album title is a "
"placeholder like [Unknown Album], so filing on it would "
"merge unrelated books onto one shelf")
continue
if not narrator:
park(path, "a chapter row that names no narrator — the artist row is a "
"placeholder like [Unknown Artist] or a chapter number a "
"ripper mistagged as the artist, so no shelf owns it")
continue
key = (narrator.casefold(), book.casefold(), (r["year"] or "").strip())
g = BOOKS.setdefault(key, {"narrators": [], "books": [], "year": key[2],
"members": [], "indexed": 0})
g["members"].append(path)
g["narrators"].append(narrator)
g["books"].append(book)
if str(r["track"] or "").isdigit():
g["indexed"] += 1
def book_folder(g):
"""The shelf Plex's own rows name: <narrator>/<book> (<year>). None when a
name is made of characters no path can hold — inventing a shelf is filing it
wrong, so those chapters park instead."""
narrator = safe(dominant(g["narrators"]))
book = safe(dominant(g["books"]))
if not narrator or not book:
return None
return os.path.join(ARGS.root, narrator,
f"{book} ({g['year']})" if g["year"] else book)
def place_books():
"""One folder per book; every chapter keeps the name it has today."""
for key in sorted(BOOKS):
g = BOOKS[key]
folder = book_folder(g)
if folder is None:
for path in sorted(g["members"]):
park(path, "a book or narrator Plex spells with characters no "
"folder name can hold")
continue
for path in sorted(g["members"]):
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
# The name the ripper wrote, unchanged: the track tag inside the file
# is what orders playback, and 72% of the books here have chapters no
# row numbers, so any number added here would be invented.
PLACED[path] = assign(path, folder, base, ext, "chapter",
"two chapters of one book carried the same name")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The book folders the chapters in `d`, or within `depth` directories below
it, were placed in. Memoised, because every loose file in a big collection
folder asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The book folder that owns this directory — the first place up the tree
where chapters sit, in the directory itself or within SUBTREE_DEPTH levels
below it, which is how a pack whose parts are `… [Part 1]`/`… [Part 2]` owns
the cover art lying above them. None when no chapter is near enough to say,
AMBIGUOUS when the chapters there belong to more than one book."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_other():
"""Everything that is not a chapter Plex matched: covers, booklets, EPUBs,
logs, checksums, playlists — and the clutter that is none of those."""
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
base = os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the audiobook library: Plex reads "
"it as video, and renaming it would claim it is a chapter")
elif ext in AUDIO_EXT:
park(path, "an audio file no chapter row names — it is not in the "
"library Plex matched, so nothing says which book it is")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a file whose folder holds more than one book, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one book folder would have carried two files of the same"
" sidecar name")
else:
park(path, "not an audiobook file, and no book's folder lies with"
" it to carry it")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Audiobook relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
unindexed = sorted(f"{safe(dominant(g['narrators']))}/{safe(dominant(g['books']))}"
for g in BOOKS.values()
if 0 < g["indexed"] < len(g["members"]))
if unindexed:
fh.write(f"{len(unindexed):>6} books have chapters Plex does not number. Their\n")
fh.write(" files keep the names they carry: playback order comes from the\n"
" track tag inside each file, and a number written here would be\n"
" invented for the chapters with no index. Fix it in the tags, not\n"
" in the names — Plex re-reads tags on a scan and never names files.\n\n")
fh.write(" Playlists move with their book and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"))
ap.add_argument("--enum", default="/tmp/audiobooks-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/plex-audiobooks.tsv",
help="the chapter rows scripts/media-census.py audiobooks wrote")
ap.add_argument("--out", default="/tmp/audiobooks-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/audiobooks-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
group_books()
place_books()
place_other()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a chapter row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+518
View File
@@ -0,0 +1,518 @@
#!/usr/bin/env python3
r"""Read-only census of a media library — the inputs a relayout plan is built from.
media-census.py enumerate [--root DIR] [--out TSV]
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
media-census.py jellyfin [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
media-census.py audiobooks [--root DIR] [--db PATH] [--out TSV]
Run it on the machine that mounts the library. Nothing here writes inside the
library and nothing is deleted; the only outputs are the TSVs below, and the
scripts/media-plan.py and scripts/media-shows.py read them instead of touching media.
enum bytes <TAB> ext <TAB> path every file under the root
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
one row per video-container file; `audio` is "codec:lang+codec:lang",
`subs` the subtitle-stream count, `spare` the audio streams carrying
neither a default nor a forced tag
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
the titles Plex already matched — the authority over any filename guess
jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
<TAB> tmdb <TAB> title <TAB> tvdb (see JF_COLS)
one row per episode Jellyfin has matched and numbered, its path mapped
from the container's view back to the host; `year`, `tmdb` and `tvdb`
are the SERIES' values, `title` is the episode title the row carries,
`extra` is Jellyfin's own extra type when the row is bonus material
rather than an episode
music file <TAB> artist <TAB> album <TAB> year <TAB> disc <TAB> track <TAB> title
<TAB> mb <TAB> performer (see MUSIC_COLS)
one row per track Jellyfin has tagged: the tag read is what the row IS,
so `artist` is the album artist (the track artist when that is empty) and
`album`/`year` are the album item's provider values when the folder is an
album item and the tags' otherwise; `mb` is the album's MusicBrainz
release group, `performer` the track artist the tags carry
audiobooks file <TAB> narrator <TAB> book <TAB> year <TAB> track <TAB> chapter
(see AB_COLS)
one row per chapter Plex matched under the audiobook root: `book` is the
album row Plex built from the files' own tags, `narrator` that book's
artist row — Plex keeps exactly one artist per book, which is why it, and
not an author parsed out of a title, names the shelf. `track` is empty
wherever Plex has no chapter number, which is most of this library.
`probe` is the slow step (every container is opened for its header, ~10 min for
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
live service must not starve it. `enumerate` is a stat pass, so re-censusing
before a plan is cheap — and a plan is only ever as good as these files.
"""
import argparse
import csv
import json
import os
import re
import sqlite3
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
# Trash and thumbnail caches are not library content, and this repo's trash policy
# means .AGENTTRASH can sit inside a movie folder at any time.
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
"parent_title", "parent_orig", "parent_year", "parent_edition",
"container", "vcodec", "acodec", "width", "height"]
PLEX_SQL = """
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
coalesce(mi.edition_title,''), coalesce(mi.year,0),
coalesce(par.title,''), coalesce(par.original_title,''),
coalesce(par.year,0), coalesce(par.edition_title,''),
m.container, m.video_codec, m.audio_codec, m.width, m.height
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
left join metadata_items par on par.id = mi.parent_id
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
"""
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
JF_COLS = ["file", "series", "year", "season", "episode", "extra",
"tmdb", "title", "tvdb"]
# Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
"MediaBrowser.Controller.Entities.TV.Season",
"MediaBrowser.Controller.Entities.TV.Series")
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType,
IsVirtualItem, Name from BaseItems where Type = ? and Path like ?""")
# The music tree is these two entity classes. The track row carries the tags
# Jellyfin read out of the file; the album row carries what the provider matched,
# which is the better folder name of the two.
JF_AUDIO = "MediaBrowser.Controller.Entities.Audio.Audio"
JF_ALBUM = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
MUSIC_COLS = ["file", "artist", "album", "year", "disc", "track", "title", "mb",
"performer"]
MUSIC_SQL = ("""select b.Path, b.AlbumArtists, b.Artists, b.Album, b.ProductionYear,
b.ParentIndexNumber, b.IndexNumber, b.Name, b.Id,
a.Name, a.ProductionYear, a.Id
from BaseItems b
left join BaseItems a on a.Id = b.ParentId and a.Type = ?
where b.Type = ? and b.Path like ?""")
# The album's identity, in preference order. A release group is the album; a bare
# release id is one reissue of it.
MUSIC_PROVIDERS = ("MusicBrainzReleaseGroup", "MusicBrainzAlbum")
# Plex's audiobook tree is the same Artist/Album/Track shape as its music tree:
# the leaf (10) is a CHAPTER, its parent (9) is the BOOK, and that book's artist
# (8) is the NARRATOR. The track's own `index` is exported because the exceptions
# report counts how many books have chapters Plex does not number — the reason the
# relayout refuses to invent numbers into file names.
PLEX_ARTIST, PLEX_ALBUM, PLEX_TRACK = 8, 9, 10
AB_COLS = ["file", "narrator", "book", "year", "track", "chapter"]
AB_SQL = f"""
select p.file, coalesce(ar.title,''), coalesce(al.title,''), coalesce(al.year,0),
coalesce(mi."index",-1), coalesce(mi.title,'')
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
join metadata_items al on al.id = mi.parent_id and al.metadata_type = {PLEX_ALBUM}
left join metadata_items ar on ar.id = al.parent_id
and ar.metadata_type = {PLEX_ARTIST}
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
and al.deleted_at is null
"""
def enumerate_lib(root, out):
"""One row per real file under root: size, extension, path."""
rows = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames
if not SKIP_DIR.search(os.path.join(dirpath, d))]
for fn in filenames:
p = os.path.join(dirpath, fn)
if SKIP_DIR.search(p):
continue
try:
st = os.lstat(p)
except OSError:
continue
if not os.path.isfile(p) or os.path.islink(p):
continue # links and sockets are not library content
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
with open(out, "w") as fh:
for size, ext, p in rows:
fh.write(f"{size}\t{ext}\t{p}\n")
vid = [r for r in rows if r[1] in PROBE_EXT]
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
def probe_one(path):
"""ffprobe header read -> the probed TSV row (never the whole file)."""
try:
os.nice(19) # a census must never outrank the media server
except OSError:
pass
try:
size = os.stat(path).st_size
except OSError:
return None
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
"-show_format", "-show_streams", path]
try:
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
data = json.loads(done.stdout or "{}")
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
json.JSONDecodeError, OSError):
data = {}
st = data.get("streams", [])
fmt = data.get("format", {})
v = next((s for s in st if s.get("codec_type") == "video"), None)
aud = [s for s in st if s.get("codec_type") == "audio"]
subs = [s for s in st if s.get("codec_type") == "subtitle"]
fps = "-"
if v and v.get("r_frame_rate"):
try:
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
except ValueError:
pass
def num(x):
try:
return round(float(x))
except (TypeError, ValueError):
return 0
return [size, path, num(fmt.get("duration")),
(v or {}).get("codec_name", "-"),
f"{v['width']}x{v['height']}" if v else "-",
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
len(subs),
sum(1 for a in aud if not re.search(
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
fmt.get("format_name", "-") or "-"]
def probe(enum, out, jobs):
"""Probe every video-container row of the enum TSV, in parallel."""
paths = []
with open(enum) as fh:
for line in fh:
f = line.rstrip("\n").split("\t", 2)
if len(f) == 3 and f[1] in PROBE_EXT:
paths.append(f[2])
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
done = unreadable = 0
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
for row in pool.map(probe_one, paths):
if row is None:
continue
if row[3] == "-":
unreadable += 1 # header unreadable: the row still gets written
done += 1
fh.write("\t".join(str(x) for x in row) + "\n")
if done % 250 == 0:
print(f" {done}/{len(paths)}", flush=True)
print(f"probed={done} unreadable={unreadable} -> {out}")
def plex(root, db, out):
"""Plex's matched titles for files under root, from its DB, read-only."""
uri = f"file:{db}?mode=ro&immutable=1"
try:
con = sqlite3.connect(uri, uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(PLEX_COLS)
n = 0
for r in con.execute(PLEX_SQL, (root + "/%",)):
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
n += 1
print(f"plex-matched={n} -> {out}")
def jellyfin(root, db, container, out):
"""The episodes Jellyfin has already numbered, for the library at root.
Jellyfin stores each path as the CONTAINER sees it (its quadlet binds the
share at /data), so `container` is the in-container prefix that stands for
root and every exported path is mapped back to the host before it is written
— the planner reads host paths, because it plans moves on the array.
The numbering is what matters, not the name: a release group's file name says
nothing about which episode a file is, and Jellyfin has already answered that
question against a provider. Season/episode and the episode's own title come
from the episode row; the series name, its year and its TMDb and Tvdb ids come
from the Series row the episode climbs to, because those are the only place
Jellyfin keeps them.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
"It is a WAL database: copy it with its -wal and -shm beside it and"
" point --db at the copy. Never sudo it — the data dir belongs to"
" the container's owner, and rootless storage is per-user.")
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
need = {"Id", "ParentId", "Type", "Path", "IndexNumber", "ParentIndexNumber",
"ExtraType", "IsVirtualItem", "ProductionYear", "Data"}
if not need <= have:
sys.exit(f"{db} is not a Jellyfin library database (BaseItems has no"
f" {','.join(sorted(need - have))})")
node = {}
for i, pid, typ, name, year, data in con.execute(
"select Id, ParentId, Type, Name, ProductionYear, Data from BaseItems"
" where Type in (?,?,?)", JF_TV):
node[i] = (pid, typ, name, year, data)
# Jellyfin 12 keeps provider ids in their own table; older builds kept them in
# the row's Data blob. Both shapes are read, because the tvdb id the series
# naming depends on is wherever this build happened to write it.
prov = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in ('Tmdb', 'Tvdb')"):
prov[(iid, pid)] = val
def provider_of(item_id, want):
got = prov.get((item_id, want))
if got:
return str(got)
try:
blob = json.loads(node[item_id][4] or "{}").get("ProviderIds") or {}
except (KeyError, ValueError, TypeError):
return ""
return str(blob.get(want, "") or "")
def series_of(item_id):
"""Climb episode -> season -> series; empty when the chain is broken."""
for _ in range(4):
if item_id not in node:
return "", "", "", ""
pid, typ = node[item_id][0], node[item_id][1]
if typ.endswith(".Series"):
return (node[item_id][2] or "", node[item_id][3] or "",
provider_of(item_id, "Tmdb"), provider_of(item_id, "Tvdb"))
item_id = pid
return "", "", "", ""
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
if not container:
container = "/data/" + os.path.basename(os.path.normpath(root))
prefix = container.rstrip("/") + "/"
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(JF_COLS)
for iid, path, idx, pidx, extra, virt, epname in con.execute(
JF_SQL, (JF_TV[0], prefix + "%")):
if virt or not path or not path.startswith(prefix):
skipped += 1 # a virtual item is a variant, not a file
continue
sname, syear, tmdb, tvdb = series_of(iid)
w.writerow([root + "/" + path[len(prefix):],
sname.replace("\t", " "), syear,
"" if pidx is None else pidx,
"" if idx is None else idx,
(extra or "").replace("\t", " "), tmdb,
(epname or "").replace("\t", " "), tvdb])
written += 1
print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
def music(root, db, container, out):
"""The tracks Jellyfin has already tagged, for the library at root.
Jellyfin reads music metadata out of the file itself, so an Audio row IS the
tag read, done once, by the vendor's own parser: its album-artist, album,
year, disc and track columns are what the embedded tags say and its Name is
the track title. That makes this export the whole authority for a music
relayout — the path a ripper chose is not evidence of anything, and the
folder Jellyfin put a track under is often a release pack, not an album.
Two gaps this export answers from the row rather than the tree. Many tracks
sit under a folder that is not an album item at all (a disc folder, or a
mixed one), so the album name and year fall back to the tags when the album
row is absent. And an album's MusicBrainz ids are not in the row's own Data
— the current schema leaves that empty for music — they are rows of
BaseItemProviders keyed by the item's Id.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
"It is a WAL database: copy it with its -wal and -shm beside it and"
" point --db at the copy. Never sudo it — the data dir belongs to"
" the container's owner, and rootless storage is per-user.")
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
need = {"Id", "ParentId", "Type", "Path", "Name", "IndexNumber",
"ParentIndexNumber", "ProductionYear", "Album", "AlbumArtists",
"Artists"}
if not need <= have:
sys.exit(f"{db} is not a Jellyfin library database with music columns"
f" (BaseItems has no {','.join(sorted(need - have))})")
# Provider ids, best identity first: the release group is the album, a bare
# release id is one reissue of it and never overrides a group.
mb = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
rank = {p: i for i, p in enumerate(MUSIC_PROVIDERS)}
best = {}
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in (?, ?)", MUSIC_PROVIDERS):
if iid not in best or rank[pid] < best[iid][0]:
best[iid] = (rank[pid], val)
mb = {k: v[1] for k, v in best.items()}
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
if not container:
container = "/data/" + os.path.basename(os.path.normpath(root))
prefix = container.rstrip("/") + "/"
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(MUSIC_COLS)
for (path, aal, arts, tag_album, tag_year, disc, track, name, iid,
alb_name, alb_year, alb_id) in con.execute(
MUSIC_SQL, (JF_ALBUM, JF_AUDIO, prefix + "%")):
if not path or not path.startswith(prefix):
skipped += 1 # a row for a file outside this library
continue
# The album artist owns the folder; the track artist is the fallback
# for the compilations whose album-artist tag was left empty.
w.writerow([root + "/" + path[len(prefix):],
((aal or "").strip() or (arts or "").strip()),
(alb_name or tag_album or "").replace("\t", " "),
alb_year or tag_year or "",
"" if disc is None else disc,
"" if track is None else track,
(name or "").replace("\t", " "),
mb.get(alb_id or "") or mb.get(iid or "") or "",
(arts or "").replace("\t", " ")])
written += 1
print(f"tracks={written} outside-root-or-blank={skipped} -> {out}")
def audiobooks(root, db, out):
"""The chapters Plex matched inside the audiobook library, read-only.
Plex stores HOST paths here (unlike Jellyfin, which stores the container's
view), so no prefix mapping is needed: the part's `file` is the path the
operator has today. Rows for files outside `root` are skipped, because one
Plex library DB carries every library on the box.
The chapter's `index` is exported as an empty field when Plex has none. That
is not a gap to fill in here: three quarters of the books on this box contain
at least one un-indexed chapter, so the relayout that reads this file must
not renumber anything from it.
"""
uri = f"file:{db}?mode=ro&immutable=1"
try:
con = sqlite3.connect(uri, uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
tables = {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}
missing = {"media_parts", "media_items", "metadata_items"} - tables
if missing:
sys.exit(f"{db} is not a Plex library database (no "
f"{', '.join(sorted(missing))})")
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(AB_COLS)
for file, narrator, book, year, track, chapter in con.execute(
AB_SQL, (root.rstrip("/") + "/%",)):
if not file or not file.startswith(root.rstrip("/") + "/"):
skipped += 1 # a row for a file outside this library
continue
w.writerow([file.replace("\t", " "), narrator.replace("\t", " "),
book.replace("\t", " "), "" if not year else year,
"" if track is None or int(track) < 0 else track,
chapter.replace("\t", " ")])
written += 1
print(f"chapters={written} outside-root={skipped} -> {out}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
"help": "library root"}
e = sub.add_parser("enumerate", help="stat every file under the root")
e.add_argument("--root", **root)
e.add_argument("--out", default="/tmp/media-enum.tsv")
p = sub.add_parser("probe", help="ffprobe every video container")
p.add_argument("--enum", default="/tmp/media-enum.tsv")
p.add_argument("--out", default="/tmp/media-probed.tsv")
p.add_argument("--jobs", type=int, default=6)
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
x.add_argument("--root", **root)
x.add_argument("--db", default=PLEX_DB)
x.add_argument("--out", default="/tmp/plex-movies.tsv")
j = sub.add_parser("jellyfin",
help="export the episodes Jellyfin has numbered for the root")
j.add_argument("--root", default=os.environ.get("MEDIA_ROOT",
"/mnt/entropy/TVShows"),
help="TV library root on the host")
j.add_argument("--db", default=JF_DB)
j.add_argument("--container", default=None,
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
m = sub.add_parser("music",
help="export the tracks Jellyfin has tagged for the root")
m.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"),
help="music library root on the host")
m.add_argument("--db", default=JF_DB)
m.add_argument("--container", default=None,
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
m.add_argument("--out", default="/tmp/jellyfin-music.tsv")
b = sub.add_parser("audiobooks",
help="export the chapters Plex matched for the audiobook root")
b.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"),
help="audiobook library root on the host")
b.add_argument("--db", default=PLEX_DB)
b.add_argument("--out", default="/tmp/plex-audiobooks.tsv")
a = ap.parse_args()
if a.cmd == "enumerate":
enumerate_lib(a.root, a.out)
elif a.cmd == "probe":
probe(a.enum, a.out, a.jobs)
elif a.cmd == "plex":
plex(a.root, a.db, a.out)
elif a.cmd == "music":
music(a.root, a.db, a.container, a.out)
elif a.cmd == "audiobooks":
audiobooks(a.root, a.db, a.out)
else:
jellyfin(a.root, a.db, a.container, a.out)
if __name__ == "__main__":
main()
+445
View File
@@ -0,0 +1,445 @@
#!/usr/bin/env python3
r"""media-music.py — a plan that re-files the music library into Jellyfin's own shape.
media-music.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Jellyfin's music contract (https://jellyfin.org/docs/general/server/media/music/)
is not the movie one. Jellyfin reads the metadata out of the file, so a folder is
an ALBUM and nothing else, every disc of an album belongs in that one folder, the
tags decide the track order, and a file name only matters when it carries a
character known to break playback (`< > : " / \ | ? *`) or when a lyric file has
to match the basename of the track it belongs to. The plan is therefore:
Music/<album artist>/<Album> (<year>)/NN - <Title>.<ext>
... the same folder, D.NN - <Title> when the album has discs
Music/Various Artists/... when its tracks name
more than one artist
The authority is the tag row Jellyfin already read — `scripts/media-census.py
music` — never the release-pack folder a ripper made. A file with no row is not
guessed at: it parks under `<root>/_unsorted` with the relative path it has today.
The rules below exist because of what a downloaded discography pack actually
contains:
* A cuesheet beside exactly one audio file means the album is one image with a
track list inside it. That file keeps its own name: numbering it `01` would
claim it is one track, and Jellyfin shows one track either way.
* Sidecars — images, cuesheets, EAC logs, checksums, booklets, playlists —
follow the album the audio in their own folder belongs to. A folder holding
two albums gives a sidecar no single owner, so it parks instead of being
guessed into one.
* A video container (.mkv, .vob, .ifo, …) inside a music library is not music.
Renaming it to .mka/.m4a would tell Jellyfin to index video as audio, so it
parks for a human decision.
* Playlists move with their album and their CONTENTS are never rewritten: the
paths inside an .m3u go stale. Editing a file the operator owns to keep a
convenience file tidy is not this tool's job.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What Jellyfin will index as music. .mp4/.mkv/.webm are deliberately NOT here:
# the vendor doc says an audio-only stream in those containers is never read as
# music, and this tool renames for a reason it can prove, not to guess at one.
AUDIO_EXT = {".flac", ".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav",
".wma", ".aiff", ".aif", ".ape", ".dsf", ".dff", ".wv", ".mka",
".mpc", ".aac"}
VIDEO_EXT = {".mkv", ".vob", ".ifo", ".bup", ".mpg", ".mpeg", ".avi", ".mov",
".ts", ".m2ts", ".wmv", ".webm", ".mp4", ".m4v", ".iso", ".img",
".bdmv", ".divx", ".ogm"}
# A lyric has to carry the basename of its track, so it is named after the audio
# file and not after itself. .txt counts only when its stem matches an audio stem:
# a loose .txt in an album folder is a note.
LYRIC_EXT = {".lrc", ".elrc", ".txt"}
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tiff"}
# The album-artist strings that mean "compilation", so such an album is not filed
# under one artist who does not own it.
VARIOUS = {"various", "various artists", "unknown", "unknown artist", "unknow"}
# The characters Jellyfin documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the tag row
PLACED = {} # audio path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
CUE_DIRS = set() # source dirs holding a .cue
AUDIO_BY_DIR = defaultdict(set) # source dir -> audio files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding audio
FOLDERS_SEEN = {} # (dir, depth) -> album folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its audio may sit
ALBUMS = {} # (artist, album, year) -> the album group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the tags mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing
unprintable, no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 90k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ---------------------------------------------------------------------- albums
def group_albums():
"""Every tagged track into one group per (album-artist tag, album, year).
The key is the identity the tags assert, not the folder the file sits in: one
album arrives as three discs in three folders, and two copies of an album in
one pack are one album to Jellyfin, whose album item is that name, artist and
year. The artist in the key is the ALBUM-artist tag and never the performer,
because a compilation is one album whose tracks name many performers — keying
on the performer would file one album once per artist.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above CD1/CD2 sees both
tag = (r["artist"] or "").strip()
album = (r["album"] or "").strip()
performer = (r["performer"] or "").strip()
if not album:
park(path, "a track no album names")
continue
if not tag and not performer:
park(path, "a track no artist names")
continue
key = (tag.casefold(), album.casefold(), (r["year"] or "").strip())
g = ALBUMS.setdefault(key, {"tags": [], "albums": [], "year": key[2],
"members": [], "performers": set(),
"mb": set()})
g["members"].append(path)
g["tags"].append(tag)
g["albums"].append(album)
g["performers"].add(performer or tag)
if r.get("mb"):
g["mb"].add(r["mb"])
def folder_artist(g):
"""Whose shelf the album sits on. The album-artist tag owns it when it names
one; when the tags name only performers, a single performer names the album
and several make it a compilation, which belongs to no one of them. The key is
case-folded, so the spelling that reaches the folder is the one the tracks
mostly carry — see dominant()."""
tags = [t for t in g["tags"] if t]
if tags:
name = dominant(tags)
return "Various Artists" if name.casefold() in VARIOUS else name
if len({p.casefold() for p in g["performers"]}) == 1:
return dominant(g["performers"])
return "Various Artists"
def album_folder(g):
name = safe(dominant(g["albums"])) or "Untitled"
return os.path.join(ARGS.root, safe(folder_artist(g)) or "Unknown Artist",
f"{name} ({g['year']})" if g["year"] else name)
def is_cue_image(path):
"""A folder holding this one audio file and a cuesheet is an album image."""
d = os.path.dirname(path)
return len(AUDIO_BY_DIR.get(d, ())) == 1 and d in CUE_DIRS
def number_of(value, width):
return f"{int(value):0{width}d}" if str(value or "").isdigit() else ""
def place_albums():
for key in sorted(ALBUMS):
g = ALBUMS[key]
folder = album_folder(g)
discs = {g0["disc"] for g0 in (ROWS[p] for p in g["members"]) if g0["disc"]}
tracks = [int(ROWS[p]["track"]) for p in g["members"]
if str(ROWS[p]["track"] or "").isdigit()]
# More than one disc, or a disc numbered past the first: the track number
# alone cannot name a file uniquely, so the disc goes in front of it.
multi = len(discs) > 1 or any(str(d).isdigit() and int(d) > 1 for d in discs)
width = 3 if multi or (max(tracks, default=0) > 99) else 2
for path in sorted(g["members"]):
r = ROWS[path]
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
if is_cue_image(path):
name = safe(base) or "image"
else:
title = safe(r["title"] or base) or "untitled"
num = number_of(r["track"], 2 if multi else width)
if multi and num:
num = f"{r['disc'] or '1'}.{num}"
name = f"{num} - {title}" if num else title
PLACED[path] = assign(path, folder, name, ext, "track",
"two tracks claimed one name inside one album")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The album folders the audio in `d`, or within `depth` directories below it,
was placed in. Memoised, because every loose file in a big discography folder
asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The album folder that owns this directory — the first place up the tree
where audio sits, in the directory itself or within SUBTREE_DEPTH levels below
it, which is how a box with its discs in CD1/CD2 owns the cover lying above
them. None when no audio is near enough to say, AMBIGUOUS when the audio there
belongs to more than one album."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_sidecars():
"""Everything that is not a tagged track: lyrics, images, cuesheets, logs,
booklets, playlists — and the clutter that is none of those."""
# The lyric rule needs the audio's stem as it is named TODAY, before the plan
# renames it, because that is what the sidecar's own name was copied from.
stem = {(d, os.path.splitext(os.path.basename(a))[0]): a
for d, audios in AUDIO_BY_DIR.items() for a in audios}
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d, base = os.path.dirname(path), os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the music library: Jellyfin will "
"not read it as music, and renaming it would claim it is")
elif ext in AUDIO_EXT:
park(path, "an audio file no provider row names — its tags are not in "
"the library Jellyfin has matched")
elif ext in (LYRIC_EXT | IMAGE_EXT) and (d, base) in stem:
# A lyric, or a picture named after its track: it follows that track
# and takes the name the track now has, or Jellyfin cannot find it.
audio = stem[(d, base)]
dst = PLACED.get(audio)
if dst:
folder, name = os.path.dirname(dst), os.path.splitext(
os.path.basename(dst))[0]
assign(path, folder, name, ext, "lyric",
"a sidecar whose name collided with another in its album")
else:
park(path, "a sidecar of a track the plan could not place")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a sidecar whose folder holds more than one album, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one album folder would have carried two files of the same "
"sidecar name")
else:
park(path, "not music, or music with nothing beside it to belong to")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Music relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
no_year = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if not g["year"])
if no_year:
fh.write(f"{len(no_year):>6} albums carry no year in any tag, so their folders\n"
" are named without one; Jellyfin still matches them.\n")
fh.write(" " + ", ".join(no_year[:300]) + "\n\n")
clash = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if len(g["mb"]) > 1)
if clash:
fh.write(f"{len(clash):>6} albums put one folder name on more than one\n"
" MusicBrainz release group — different releases, one name.\n"
" They share the folder Jellyfin's own album item names.\n")
fh.write(" " + ", ".join(clash[:200]) + "\n\n")
fh.write(" Playlists move with their album and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"))
ap.add_argument("--enum", default="/tmp/music-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/jellyfin-music.tsv",
help="the tag rows scripts/media-census.py music wrote")
ap.add_argument("--out", default="/tmp/music-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/music-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
for path, (_size, ext) in HELD.items():
if ext == ".cue":
CUE_DIRS.add(os.path.dirname(path))
group_albums()
place_albums()
place_sidecars()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a tag row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+881
View File
@@ -0,0 +1,881 @@
#!/usr/bin/env python3
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
[--out TSV] [--exceptions TXT]
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
metadata-cache export whose clean titles outrank the filename parser; the title
authority for everything Plex never matched is scripts/media-titles.py beside this
file. No media file is opened, so planning the whole library costs seconds.
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
parents exist before children. EXCEPTIONS is the human review list: every file the
plan could not name a film for, every collision it refused, the READMEs the
executor must write into the piles it created.
Target contract (jellyfin.org/docs/general/server/media/movies):
* one folder per movie, named Title (year)
* the video inside is named exactly like the folder
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
match the folder name character-for-character or Jellyfin splits them apart
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
* <>:"/\|?* break the matcher, so they never survive here
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
"""
import argparse
import collections
import csv
import importlib.util
import os
import re
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
# this file is the title authority. Nothing here opens a media file.
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="library root the plan moves files inside")
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
help="Plex metadata export (clean titles); absent = parser only")
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
return ap.parse_args()
ARGS = parse_args()
ROOT = ARGS.root
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
r"|(additional|extra) languages|languages|resources")
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
r"|(additional|extra) languages|commentar(y|ies)"
r"|audio commentary|audio tracks?")
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
".opus", ".ogg", ".wma"}
NFO_EXT = {".nfo"}
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
"banner": "banner", "logo": "logo", "clearlogo": "logo",
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
"discart": "poster", "disc": "poster"}
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
# library holds six of those films), so leaving it in a name turns ONE movie folder
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
# folders, none of them a movie.
BAD_CHARS = re.compile(r'[<>:"/|?*]')
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
"eng", "english", "italian", "dual", "audio", "forced", "default"}
# Canonical title where the library only ever carries a localised name.
OVERRIDES = {"28 giorni dopo": "28 Days Later",
"28 settimane dopo": "28 Weeks Later"}
rows = []
for line in open(ENUM):
size, ext, path = line.rstrip("\n").split("\t")
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
ext_by = {r["path"]: r["ext"] for r in rows}
size_by = {r["path"]: r["bytes"] for r in rows}
def rel(p):
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
def parts_of(p):
return [q for q in rel(p).split("/") if q]
def is_qb(p):
return ".!q" in p.lower()
def disc_root(p, fold=True):
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
ps = parts_of(p)
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
for i, q in enumerate(ps):
if q.upper() in kinds:
return ROOT + "/" + "/".join(ps[:i])
return None
def row_name(r):
"""The name a disc-tree entry should be identified by: the engine stores the
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
return r.get("name_src") or r["path"]
def sanitize(name):
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return (name[:180] or "untitled")
def ok_year(y):
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
def words_of(text):
"""The matching words of a name — or of a whole path, which is how callers
pass a file together with the folders it sits in.
Split on everything that is not a letter or a digit: eng.norm does not touch
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
film could not be found by the folder that holds it — which is how one box
set's featurettes ended up in another film's folder. Both sides of every
comparison below go through here. A bare number counts only when it could be
a release year, so release junk ("1080", "2160", "2", "7") never poses as
agreement between two names.
"""
out = set()
for w in WORD_SPLIT.split(eng.norm(text)):
if not w or w in eng.STOP:
continue
if w.isdigit() and not 1870 <= int(w) <= 2100:
continue
out.add(w)
return out
# ------------------------------------------------------------------ Plex titles
# Plex has matched a large part of the library's media files to real titles
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
# holds a MATCH, its title wins; the filename parser carries everything else
# (extras, disc trees, TV, and whatever Plex never matched).
def plex_titles():
if not os.path.exists(PLEX):
return {}
out = {}
with open(PLEX, newline="") as fh:
for r in csv.DictReader(fh, delimiter="\t"):
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
# Only a match is an authority, and a match carries a year: an
# unmatched row echoes the filename, which the parser reads better
# than a copied release name. The junk filter stays for the matches
# whose title still carries the release.
if t and ok_year(y) and not re.search(
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
out[r["file"]] = (t, y)
return out
PLEX_T = plex_titles()
# ------------------------------------------------------------- classification
erows = eng.load(PROBE)
for r in erows:
for k, v in OVERRIDES.items():
if k in r["title"]:
r["title"] = v
r["key"] = eng.key_of(r["title"])
r["base"] = eng.base_words(r["title"], r["key"])
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
extras = {r["path"] for r in erows
if r["path"] not in marks
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
matched=r["path"] in PLEX_T)}
eps = collections.defaultdict(list)
for r in erows:
if r["path"] in marks | extras:
continue
ep = eng.ep_of(r["path"])
if ep:
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
# A folder INSIDE a movie's folder is bonus territory even when its own name is
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
# misspelled Extras folder, so the engine's folder rule never fired and the
# featurette became a movie folder of its own — whose words then out-scored the
# real Saboteur and drew that box set's other extras into it. The rule needs no
# metadata: the folder holding the longest film owns the shorter video filed
# under it, provided that shorter file's own name speaks the film's name — a
# whole other film filed there stays a film, exactly as the engine's folder rule
# exempts it. A different year in the file's own name IS that other film
# speaking, so it vetoes the rule.
def longest_of(rows):
"""The longest video in each folder: that folder's feature."""
out = {}
for r in rows:
if r["dur"] > 0:
d = os.path.dirname(r["path"])
if d not in out or r["dur"] > out[d]["dur"]:
out[d] = r
return out
def not_disc(r):
"""A disc tree's innards belong to the tree, never to the folder's film."""
return not any(c in DISC_DIRS for c in r["path"].split("/"))
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
def under_a_feature(path, dur):
"""True when a folder above this file holds a longer film that this file's
own name names."""
d = os.path.dirname(os.path.dirname(path))
while len(d) > len(ROOT):
f = FEATURE_OF.get(d)
if f:
if f["dur"] < dur:
return False
theirs = (words_of(os.path.basename(f["path"]))
| words_of(os.path.basename(d)))
if not words_of(os.path.basename(path)) & theirs:
return False
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
return not (y1 and y2 and y1.group() != y2.group())
d = os.path.dirname(d)
return False
extras |= {r["path"] for r in erows
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
buckets = collections.defaultdict(list)
for r in film_rows:
buckets[eng.key_of(r["title"])].append(r)
def res_label(r):
m = RES_RE.search(r["res"] or "")
if m:
n, kind = m.group(1).lower(), m.group(2).lower()
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
try:
w = int((r["res"] or "0x0").split("x")[0])
except (ValueError, IndexError):
w = 0
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
def version_label(r):
bits = [res_label(r)]
vc = (r["vcodec"] or "").lower()
if vc in ("av1", "libaom", "libsvtav1"):
bits.append("AV1")
elif vc in ("vc1", "wmv3"):
bits.append("VC1")
if r["bytes"] > 40 * 2**30:
bits.append("REMUX")
if "imax" in r["path"].lower():
bits.append("IMAX")
return sanitize(" ".join(bits))
def group_name(items):
titles = collections.Counter(r["title"] for r in items)
ascii_t = [t for t in titles if t.isascii()]
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
years = [r["year"] for r in items if r["year"] != "?"] or \
[r["soft"] for r in items if r["soft"]]
year = collections.Counter(years).most_common(1)[0][0] if years else None
return sanitize(f"{title} ({year})" if year else title)
groups = []
for key, items in buckets.items():
for c in eng.dur_clusters(items):
g = {"rows": c, "base": group_name(c)}
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
if known:
# one film, one name: a Plex match for any copy names every copy
t, yr = collections.Counter(known).most_common(1)[0][0]
g["plex"] = t
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
groups.append(g)
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
for g in groups:
if g["base"] in clash:
# two genuinely different films share title+year: separate folders, and
# say so in the name, because Jellyfin can only match one item per folder
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
g["flag"] = "same title and year as another film"
moves, mkdirs, problems, readmes = [], set(), [], []
# --------------------------------------------------------------- unnamed groups
# A movie folder has to READ as a title. The engine's runtime veto says any file
# over 45 minutes that is not named like an extra is a film whatever it is called
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
# with no identity whose numeric keys went on to capture every subtitle named
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
# real film), so the gate is: a year, or a word that is not release noise. With
# neither, the rows are bonus material until a human names them.
def named(g):
if re.search(r"\(\d{4}\)$", g["base"]):
return True
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
unnamed = [g for g in groups if not named(g)]
groups = [g for g in groups if named(g)]
for g in unnamed:
for r in g["rows"]:
extras.add(r["path"])
problems.append(
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
f"bonus pile, not a folder — give the file a real name and re-plan")
group_of = {}
for g in groups:
g["dir"] = f"{ROOT}/{g['base']}"
for r in g["rows"]:
group_of[r["path"]] = g
dr = disc_root(r["path"])
if dr:
group_of[dr] = g
by_path = {r["path"]: r for r in erows}
# -------------------------------------------------------------- bookkeeping
claimed = {}
existing = set(rows_map := {r["path"] for r in rows})
at_target, booked_src = set(), set()
def book(src, dst, kind):
if src == dst:
at_target.add(src)
booked_src.add(src)
return
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
why = "rebook" if src in booked_src else "collision"
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
if src in booked_src: # one source may only ever move once
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
return
if dst.lower() in claimed:
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
return
claimed[dst.lower()] = src
booked_src.add(src)
comp = dst.split("/")
for i in range(1, len(comp)):
mkdirs.add("/".join(comp[:i]))
moves.append((src, dst, kind))
def uniq_dir(d, stem, ext, hint=""):
"""A destination that no other source already took."""
cand = f"{d}/{stem}{ext}"
if cand.lower() not in claimed:
return cand
n = 2
while True:
cand = f"{d}/{stem} {n}{ext}"
if cand.lower() not in claimed:
return cand
n += 1
def media_dir(p):
dr = disc_root(p)
return os.path.dirname(dr) if dr else os.path.dirname(p)
# ------------------------------------------------------------------ films
# film / disc moves: the largest copy carries the folder's own name, the rest
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
def vext(p):
return ext_by.get(p) or os.path.splitext(p)[1].lower()
disc_children = collections.defaultdict(list)
for q in rows:
dr = disc_root(q["path"])
if dr:
disc_children[dr].append(q["path"])
for g in groups:
primary_taken = False
disc_kinds = set()
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
dr = disc_root(r["path"])
if dr:
inner = os.path.basename(dr).upper()
if inner in disc_kinds:
problems.append(
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
f"share the folder — discs have no multiple versions")
continue
disc_kinds.add(inner)
for src in disc_children[dr]:
tail = os.path.relpath(src, dr)
fold = disc_root(src, fold=False)
if fold != dr and os.path.dirname(src) != dr:
tail = os.path.basename(dr) + "/" + tail
book(src, f"{g['dir']}/{tail}", "disc-file")
continue
label = "" if not primary_taken else f" - {version_label(r)}"
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
n = 2
while dst.lower() in claimed:
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
n += 1
r["dst"] = dst
primary_taken = True
book(r["path"], dst, "film")
g["primary"] = primary_taken
# ------------------------------------------------------------------ extras
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
("interviews", r"interview|cast and crew|session piece|commentary"),
("featurettes", r"featurette|behind the scenes|making of|documentary"
r"|heritage|archival|video archive|music video"),
("samples", r"sample"),
("shorts", r"\bshort\b|short film"),
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
r"|audition|test footage|watermark|workprint|poster"
r"|picture in picture|singalong|karaoke|original score")]
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
r"|\bost\b|music|song", re.I)
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
def extras_kind(path):
low = path.lower()
if THEME_RE.search(low):
return "theme-music"
if BACKDROP_RE.search(low):
return "backdrops"
for name, rx in EXTRA_KIND:
if re.search(rx, low):
return name
return "extras"
def attachable_title(path):
t, _y = eng.title_of(path)
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
for g in groups:
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
top_groups = collections.defaultdict(list)
for g in groups:
for t in g["tops"]:
top_groups[t].append(g)
group_by_key = collections.defaultdict(list)
for g in groups:
for r in g["rows"]:
group_by_key[eng.key_of(r["title"])].append(g)
group_by_dir = {}
for g in groups:
for r in g["rows"]:
group_by_dir.setdefault(media_dir(r["path"]), g)
dr = disc_root(r["path"])
if dr:
group_by_dir.setdefault(os.path.dirname(dr), g)
def group_words(g):
"""The words that identify a film group: its titles and the names of the
folders its files live in."""
if "words" not in g:
w = set()
if g.get("plex"):
w |= words_of(g["plex"])
for r in g["rows"]:
w |= words_of(r["title"])
w |= words_of(os.path.basename(media_dir(r["path"])))
g["words"] = w
return g["words"]
def score_words(text, groups):
"""Ranks film groups by how many of their words a path carries."""
words = words_of(text)
out = [(len(words & group_words(g)), g) for g in groups]
out.sort(key=lambda x: -x[0])
return out
def group_for_orphan(path):
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
several films must be resolved by WORDS, never by position: taking the first
film of the folder attached every loose subtitle in a box set to whichever
movie happened to sort first."""
stem = os.path.splitext(os.path.basename(path))[0]
key = attachable_title(path)
if key and group_by_key.get(key):
cands = group_by_key[key]
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
pool = same_top or cands
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
pool = top_groups.get(gtop, [])
if len(pool) > 1: # a loose collection folder (box set, bonus
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
return g if n >= 2 else None # album) is resolved by words, not position
if len(pool) == 1:
return pool[0]
d = os.path.dirname(path)
while d and d != ROOT:
if d in group_by_dir:
return group_by_dir[d]
d = os.path.dirname(d)
return None
tv_tops = {parts_of(p)[0] for p in tv_set}
def film_folder_for(path):
"""A movie folder for a lone feature that has no group: derived from its own
title when a year proves it, else None (goes to the review pile)."""
t, y = eng.title_of(path)
if not t or y == "?":
return None
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
mkdirs.add(d)
return d
for p in sorted(extras):
if disc_root(p):
continue
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
continue # a special inside a TV set: staged with it
g = group_for_orphan(p)
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
stem = os.path.splitext(os.path.basename(p))[0]
if g is None:
solo = film_folder_for(p)
if solo:
dst = f"{solo}/{sanitize(stem)}{ext}"
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
f"{os.path.basename(solo)}")
else:
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
problems.append(f"NO-FILM extra {rel(p)}")
else:
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
ext, os.path.dirname(p)), "extra")
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
# they belong to the film of the folder they sit in, kept as a named subtree.
for q in rows:
pth = q["path"]
dr = disc_root(pth)
if not dr or pth in by_path or pth in booked_src:
continue
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
if g is None:
continue
inner = os.path.basename(dr)
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
f"(a second master: compare before keeping both)")
for src in disc_children[dr]:
if src not in booked_src:
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
# ----------------------------------------------------------------- sidecars
media_by_dir = collections.defaultdict(list)
for r in erows:
if r["path"] in marks:
continue
dst = r.get("dst")
if dst:
media_by_dir[media_dir(r["path"])].append(r)
def sidecar_target(path, e):
d = os.path.dirname(path)
stem = os.path.splitext(os.path.basename(path))[0]
if d == ROOT:
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
# set of root-level MOVIE files, and scoring a stray name against it
# attached every root-level leftover to whichever root-level film was
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
return None, None
cands = media_by_dir.get(d, [])
if not cands:
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
while nd and nd != ROOT and not media_by_dir.get(nd):
nd = os.path.dirname(nd)
if media_by_dir.get(nd):
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
g = group_for_orphan(path) # box set's own root) carries evidence
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
if g is None or g["dir"] not in dirs:
# ...but only a STRONG word match outranks the folder the file
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
# against a junk group named "3" and escaped its own movie's
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
# and genuinely belongs to the other film of the set.
n = score_words(stem, [g])[0][0] if g else 0
if n >= 2:
return (g, None)
cands = media_by_dir[nd]
if not cands:
g = group_for_orphan(path)
return (g, None) if g else (None, None)
if len(cands) > 1:
w = [t for t in eng.norm(stem).split() if t not in NOISE]
cands = sorted(cands, key=lambda r: (
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
-r["bytes"]))
return (None, cands[0])
for q in rows:
p, e = q["path"], q["ext"]
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
continue
if os.path.basename(p).startswith("._"):
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
# dead files through four movie folders (and they are video extensions in
# the AppleDouble case, so they also skipped every other filter). They
# claim nothing: one pile, and they stay in it — nothing is deleted here.
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
"", os.path.dirname(p)), "junk")
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
continue
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
g = group_for_orphan(p)
stem = os.path.splitext(os.path.basename(p))[0]
if g is not None:
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
else f"{g['dir']}/{g['base']}{e}", "iso")
else:
solo = film_folder_for(p)
if solo:
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
f"folder -> {os.path.basename(solo)}")
else:
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
problems.append(f"NO-FILM disc image {rel(p)}")
continue
if e in VIDEO_EXT:
# A video the engine never probed (unreadable header) and not an AppleDouble
# remnant: the plan does not guess at it, so it stays where it is.
continue
stem = os.path.splitext(os.path.basename(p))[0]
g, media = sidecar_target(p, e)
if media is not None:
base = os.path.splitext(os.path.basename(media["dst"]))[0]
d = os.path.dirname(media["dst"])
elif g is not None:
base, d = g["base"], g["dir"]
else:
# Nothing names this file's film. NEVER scatter it: an archive set or an
# album tree must move as the unit it is, so the whole loose folder goes
# to the review pile and a film folder's own leftovers stay with it.
ps = parts_of(p)
if len(ps) < 2:
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
elif len(top_groups.get(ps[0], [])):
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
else:
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
book(dst=dst, src=p, kind="orphan")
problems.append(f"NO-OWNER sidecar {rel(p)}")
continue
if e in IMG_EXT:
low = eng.norm(stem)
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
kind = "poster" if word else next(
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
if re.search(r"\btheme\b", low):
dst = f"{d}/theme{e}"
else:
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
e, os.path.dirname(p)), "image")
elif e in NFO_EXT:
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
elif e in SUB_EXT | AUDIO_EXT:
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
langs = [t.lower() for t in toks if t.lower() in LANGS]
flags = [t.lower() for t in toks if t.lower() in FLAGS]
keep = [sanitize(t) for t in toks
if t.lower() not in langs + flags + list(NOISE)
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
if e in SUB_EXT:
# subtitles: <movie>/srt[/language]
d = f"{d}/srt"
pd = os.path.basename(os.path.dirname(p))
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}" # a real language folder survives
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
d = f"{d}/theme-music" # bonus audio: one canonical folder
pd = os.path.basename(os.path.dirname(p))
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
d = f"{d}/{sanitize(pd)}"
else:
pd = os.path.basename(os.path.dirname(p))
if not AUDIO_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}"
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
else:
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
os.path.dirname(p)), "misc")
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
"one-folder-per-movie layout; `_release-group watermarks/`, "
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
for key in tv_keys:
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
"these are TV episodes found in the MOVIES library; move the folder "
"to a TV library source, else Jellyfin groups them wrongly as one "
"movie.")
# --------------------------------------------------------------------- TV
for key in tv_keys:
items = eps[key]
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
yr = year.most_common(1)[0][0] if year else None
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
tops = {parts_of(r["path"])[0] for r in items}
members = sorted([r["path"] for r in items]
+ [q for q in extras | marks if len(parts_of(q)) > 1
and parts_of(q)[0] in tops],
key=lambda q: q)
for src in members:
rel_parts = rel(src).split("/")
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
book(src, f"{ROOT}/{base}/{tail}", "tv")
for p in sorted(marks):
stem, ext = os.path.splitext(os.path.basename(p))
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
os.path.dirname(p)), "watermark")
for q in rows:
if is_qb(q["path"]):
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
book(q["path"], q["path"], "qb-partial") # stays: resume data
# --------------------------------------------------------------- debug hook
if os.environ.get("JF_ISO"):
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
raise SystemExit
if os.environ.get("JF_DEBUG"):
import json
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
print("at_target", len(at_target))
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
for up in unp[:20]:
d = os.path.dirname(up)
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
# -------------------------------------------------------------- validation
srcs = [m[0] for m in moves]
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
for s in dup_src:
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
all_src = set(srcs)
untouched = {r["path"] for r in rows} - all_src
bad_names, into_alive, lost = [], [], []
for src, dst, kind in moves:
for comp in dst[len(ROOT) + 1:].split("/"):
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
or len(comp) > 200):
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
if dst in untouched: # something staying put is there
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
if dst.lower() in {s.lower() for s in untouched}:
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
if dst.startswith(src + "/") and src in all_src:
lost.append(f"{rel(src)} moves into its own subtree")
# every accounted-for source must exist exactly once in the plan
covered = {r["path"] for r in rows}
not_planned = sorted(covered - all_src - at_target)
with open(PLAN, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
if d.startswith(ROOT + "/") and d != ROOT:
out.write(f"mkdir\t\t{d}\tdir\n")
for p in sorted(booked_src - {m[0] for m in moves}):
out.write(f"keep\t{p}\t{p}\n")
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
out.write(f"move\t{src}\t{dst}\t{kind}\n")
kinds = collections.Counter(k for _s, _d, k in moves)
films_n = sum(1 for _s, _d, k in moves if k == "film")
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
with open(EXCEPTS, "w") as out:
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
out.write(f"kinds={dict(kinds)}\n")
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
out.write("\n".join(sorted(set(readmes))) + "\n")
out.write("\n=== PROBLEMS ===\n")
out.write("\n".join(sorted(set(problems))) + "\n")
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+ "\n".join(sorted(set(into_alive))) + "\n")
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
f"mkdirs={len(mkdirs)} film-files={films_n}")
print(f"kinds={dict(kinds)}")
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
print(f"left-unmoved={len(not_planned)}")
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
f"disc-trees={len(disc_children)}")
+1009
View File
File diff suppressed because it is too large. Load diff
+579
View File
@@ -0,0 +1,579 @@
#!/usr/bin/env python3
r"""The fleet's release-filename title parser: the grouping authority the relayout
plan uses for every file the media server never matched.
report: media-titles.py report [--probed TSV] [--out TSV]
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
ffprobe census scripts/media-census.py writes.
parse: media-titles.py parse NAME [NAME…]
what title_of() makes of a name — the way to check a naming accident
before it merges two films into one folder.
scripts/media-plan.py imports this file; over there the paths are arguments.
Method — three independent signals, so a naming accident cannot merge two films:
* franchise KEY : first two significant title tokens. Stable across parts of a
collection ("resident evil", "jump street") but NOT film-specific — it only
selects candidates.
* DURATION : the identity check. Same master encodes differ by at most a
couple of seconds; different films / cuts do not collide.
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
are byte-twins (the hash pass confirms); same duration but different sizes are
two different releases of one film (deletable redundancy).
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
would otherwise collapse a whole series into one "duplicate" group.
"""
import argparse
import collections, os, re, sys, unicodedata
# Library root: the folder directly under this one names the film, and every
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
# their own; nothing here reads media, only the census TSVs about it.
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
# Release-tag vocabulary: never part of a title, so the title stops here.
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
JUNK_RE = re.compile(
r"^("
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
r")$", re.I)
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
"vs", "avec", "con"}
def norm(s):
s = unicodedata.normalize("NFKD", s)
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
def fold_sep(s):
"""norm(), plus: every character that is left over is not a letter, digit or
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
of the word. Release groups file titles with lookalikes for punctuation the
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
those (no decomposition mapping), and the scrubber inside title_of DELETES
what it does not recognise, so five different V/H/S films squashed into the one
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
they separate. Accents are gone by then — norm() removed the combining marks —
so "Amélie" stays "amelie" and never splits into "ame lie"."""
return " ".join("".join(c if c.isalnum() or c == "'" else " "
for c in norm(s)).split())
def title_of(path):
"""(title, year) from the basename.
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
(2) Find the boundary: the first release-noise token (resolution / codec /
source / audio / language / group vocabulary) ends the title, so trailing
tags can never leak in. (3) Collect up to that boundary: leading index
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
are skipped while nothing is collected; once a real word appears a sentinel
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
"""
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
# docstring): the tokens below are then plain alnum/apostrophe words.
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
raw = [x for x in SEP_RE.split(name) if x]
years = [t for t in raw if YEAR_RE.match(t)]
two_year = len(years) > 1 # "2001…1968": one of them is the title
toks, year = [], None
for t in raw:
if YEAR_RE.match(t):
year = t # last year wins
toks.append("\x00")
else:
toks.append(t)
stop, seen_word = len(toks), False
for i, t in enumerate(toks):
if t == "\x00":
continue
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
stop = i
break
if c.isalpha():
seen_word = True
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
# it is three digits or more ("007", "000", "1408") or the name carries no year
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
# loses its zero to the parser — Plex's matched title overrides it anyway.
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
keep, num_title = [], None
for t in toks[:stop]:
if t == "\x00":
if keep:
break # year closes the title
if two_year and t is not None and years[0] != year:
# two DIFFERENT years ("2001…1968"): the leading one is the title;
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
num_title = num_title or years[0]
continue # leading year: not the title
if track_idx and IDX_RE.match(t) and not keep and t == lead:
continue # "03." / "1c." ordering
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if not keep and not c.isalpha():
num_title = num_title or (c or None)
continue # numeric title candidate
keep.append(c)
if len(keep) >= 8:
break
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
title = " ".join(keep)
if not keep:
# nothing collected: a pure-numeric leading token is the title even when it
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
for t in raw:
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
return c, year or "?" # never "2160p" / "1080": they are not pure digits
if (num_title and title and not keep[0].isdigit()
and (two_year or not track_idx)):
# a leading number that only echoes the release year ("1917.2019.2160p…") is
# not a title; one that stands alone ("300.2006…", "1900") is
title = f"{num_title} {title}"
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
return title, year or "?"
# Bonus-material vocabulary: names that say nothing about the film, so an extras
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
# Clips) — and the running-time guard in is_extra() carries the rest.
SEP_CLS = r"[\s._-]+"
def _bonus_rx(word):
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
EXTRA_WORDS = [
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
"tv spot", "advert", "commercial", "behind the scenes", "making of",
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
"menus", "gallery", "still gallery", "stills", "poster", "posters",
"montage", "preview", "previews", "epilogue", "visual effects",
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
"test footage", "screen test", "audition", "cast and crew", "isolated score",
"picture in picture", "original score", "session piece", "sample",
"watermark",
]
# Words that describe bonus material which can run LONG (a feature-length
# commentary-track cut, a documentary about the film). Trusted only below
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
"archival", "documentary"]
EXTRA_NAME_RE = re.compile(
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
EXTRA_LONG_NAME_RE = re.compile(
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
# search = it only mentions extras ("... + Extras" box sets hold feature films).
EXTRA_FOLDER_RE = re.compile(
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
# Anything longer than this is a feature film whatever it is called: a 118-minute
# file named "The Menu" or "Trailer" is a movie, not bonus material.
SHORT_MAX = 45 * 60
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
# commentary cut is an alternate version of the film, not a featurette.
LONG_BONUS_MAX = 25 * 60
def name_words(x):
"""Bare words of a file or folder name: no release metadata, no year."""
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
return [w for w in SEP_RE.split(fold_sep(x))
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
and not JUNK_RE.match(w) and w not in JUNK]
def mentions(short, long_):
"""True when one word list appears in the other, anywhere: a release names
the film at the front ("Alien (1979) - Trailer") or in the middle
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
a, b = sorted([short, long_], key=len)
if not a:
return False
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
def folder_is_bonus(folder):
"""True when a folder name carries bonus vocabulary as WHOLE words: the
vocabulary is matched against a padded word string, so "Extras" decides and
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
"""True for bonus material, which must never be counted as a film: a file
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
extras otherwise pair with each other across the whole library.
Evidence, strongest first:
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
unless it sits in an explicit bonus folder and metadata never matched it.
Bonus words that describe long-form material (commentary, documentary,
interview) are believed only
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
* The file's OWN name carries bonus vocabulary -> bonus.
* ANY folder of the path below the film's own root folder carries bonus
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
is exempt because it names the film, and a film may be titled with a
bonus word; and a file whose own name IS that folder's film name is the
feature, not the bonus ("Ghost in the Shell Movies/Extras/
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
there stays a film).
Bonus vocabulary is matched as whole words only, so the film
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
the folder rule, and "…Extras" as a fragment of a word is never evidence.
"""
parts = [q for q in path.split("/") if q]
if not parts:
return False
name = (os.path.splitext(parts[-1])[0]
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
# A bonus folder anywhere BELOW the film's own folder decides for its
# contents; the film's own folder is the film's name, whatever words it
# contains, so it is exempt. It is the first component AFTER ROOT's own
# components — those are the mount path, not folders of the library.
head = len([q for q in root.split("/") if q]) + 1
bonus_folder = False
for folder in parts[head:-1]:
if folder_is_bonus(folder):
# The folder's words are often the film's own name ("Angel's Egg
# Extras/"), and then they are not bonus evidence. Only WORDS can name
# a film: "extras" is release junk that name_words drops, so a folder
# like "01. Extras" reduces to its index — and an index matching an
# index ("NCOP 01") is a coincidence, not a title.
nw = [w for w in name_words(name) if not w.isdigit()]
fw = [w for w in name_words(folder) if not w.isdigit()]
bonus_folder = not fw or not mentions(nw, fw)
break
if dur and dur > SHORT_MAX:
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
# it as a film. Long-form bonus words still describe a film, so a
# feature-length commentary cut or documentary keeps its runtime veto.
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
return True
return False
if EXTRA_NAME_RE.search(name):
return True
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
return True
return bonus_folder
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
# "RARBG.com.mp4") into many different films, so its name carries no film
# identity at all and it would otherwise pair with itself across the library.
WATERMARK_MAX = 20 * 2**20
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
def is_watermark(path, size):
"""True for a release group's own watermark video, not a film or an extra."""
return (size < WATERMARK_MAX
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
def key_of(title):
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
Deliberately NOT film-specific — it only selects candidates for the duration
check, so a two-token collision (two films starting "star trek") cannot by
itself create a duplicate group."""
toks = [t for t in title.split() if t]
words = [t for t in toks if not t.isdigit()]
if not words:
return " ".join(toks)
sig = [t for t in words if t not in STOP] or words
return " ".join(sig[:2])
def ep_of(path):
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
The series is the name BEFORE the marker. The box-set convention writes
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
follows the marker is not the show's name: keying on the whole title gave
every episode of one series its own group, so a 124-episode box set never
reached the size that calls something a series and each episode was filed
as its own film. A file named only by its marker has nothing before it, so
its whole title stands in."""
name = os.path.splitext(os.path.basename(path))[0]
m = EP_RE.search(name)
if not m:
return None
head = name[:m.start()].strip(" ._\u2013-")
series = title_of(head)[0] if head else None
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
def base_words(title, key):
"""Title tokens beyond the franchise key — the FILM identity part.
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
kt = set(key.split())
return frozenset(t for t in title.split() if t not in kt and t not in STOP
and not t.isdigit() and not YEAR_RE.match(t))
def folder_year(path):
"""Soft year evidence: a 19xx/20xx token in any parent directory name
(collection folders are written both "(2002)" and "[2009]"; a range such as
"(1998-2000)" deliberately yields nothing)."""
for d in reversed(os.path.dirname(path).split("/")):
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
if len(ys) == 1:
return ys[0]
return None
def same_film(a, b):
"""Two files are copies of ONE film only when the film-identity tokens agree
(one side may carry none) AND no year evidence contradicts them. The key says
the franchise matches; this says the actual film matches — it is what keeps
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
Powers apart when their runtimes happen to agree."""
ba, bb = a["base"], b["base"]
if ba and bb and ba != bb:
return False
ya = a["year"] if a["year"] != "?" else a["soft"]
yb = b["year"] if b["year"] != "?" else b["soft"]
if ya and yb and ya != yb:
return False
return True
def dur_close(a, b):
if a <= 0 or b <= 0:
return True # unknown duration: let titles decide
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
# is folded into ONE entry, described by its longest main-feature clip (measured
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
BDMV_MAIN = {
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
}
def load(probed):
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
BDMV tree collapses to one entry carrying the tree's total size."""
trees = {}
rows = []
for line in open(probed):
f = line.rstrip("\n").split("\t")
if len(f) < 12:
continue
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
base = os.path.basename(p)
if base.startswith("._") or base == ".DS_Store":
continue # macOS sidecar, not media
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
root = p[:m.start()] if m else None
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
if not root:
continue # disc bookkeeping: folds into its tree, never a film
if root:
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
t["bytes"] += int(b)
t["audio"].update(a for a in aud.split("+") if a)
t["subs"] += int(subs or 0)
continue
try:
dur = float(dur)
except ValueError:
dur = 0.0
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
for root, t in trees.items():
folder = os.path.basename(root)
m = BDMV_MAIN.get(folder, {})
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
name_src=folder, dur=m.get("dur", 0.0),
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
fmt="BDMV tree"))
for r in rows:
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
r["key"] = key_of(r["title"])
r["base"] = base_words(r["title"], r["key"])
r["soft"] = folder_year(r["path"])
r["ep"] = ep_of(r["path"])
return rows
def dur_clusters(items):
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
buckets come from the franchise key; same_film then rejects the different
installments whose runtimes happen to agree."""
clusters = []
for r in sorted(items, key=lambda r: -r["dur"]):
for c in clusters:
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
c.append(r)
break
else:
clusters.append([r])
return clusters
def tier(groups):
"""Split same-film/same-duration clusters: DUP = different releases of one
film (real, deletable redundancy); REVIEW = identical size AND duration
(byte-twins — the hash pass is the authority) or clashing year evidence."""
dup, review = [], []
for g in groups:
g.sort(key=lambda r: -r["bytes"])
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
years.discard(None)
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
if len(years) > 1 or spread < 0.005:
review.append(g)
else:
dup.append(g)
return dup, review
def redun(g):
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
def line(r):
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
def dump(tag, groups, out, note=""):
for gi, g in enumerate(groups, 1):
g.sort(key=lambda r: -r["bytes"])
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
for rank, r in enumerate(g):
print(f" {line(r)} [{r['year']}] "
f"{r['path'].replace(ROOT + '/', '')}")
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
f"{r['path']}\n")
def report(probed, out_path):
rows = load(probed)
for r in rows:
r["mark"] = is_watermark(r["path"], r["bytes"])
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
marks = [r for r in rows if r["mark"]]
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
extras = [r for r in rows if r["extra"]]
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
f"TV episodes + {len(extras)} bonus/trailers "
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
f"+ {len(marks)} release-group watermarks")
out = open(out_path, "w")
buckets = collections.defaultdict(list)
for r in movies:
buckets[r["key"]].append(r)
cand = {k: v for k, v in buckets.items() if len(v) > 1}
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
dup, review = tier(clusters)
dup.sort(key=lambda g: -redun(g))
review.sort(key=lambda g: -redun(g))
dump("DUP", dup, out)
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
ebuf = collections.defaultdict(list)
for r in episodes:
ebuf[(r["ep"][0], r["ep"][1])].append(r)
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
for c in dur_clusters(v) if len(c) > 1])
edup.sort(key=lambda g: -redun(g))
dump("EP-DUP", edup, out)
shows = collections.Counter()
for r in episodes:
shows[r["ep"][0]] += 1
multi = {s: n for s, n in shows.items() if n > 4}
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
print(f" {n:4d} episodes {s}")
out.close()
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
print(f"REVIEW groups={len(review)} "
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
print(f"EP-DUP groups={len(edup)} "
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
def parse(names):
for n in names:
t, y = title_of(n)
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
rep.add_argument("--out", default="/tmp/media-similar.tsv")
par = sub.add_parser("parse", help="show the parsed title/year of a name")
par.add_argument("names", nargs="+")
a = ap.parse_args()
if a.cmd == "report":
report(a.probed, a.out)
else:
parse(a.names)
if __name__ == "__main__":
main()
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""media-tv.py — move TV episodes out of the movie library into a TV library.
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
`_TV needs a TV library - <series>` pile for a human. This is the other half of
that refusal. It walks the movie library, groups the files whose own name
carries an episode marker by SERIES, and writes a plan that places each series
in the shape both Jellyfin and Plex read without help:
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
It writes a plan and nothing else. Every rename belongs to
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
the executor's own path guard still has teeth, and give it its own journal —
one journal is one undo, and a TV move must not be welded to a movie relayout:
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py sweep --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
Guards, and why each one is here:
* The same five-file threshold as the planner. An episode-shaped name on one
file is not a series — `The Godfather Part 2` is a film — so a smaller group
is reported and left exactly where it is.
* An episode whose name carries no season number is reported and left, never
filed into an invented `Season 01`.
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
episode; an episode without its subtitles is a half-move.
* A group that would claim one destination twice is not resolved here. The
executor refuses the whole plan before its first rename, which is the right
outcome: nobody but Frank decides which copy of an episode is the keeper.
* An episode the destination library already holds at the same byte count is
left where it is. The TV library has held a complete Monk box set for years
while the movie library carried a second copy of all 124 episodes; a mover
that did not look where it was filing would have put both inside one library.
* A symlink is never followed and never moved.
"""
import argparse
import collections
import importlib.util
import os
import re
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="the movie library to empty of TV")
ap.add_argument("--library", default="/mnt/entropy/TVShows",
help="the TV library to fill (a different share or folder)")
ap.add_argument("--out", default=None,
help="plan TSV for scripts/media-apply.py"
" (default: media-tv-plan.tsv beside --root)")
ap.add_argument("--min", type=int, default=5,
help="episode files that make something a series")
a = ap.parse_args()
if not a.out:
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
"media-tv-plan.tsv")
return a
ARGS = parse_args()
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
BAD_CHARS = re.compile(r'[<>:"/|?*]')
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
# Sidecars that follow an episode; macOS forks and resume data do not.
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
# A box set that stamps the show's year into every file name; the folder's year
# must not be added on top of it.
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
"by", "de", "la", "vs"}
def clean(name):
"""A name the filesystem and both matchers can read (same law as the plan)."""
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return name[:180] or "untitled"
def episode_name(path):
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
already carries spaces is left exactly as it was written."""
stem, ext = os.path.splitext(os.path.basename(path))
if " " not in stem and stem.count(".") >= 2:
stem = stem.replace(".", " ")
return clean(stem) + ext.lower()
def head_of(path):
"""The publisher's series name: the text BEFORE the episode marker.
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
after the marker is not the show's name, and case is kept because it is the
library's own spelling, not a normalised key."""
name = os.path.splitext(os.path.basename(path))[0]
m = eng.EP_RE.search(name)
head = name[:m.start()] if m else name
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
def title_series(head):
"""The series folder in the case a TV library is written in.
Only a name with no capital at all is retitled: a box set that was
lower-cased on the way in would otherwise file `monk (2002)` among
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
exactly as it was written."""
if any(c.isupper() for c in head):
return head # the publisher wrote it with case; keep it
words = head.split()
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
for i, w in enumerate(words))
def season_episode(token):
"""(season, episode) an episode token states, or None when it states none."""
m = SEASON_RE.match(token)
if not m:
return None
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
def sidecars(path):
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
d, stem = os.path.split(path)
stem = os.path.splitext(stem)[0]
out = []
try:
entries = sorted(os.listdir(d))
except OSError:
return out
for e in entries:
p = os.path.join(d, e)
if p == path or os.path.islink(p) or not os.path.isfile(p):
continue
s, _ext = os.path.splitext(e)
if s == stem and not SKIP_SIDECAR.match(e):
out.append(p)
return out
def episodes_under(root):
"""(path, series, token) for every episode-shaped video file under root."""
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames
if not d.startswith(".") and d != "@eaDir")
for f in filenames:
p = os.path.join(dirpath, f)
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
continue
e = eng.ep_of(p)
if e:
yield p, e[0], e[1]
def walk(root, library):
"""Episode files under ROOT keyed by series, ignoring the destination."""
groups = collections.defaultdict(list)
lib = os.path.abspath(library)
for p, series, token in episodes_under(root):
if os.path.abspath(p).startswith(lib + os.sep):
continue
groups[series].append((p, token))
return groups
def holdings(library):
"""What the destination already carries: (series, season, episode) -> sizes.
The one expensive read here, and the one that matters. The movie library
holds a complete Monk box set that the TV library has held for years: 124
episodes, the same episode list, identical byte counts. Moving it would
have filed a second copy of a whole show into the library that already had
it. Byte count is the test on purpose — two encodes of one episode are two
files and both are wanted; the same bytes are the same file."""
out = collections.defaultdict(set)
for p, series, token in episodes_under(library):
se = season_episode(token)
if se is None:
continue
try:
out[(series, *se)].add(os.path.getsize(p))
except OSError:
continue
return out
def main():
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
groups = walk(root, library)
held_here = holdings(library)
mkdirs = {library}
rows, lines, skipped = set(), [], []
for series, items in sorted(groups.items()):
if len(items) < ARGS.min:
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
f"episode-shaped file{'s' if len(items) != 1 else ''},"
f" below {ARGS.min}: not a series")
continue
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
if y)
year = years.most_common(1)[0][0] if years else None
heads = collections.Counter(head_of(p) for p, _t in items)
head = heads.most_common(1)[0][0]
if year and YEAR_TAIL.search(head):
year = None
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
series_dir = os.path.join(library, name)
moved = held = 0
for path, token in items:
se = season_episode(token)
if se is None:
skipped.append(f"skipped: {path} — no season number in its name,"
" nothing here invents one")
continue
try:
size = os.path.getsize(path)
except OSError:
continue
if size in held_here.get((series, *se), ()):
held += 1
continue
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
mkdirs.update([series_dir, dst_dir])
base = episode_name(path)
rows.add((path, os.path.join(dst_dir, base)))
moved += 1
for side in sidecars(path):
rows.add((side, os.path.join(
dst_dir, clean(os.path.splitext(base)[0]
+ os.path.splitext(side)[1].lower()))))
if moved:
lines.append(f"{name}: {moved} episodes"
f"{'' if year else ' (no year in any folder name)'}"
f" -> {series_dir}")
if held:
lines.append(f"{name}: {held}"
f" {'episode' if held == 1 else 'episodes'} already in"
f" {library} at the same byte count — left there")
# An empty plan creates nothing, not even the library folder: a run that
# found no series must not be able to leave a stray directory behind.
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
with open(ARGS.out, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in dirs:
out.write(f"mkdir\t\t{d}\tdir\n")
for src, dst in sorted(rows):
out.write(f"move\t{src}\t{dst}\ttv\n")
for line in lines + skipped:
print(line)
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
if rows:
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
f" --root {os.path.dirname(root)}"
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
" (add --apply to run it)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env python3
"""Prove a relayout landed, using the media server's own tables.
A scan answers `204` whether or not it found anything, and a rename that the
scanner has not followed looks exactly like a rename that never happened from
the outside. The only honest proof is the run's journal replayed against the
database the server actually serves from: every `move` row must have a row at
its destination, and NO row may still name its source — a source row is a scan
that has not caught up, and it is the one signature of a library pointing at
files that no longer exist.
Two doors, both read-only, both without a lock while the server runs
(`mode=ro&immutable=1`; there is no `sqlite3` binary on these hosts):
jellyfin `BaseItems.Path`. The container binds the share at `/data`, so the
database holds `/data/<library>/…` where the journal holds
`/mnt/entropy/<library>/…`. Translate, or every row reads as missing.
plex `media_parts.file`, joined up to `metadata_items`. Plex is bound at
the host path, so its rows match the journal verbatim. Filter
`deleted_at is null` on BOTH tables — a deleted scan leaves its parts
behind, and one of those rows is not proof anything landed.
Neither server holds a row for everything the journal renames: a `.log`, `.cue`
or cover image has no row to land in, so its absence is not a fault. The report
therefore breaks the missing rows down by extension, and `--ignore-ext` names
the extensions this server does not index, which are then counted separately and
kept out of the verdict.
Exit codes: 0 every move landed; 1 something is stale or missing (the details are
printed); 2 the journal or the database is not usable at all.
"""
import argparse
import json
import os
import sqlite3
import sys
from collections import Counter
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
# Per kind: the table whose columns are required, those columns, and the query
# that returns every path the server believes it has.
KINDS = {
"jellyfin": {
"table": "BaseItems",
"columns": {"Path"},
"sql": "select Path from BaseItems",
"default_db": JF_DB,
},
"plex": {
"table": "media_parts",
"columns": {"file", "media_item_id", "deleted_at"},
"sql": ("select p.file from media_parts p"
" join media_items m on m.id = p.media_item_id"
" join metadata_items mi on mi.id = m.metadata_item_id"
" where p.deleted_at is null and mi.deleted_at is null"),
"default_db": PLEX_DB,
},
}
def die(msg):
print(f"media-verify: {msg}", file=sys.stderr)
sys.exit(2)
def read_moves(path):
"""The (src, dst) pairs of a journal, in the order the run performed them."""
moves = []
try:
fh = open(path, encoding="utf-8")
except OSError as e:
die(f"cannot read the journal {path}: {e.strerror}")
with fh:
for n, line in enumerate(fh, 1):
line = line.strip()
if not line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
die(f"{path} line {n} is not a journal record")
if rec.get("op") == "move":
if not rec.get("src") or not rec.get("dst"):
die(f"{path} line {n} is a move without both src and dst")
moves.append((rec["src"], rec["dst"]))
return moves
def server_paths(db, kind):
"""Every path the server holds, or a refusal that names what is missing."""
spec = KINDS[kind]
try:
con = sqlite3.connect(f"file:{db}?mode=ro&immutable=1", uri=True)
except sqlite3.Error as e:
die(f"cannot open {db}: {e}")
with con:
have = {r[1] for r in con.execute(f"pragma table_info({spec['table']})")}
if not spec["columns"] <= have:
die(f"{db} is not a {kind} library database ({spec['table']} has no "
f"{','.join(sorted(spec['columns'] - have))})")
try:
return {r[0] for r in con.execute(spec["sql"]) if r[0]}
except sqlite3.Error as e:
die(f"{db} refused the {kind} query: {e}")
def to_host(path, container, share):
"""The database's path in the journal's terms; identity when it is not a
container-side path."""
if not container:
return path
prefix = container.rstrip("/") + "/"
if path.startswith(prefix):
return share.rstrip("/") + "/" + path[len(prefix):]
return path
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--journal", required=True,
help="the run's journal (scripts/media-apply.py writes it)")
ap.add_argument("--db-kind", choices=sorted(KINDS), required=True)
ap.add_argument("--db", default=None, help="the server's database on this host")
ap.add_argument("--share", default="/mnt/entropy",
help="the share every journal path lives under")
ap.add_argument("--container", default=None,
help="what the share is called inside the container; empty"
" compares verbatim (default /data for jellyfin, nothing"
" for plex)")
ap.add_argument("--show", type=int, default=20,
help="how many offenders to name before saying 'and N more'")
ap.add_argument("--ignore-ext", default="",
help="comma-separated extensions this server does not index,"
" kept out of the verdict (e.g. log,cue,nfo,srt)")
args = ap.parse_args()
db = args.db or KINDS[args.db_kind]["default_db"]
container = ("/data" if args.db_kind == "jellyfin" else "") \
if args.container is None else args.container
ignore = set()
for ext in args.ignore_ext.split(","):
ext = ext.strip().lower()
if ext:
ignore.add(ext if ext.startswith(".") else "." + ext)
moves = read_moves(args.journal)
stored = {to_host(p, container, args.share) for p in server_paths(db, args.db_kind)}
landed, stale, missing, ignored = 0, [], [], 0
for src, dst in moves:
if os.path.splitext(dst)[1].lower() in ignore:
ignored += 1
continue
at_dst, at_src = dst in stored, src in stored
if at_dst and not at_src:
landed += 1
continue
if at_src:
stale.append(src)
if not at_dst:
missing.append(dst)
print(f"{args.db_kind}: {len(moves)} moves checked")
print(f"landed: {landed}")
print(f"still at src: {len(stale)}")
print(f"missing from the DB: {len(missing)}")
if ignored:
print(f"not indexed (extension ignored): {ignored}")
if missing:
exts = Counter(os.path.splitext(d)[1].lower() or "(no extension)"
for d in missing)
print("missing by extension: "
+ ", ".join(f"{e} {n}" for e, n in sorted(exts.items())))
for label, items in (("stale", stale), ("missing", missing)):
for path in items[:args.show]:
print(f" {label}: {path}")
if len(items) > args.show:
print(f" {label}: … and {len(items) - args.show} more")
return 1 if (stale or missing) else 0
if __name__ == "__main__":
sys.exit(main())
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env bash
# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
#
# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
# ONLY when it sits next to the media file. The media lives on the RAID6 array at
# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
# container, and slow to seek for a thousand tiny XML reads. So neither media
# server binds the array any more: they bind an OverlayFS merge of it with a
# writable layer on the NVMe.
#
# lowerdir /mnt/entropy RAID6 (ROTA=1) — source of truth, never written
# upperdir /var/lib/media-meta/upper NVMe — every sidecar and image byte
# workdir /var/lib/media-meta/work NVMe — kernel bookkeeping
# Where /var/lib/media-meta/view the merged tree jellyfin and plex bind
#
# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
#
# Consequences worth knowing:
# * A sidecar write is a few KB onto the SSD; the media bytes never move and the
# array is still never written by anything automated.
# * Deleting through the view writes a whiteout into the upper layer instead of
# deleting from the array. Nothing should delete through the view: deletions
# go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
# * inotify events on the lower layer do not cross the merge, so a file that
# arrives on the raw array is invisible to a file-system watcher in the view:
# trigger a library refresh (see the Media Relayout page).
#
# Subcommands
# plan [--upper-base D] [--lower D] print the resolved shape, touch nothing
# verify prove the live mount IS that shape
# install [--upper-base D] create the layers, install+enable+start the unit
# reap [--apply] [--upper-base D] [--lower D]
# list upper-layer entries whose media is gone
# status one line per fact a human wants
#
# install must run as root (it writes /etc/systemd/system and mounts); the others
# are read-only and safe unprivileged.
set -euo pipefail
LOWER=/mnt/entropy
UPPER_BASE=/var/lib/media-meta
VIEW=$UPPER_BASE/view
# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
# repo copy keeps a readable name; install writes it under the escaped one.
UNIT='var-lib-media\x2dmeta-view.mount'
UNIT_DIR=/etc/systemd/system
# The rootless containers write as this host uid (the deploy user). The upper
# layer has to accept those writes, so it is owned by them.
MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
die() {
printf 'media-view: %s\n' "$*" >&2
exit 1
}
usage() {
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
exit 2
}
cmd="${1:-}"
[ -n "$cmd" ] || usage
shift || true
upper_base="$UPPER_BASE"
lower="$LOWER"
apply=0
while [ $# -gt 0 ]; do
case "$1" in
--upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
--lower) [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
--apply) apply=1; shift ;;
*) usage ;;
esac
done
upper="$upper_base/upper"
work="$upper_base/work"
# The device backing a path, and whether the kernel calls it rotational.
rotational_of() {
local src rota
src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
[ -n "$src" ] || die "cannot find a block device behind $1"
rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
[ -n "$rota" ] || die "cannot read the rotational flag of $src"
printf '%s %s\n' "$src" "$rota"
}
require_upper_base() {
[ -d "$upper_base" ] \
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
}
do_plan() {
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
}
do_verify() {
local fail=0 fst opts src rota
if ! mountpoint -q "$VIEW"; then
printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
printf ' library that scans itself empty, not a metadata view.\n' >&2
return 1
fi
fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
if [ "$fst" != overlay ]; then
printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
fail=1
fi
opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
local want
for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
case ",$opts," in
*",${want},"*) ;;
*) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
fail=1 ;;
esac
done
read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
if [ "$rota" != 0 ]; then
printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
printf ' belongs on the SSD, not on the array.\n' >&2
fail=1
fi
if [ ! -w "$UPPER_BASE/upper" ]; then
printf 'refused: %s is not writable\n' "$upper" >&2
fail=1
fi
[ "$fail" -eq 0 ] || return 1
printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
}
do_install() {
[ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
require_upper_base
mountpoint -q "$LOWER" \
|| die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
local src rota
read -r src rota <<<"$(rotational_of "$upper_base")"
[ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
[ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
# systemd will not create a mount point; a missing directory fails the start.
install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
install -d -m 0700 "$upper_base/.AGENTTRASH"
install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
systemctl daemon-reload
systemctl enable "$UNIT" >/dev/null 2>&1 || true
systemctl start "$UNIT"
do_verify
}
# An upper-layer DIRECTORY with no counterpart below is metadata for a media
# folder that has been moved or renamed on the RAW path — a ghost the media
# servers would still browse. Files are never candidates on their own: a sidecar
# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
# is for. A copy-up whose lower file was deleted by hand is the one case this
# leaves in place; the raw path is where deletions belong, so it stays visible.
do_reap() {
[ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
count=0
ghosts=0
reap_walk ""
if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
printf 'reap: no ghosts under %s\n' "$upper"
return 0
fi
if [ "$ghosts" -gt 0 ]; then
printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
"$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
printf ' Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
printf ' Deletions belong in .AGENTTRASH on the raw path.\n' >&2
fi
if [ "$count" -eq 0 ]; then return 0; fi
if [ "$apply" -eq 1 ]; then
printf 'reap: moved %d entr%s to %s\n' "$count" \
"$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
else
printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
"$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
fi
}
# Descend only where the lower layer still has the directory: a directory that is
# missing below is itself the ghost, and its whole subtree goes with it.
reap_walk() {
local rel="$1" entries abs r name
entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
[ -n "$entries" ] || return 0
while IFS= read -r abs; do
r="${abs#"$upper"/}"
name="${abs##*/}"
if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
ghosts=$((ghosts + 1))
continue
fi
if [ -e "$lower/$r" ]; then
[ -d "$abs" ] && reap_walk "$r"
continue
fi
[ -d "$abs" ] || continue
count=$((count + 1))
printf '%s\n' "$r"
if [ "$apply" -eq 1 ]; then
mkdir -p "$stamp/$(dirname "$r")"
mv "$abs" "$stamp/$r"
fi
done <<<"$entries"
}
do_status() {
if mountpoint -q "$VIEW"; then
printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
else
printf 'view: %s NOT MOUNTED\n' "$VIEW"
fi
if [ -d "$upper" ]; then
printf 'upper: %s %s files %s\n' "$upper" \
"$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
else
printf 'upper: %s missing\n' "$upper"
fi
systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
}
case "$cmd" in
plan) do_plan ;;
verify) do_verify ;;
install) do_install ;;
reap) do_reap ;;
status) do_status ;;
*) usage ;;
esac
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
#
# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
# first, which is how a 16-hour phase gets paid for twice.
#
# Every phase is gated on the previous one, because the last time this ran unguarded
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
#
# 1. the runner must still be alive and its container must answer /health AND own the
# listening socket, or this script kills it, restores the service and exits non-zero —
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
# first, the dying process keeps its listener for a moment, and a rig container that
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
# then the pusher refused on the container name and the window died at second zero);
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
# marker names this container, and the container is running (scripts/calib-push.py
# --tap-marker), so a second lock survives a mistake in this file;
# 3. the reference container is banked into trash only when BOTH the serve and the push
# exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
#
# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
# hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
# (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
# Usage: scripts/qwen38-calib-window.sh [--dry-run]
set -uo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
DRY=0
[ "${1:-}" = "--dry-run" ] && DRY=1
say() { printf '%s %s\n' "$(date -Is)" "$*"; }
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
RIG=qwen38-bf16
SERVICE=frank-archon
PY="${PY:-python3}"
# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
# The checks that used to live here were proxies, and each lied at least once: the space check
# asked a directory that merely LOOKED related, `/health` answered for a dying production
# process, `podman inspect` named a container that existed but had bound nothing. The shared
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
# keeps an existing tap, so the tap-empty fact is waived.
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
if [ "$DRY" = 1 ]; then
echo "dry run: calib window — the service is left running, nothing is mounted or written"
echo "would run: bash scripts/run-qwen38-bf16.sh calib (rig boots, writes $CALIB/.rig-boot)"
echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/ (only if both phases exit 0)"
exit 0
fi
say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
# ---- phase 1: the rig -------------------------------------------------------
say "phase=calib serve begin"
bash scripts/run-qwen38-bf16.sh calib & spid=$!
rig_up() {
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
# dying production process keeps its listener for a moment while the freshly created
# rig container is still loading — so `container running` + `/health` 200 both hold
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
# pusher refused on the container name and the window died at second zero. Ask the kernel
# who owns the listening socket and require it to be a rig process.
local lpid
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
[ -n "${lpid:-}" ] || return 1
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
}
waited=0
until rig_up; do
if ! kill -0 "$spid" 2>/dev/null; then
wait "$spid"; crc=$?
say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
exit 1
fi
if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
podman stop "$RIG" >/dev/null 2>&1 || true
wait "$spid"
echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
exit 1
fi
sleep 15
waited=$((waited + 15))
done
say "phase=calib health ok waited=${waited}s"
# ---- phase 2: the tap -------------------------------------------------------
say "phase=calib push begin"
"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
--tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
prc=$?
say "phase=calib push end rc=$prc"
podman stop "$RIG" >/dev/null 2>&1 || true
wait "$spid"; crc=$?
say "phase=calib serve end rc=$crc"
if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
exit 1
fi
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
# ---- phase 3: bank the reference container ----------------------------------
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
else
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
fi
say "calib window end"
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
#
# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
# 200 from the *production* process that was still draining its listener, while the freshly
# created rig container had not bound anything. Other runs died from a port that answered for
# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
# them were a check that measured something adjacent to the thing it meant.
#
# Usage: bash scripts/qwen38-preflight.sh [--allow-existing-tap]
# Exit: 0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
set -uo pipefail
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
TAP="$CAMPAIGN/calib"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
ALLOW_EXISTING=0
[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
SERVICE="frank-archon"
PORT=7770
fails=0
ok() { printf 'PASS %-14s %s\n' "$1" "$2"; }
bad() { printf 'FAIL %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
skip() { printf 'WARN %-14s %s\n' "$1" "$2"; }
# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
ok cwd "running from the repo root ($REPO_ROOT)"
else
bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
fi
# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
if command -v podman >/dev/null 2>&1; then
# An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
# asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
| sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
if [ "${got:-0}" -ge 1 ]; then
ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
elif podman image exists "$IMAGE_REF" 2>/dev/null; then
bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
else
bad image "$IMAGE_REF is not present on this host"
fi
else
skip image "podman not on PATH — cannot resolve $IMAGE_REF"
fi
# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
else
bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
fi
# 4. tap cleanliness — stale grams from another run must never be mixed in.
n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
ok tap-empty "$n entries in $TAP"
else
bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
fi
else
bad space "cannot create $TAP"
fi
# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
# orphaned engine from a previous run makes the next boot die with `startup stopped: device
# error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
# CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
# carries the container ID, never the service name — both are proxies that lie.
PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
if [ -e /dev/kfd ]; then
mine="$(service_pids | tr '\n' ' ')"
foreign=""
for p in $(fuser /dev/kfd 2>/dev/null); do
case " $mine " in
*" $p "*) : ;; # the service we will stop
*) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
esac
done
if [ -z "$foreign" ]; then
ok device "no foreign /dev/kfd holders"
else
bad device "foreign holders:$foreign — kill by pid, never by pattern"
fi
else
skip device "/dev/kfd absent — not an AMD host?"
fi
# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
if [ -z "${lpid:-}" ]; then
ok port ":$PORT is free"
else
mine="$(service_pids | tr '\n' ' ')"
case " $mine " in
*" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
*) bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
esac
fi
# 7. corpus — the push source must exist and be the re-chunked file the window documents.
if [ -s "$CORPUS" ]; then
ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
else
bad corpus "$CORPUS is missing or empty"
fi
# 8. reference — the KL gate has nothing to compare against without it.
if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
else
bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
fi
echo
if [ "$fails" -gt 0 ]; then
echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
exit 2
fi
echo "PREFLIGHT PASSED: every fact checked by identity."
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""Shared plumbing for the qwen38 campaign corpus tools.
The three tools move the same two shapes — a JSONL of prompt records and a
one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
last part is not a detail: a KLD reference is only meaningful against the exact
token sequence the corpus recorded, so `n_tokens` exists once, here, and every
tool counts the same way. A window that decode/encode does not round-trip is a
refusal, never a warning.
Loaded as a sibling module. Each tool puts its own directory on `sys.path`
first, so `python3 scripts/<tool>.py` and the driver's container mount
(`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
run the tools by path, not piped through stdin.
"""
import contextlib
import json
import os
import sys
import tempfile
def refuse(msg: str) -> None:
"""Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
print(f"REFUSED: {msg}", file=sys.stderr)
sys.exit(2)
def load_tokenizer(path: str):
try:
from tokenizers import Tokenizer
except ImportError as exc: # the corpus tools carry their own venv
refuse(f"tokenizers is not importable by {sys.executable} — run the "
f"corpus venv's python (on dankserver: "
f"/home/dank/venvs/radcorpus/bin/python) or set "
f"FRANK_QWEN38_TOKENIZER: {exc}")
if not os.path.isfile(path):
refuse(f"no tokenizer at {path}")
return Tokenizer.from_file(path)
def n_tokens(tok, text: str) -> int:
"""The corpus's own count: no special tokens added, prompt side only."""
return len(tok.encode(text, add_special_tokens=False).ids)
def iter_jsonl(path: str):
"""Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
Field validation stays with the caller: the three corpora carry different
fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
a record missing the one the tool needs is that tool's refusal to make.
"""
with open(path, encoding="utf-8", errors="replace") as fh:
for lineno, line in enumerate(fh, 1):
if not line.strip():
continue
try:
rec = json.loads(line)
except json.JSONDecodeError as exc:
refuse(f"{path}:{lineno} is not JSON: {exc}")
yield lineno, rec
def write_jsonl(path: str, rows) -> None:
with open(path, "w", encoding="utf-8") as fh:
for row in rows:
fh.write(json.dumps(row, ensure_ascii=False) + "\n")
@contextlib.contextmanager
def atomic_writer(path: str):
"""Write `path` through a temporary sibling and rename on success.
An interrupted run then leaves no half corpus behind, and the temporary name
keeps the one-artifact-one-name law the driver applies to a score report.
"""
d = os.path.dirname(os.path.abspath(path)) or "."
fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
try:
with os.fdopen(fd, "w", encoding="utf-8") as fh:
yield fh
os.replace(tmp, path)
except BaseException:
if os.path.exists(tmp):
os.unlink(tmp)
raise
os.chmod(path, 0o644)
def write_json(path: str, obj) -> None:
with atomic_writer(path) as fh:
json.dump(obj, fh, indent=1)
fh.write("\n")
+94 -80
View File
@@ -38,12 +38,19 @@ one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
is written through a temporary name and renamed, so an interrupted run leaves no
half corpus behind. A JSON summary of what was written goes to stdout.
"""
import argparse
import dataclasses
import json
import os
import sys
import tempfile
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import qwen38_common as qc
MISSING = object() # "the parent record carried no source field"
def parse_args() -> argparse.Namespace:
@@ -69,95 +76,96 @@ def parse_args() -> argparse.Namespace:
return ap.parse_args()
def load_tokenizer(path: str):
try:
from tokenizers import Tokenizer
except ImportError as exc: # the corpus scripts carry their own venv
raise SystemExit(
f"REFUSED: tokenizers is not importable by {sys.executable} — run the "
f"corpus venv's python (on dankserver: /home/dank/venvs/radcorpus/bin/python) "
f"or set FRANK_QWEN38_TOKENIZER: {exc}") from None
if not os.path.isfile(path):
raise SystemExit(f"REFUSED: no tokenizer at {path}")
return Tokenizer.from_file(path)
@dataclasses.dataclass(frozen=True)
class Window:
"""One scoring document cut out of a parent record."""
prompt: str
score_from: int
n_tokens: int
clamped: bool
source: object = MISSING
def record(self) -> dict:
rec = {"prompt": self.prompt, "score_from": self.score_from}
if self.source is not MISSING:
rec["source"] = self.source
return rec
def refuse(msg: str) -> None:
print(f"REFUSED: {msg}", file=sys.stderr)
sys.exit(2) # 2 is the code argparse uses for unusable arguments
def main() -> None:
a = parse_args()
def validate(a: argparse.Namespace) -> None:
"""Everything that makes this run unusable, named before a token is cut."""
if a.max_doc_tokens <= 0:
refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
if a.score_tail <= 0:
refuse(f"--score-tail must be positive (got {a.score_tail})")
qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
if a.max_doc_tokens <= a.score_tail:
refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a {a.score_tail}-position "
"score tail: every window would be all tail and no context")
qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
f"{a.score_tail}-position score tail: every window would be all "
"tail and no context")
if not os.path.isfile(a.corpus):
refuse(f"no corpus at {a.corpus}")
qc.refuse(f"no corpus at {a.corpus}")
if os.path.realpath(a.out) == os.path.realpath(a.corpus):
refuse(f"--out {a.out} is the corpus itself; move it to trash first — the trash law "
"means the old segmentation stays recoverable")
qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
"trash law means the old segmentation stays recoverable")
if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
tok = load_tokenizer(a.tokenizer_json)
def cut(rec: dict, tok, a: argparse.Namespace,
where: str) -> tuple[list[Window], int]:
"""Cut one record's token ids into windows, proving each one round-trips.
The round-trip encode is the whole reason this tool may claim the reference
scores the corpus's own token sequence: decoding a window to text and
encoding it back must return the identical ids, or the score is of a
different document than the one on disk.
"""
ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
windows: list[Window] = []
for start in range(0, max(len(ids), 1), a.max_doc_tokens):
window = ids[start:start + a.max_doc_tokens]
if not window:
continue
text = tok.decode(window)
if tok.encode(text, add_special_tokens=False).ids != window:
qc.refuse(f"{where} window at token {start} does not round-trip through "
"text — the reference would score a different token sequence "
"than the corpus holds")
windows.append(Window(prompt=text,
score_from=max(len(window) - a.score_tail, 0),
n_tokens=len(window),
clamped=len(window) < a.score_tail,
source=rec["source"] if "source" in rec else MISSING))
return windows, len(ids)
def rechunk(a: argparse.Namespace, tok) -> dict:
"""Write the re-chunked corpus and return the summary it printed."""
docs_in = docs_out = tokens_in = tokens_out = 0
longest_child = 0
clamped = scored = 0
longest_child = clamped = scored = 0
sources: dict[str, int] = {}
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(a.out)) or ".",
prefix=".rechunk-")
try:
with os.fdopen(fd, "w", encoding="utf-8") as out:
with open(a.corpus, encoding="utf-8") as src:
for lineno, line in enumerate(src, 1):
if not line.strip():
continue
try:
rec = json.loads(line)
except json.JSONDecodeError as exc:
refuse(f"{a.corpus}:{lineno} is not JSON: {exc}")
prompt = rec.get("prompt")
if not isinstance(prompt, str):
refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
ids = tok.encode(prompt, add_special_tokens=False).ids
docs_in += 1
tokens_in += len(ids)
for start in range(0, max(len(ids), 1), a.max_doc_tokens):
window = ids[start:start + a.max_doc_tokens]
if not window:
continue
score_from = max(len(window) - a.score_tail, 0)
if len(window) < a.score_tail:
clamped += 1
scored += len(window) - score_from
child = {"prompt": tok.decode(window),
"score_from": score_from}
if "source" in rec:
child["source"] = rec["source"]
out.write(json.dumps(child, ensure_ascii=False) + "\n")
docs_out += 1
tokens_out += len(window)
longest_child = max(longest_child, len(window))
if "source" in rec:
for cls in str(rec["source"]).split("+"):
sources[cls] = sources.get(cls, 0) + 1
if tokens_out != tokens_in:
raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
f"has {tokens_in} — re-chunking lost or invented tokens")
os.replace(tmp, a.out)
except BaseException:
if os.path.exists(tmp):
os.unlink(tmp)
raise
os.chmod(a.out, 0o644)
print(json.dumps({
with qc.atomic_writer(a.out) as out:
for lineno, rec in qc.iter_jsonl(a.corpus):
where = f"{a.corpus}:{lineno}"
if not isinstance(rec.get("prompt"), str):
qc.refuse(f"{where} has no string 'prompt' field")
windows, n_parent = cut(rec, tok, a, where)
docs_in += 1
tokens_in += n_parent
for w in windows:
out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
docs_out += 1
tokens_out += w.n_tokens
longest_child = max(longest_child, w.n_tokens)
clamped += 1 if w.clamped else 0
scored += w.n_tokens - w.score_from
if "source" in rec:
for cls in str(rec["source"]).split("+"):
sources[cls] = sources.get(cls, 0) + 1
if tokens_out != tokens_in:
raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
f"has {tokens_in} — re-chunking lost or invented tokens")
return {
"corpus": a.corpus,
"out": a.out,
"tokenizer": os.path.basename(a.tokenizer_json),
@@ -171,7 +179,13 @@ def main() -> None:
"scored_positions": scored,
"clamped_score_from": clamped,
"source_documents": sources,
}, indent=1))
}
def main() -> None:
a = parse_args()
validate(a)
print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
if __name__ == "__main__":
+141 -19
View File
@@ -28,10 +28,18 @@
# (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
# separate driver — scripts/calib-push.py --port 7770
# --container qwen38-bf16 --corpus
# /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
# 12M pooled positions the engine writes the Grams and the
# tap switches itself off (calib-push prints TAP_DONE). Stop
# this script after that; the trap brings frank-archon back.
# /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
# at the floor the engine writes the Grams and the tap switches
# itself off (calib-push prints TAP_DONE). Stop this script
# after that; the trap brings frank-archon back. THE FLOOR IS
# THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
# served iq4r this arm competes with was calibrated on a 10 M-
# token set, so the arm is calibrated at least as deeply. One
# round of the re-chunked corpus is ~9.31 M tokens, so sealing
# takes round 1 plus part of round 2 — a round that seals
# nothing is NORMAL (the floor is pooled), and the pusher's
# round_verdict treats a flat round as "continue", never as a
# failure. One seal writes ~40 GiB.
#
# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
# weights disk tier, deterministic (the resident set pinned to the loader's plan,
@@ -92,7 +100,8 @@
# the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
# and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
# FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
# carries its own step measures a shape nothing serves) —
# carries its own step measures a shape nothing serves; calib is exempt, see
# Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
# and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
# see the corpus census above, and raise it together with the corpus).
# kldref refuses (exit 2) when the longest record of the corpus it is about to
@@ -112,6 +121,18 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-archon.env"
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
# moving target and a tap must name the producer that wrote it. `:latest` on
# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
# calib boot prints the arming line — "recorded passes: off, the program
# accumulates calibration Grams (gram_accum)" — where the production pin
# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
# into the tap dir while /health, /stats and the push all looked healthy
# (measured 2026-10-30). The tap code is identical in both trees' sources, so
# the shipped 1.8.1 build is what drops it; do not spend another window on it.
# Any other image must be named deliberately with --image.
CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
IMAGE_EXPLICIT=0
MODE=""
POS=()
@@ -120,7 +141,8 @@ while [ $# -gt 0 ]; do
case "$1" in
kldref|calib|score) MODE="$1" ;;
--dry-run) DRY=1 ;;
--image) IMAGE="$2"; shift ;;
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
*)
# score's two positional arguments, and nothing else may be bare.
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
@@ -150,7 +172,7 @@ TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
KLD_REF_DIR="$CAMPAIGN/kld-ref"
CALIB_DIR="$CAMPAIGN/calib"
CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
C=qwen38-bf16
SERVICE=frank-archon
# The card split, taken from the file that owns it instead of a private pair:
@@ -170,19 +192,38 @@ fi
# ONE step size for every mode, quoted from the file that owns it — the audit
# law for a radiance rig is to inherit the proven number, not to carry a
# campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS is 2,048,
# the widest step the quantised expert FILE tiers can stream (above it the
# engine charges a 10.65 GiB arena to the card and the load refuses), and the
# arena it charges comes out of the cache — so the corpora are cut to the window
# the boot ledger reports, never the other way round. The 1,024 default this
# script carried was PR #80's answer to a starved cache; the pool raise
# (frank-archon.env: 66,560 MiB a rank) is the fix, not a narrower step.
# campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS. The
# 1,024 default this script carried was PR #80's answer to a starved cache;
# production moved to its widest legal step (2,048) on 2026-10-29 and the rig
# follows. The pool raise that shipped with it (66,560 a rank) bought this
# container 1.26 MiB of cache (`vram_kv 269.99 -> 271.25 MiB`, window still
# 7,436 tokens — the 95.37 GiB n-gram table's claim decides the rig's cache,
# not the pool) and was folded back the same day. The corpora stay cut to the
# window the boot ledger reports, never the other way round.
MNBT="${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
echo "REFUSED: FRANK_ARCHON_MAX_BATCHED_TOKENS must be a token count (got: $MNBT)" >&2
exit 2
fi
# Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
# comparable across boots at an identical shape, which is why they inherit production's step and
# slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
# its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
# document cut into four 2,048-token steps re-streams each layer's routed expert union four
# times from the container, so one wide step is worth up to ~4x the read volume. Measured
# 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
# find out what the wide step is worth before the campaign pays for two days of drained serving.
# Both default to the inherited value, so nothing moves unless a probe asks it to.
CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
exit 2
fi
done
# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
# env file says so for production), so this is the number the corpus must fit in.
MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
@@ -191,6 +232,25 @@ if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
exit 2
fi
# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
# PINS it: the calibration boot died at "startup stopped: out of memory" with
# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
# while the reference recording and every score boot had already succeeded with
# the same pool and slots. The difference that matters is the job: calib only
# needs the routed weights prefilled, so it reads the table from the container's
# own file tier and the pin never happens; kldref and score keep production's
# placement, because their numbers are only comparable when every other term of
# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
case "$MODE" in
calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
*) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
esac
case "$NGRAM_PLACEMENT" in
auto|vram|ram|disk) ;;
*) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
exit 2 ;;
esac
# Every container this campaign boots is read through the ro /models mount, so
# the host path must live under it; the container-side spelling is what goes
# into the argv. Same rule for the container a score pass opens.
@@ -228,7 +288,7 @@ ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
--host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
--deterministic --num-speculative-tokens 0
--kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
--ngram-placement ram
--ngram-placement "$NGRAM_PLACEMENT"
--override-chat-template /models/tpl.jinja)
MOUNTS=(-v "$MODELS:/models:ro")
TAPENV=()
@@ -249,7 +309,8 @@ case "$MODE" in
# time the engine reads a recording and writes a report instead of the
# reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
# able to touch the zero point — and the report gets its own writable bind.
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
|| [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
exit 2
fi
@@ -273,11 +334,45 @@ case "$MODE" in
# buys throughput with the expert slab, so price it like any other
# VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
# through MAX_SEQS, exactly as kldref and score do.
[ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
# ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
# 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
# the filesystem to zero, after which every write came back short ("calibration
# cannot write", "write was short") and the system itself began refusing
# writes. The room is demanded up front, on the tap's own filesystem.
if [ "$DRY" != 1 ]; then
avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
if [ "$avail_mib" -lt "$need_mib" ]; then
echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
exit 2
fi
fi
ARGS+=(--model "$MODEL_IN" --port 7770
--max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
--max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
# The tap is only real if this directory exists and the engine can write it:
# without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
# whatever path was given, and the run burns hours serving prompts into a tap
# nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
[ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
exit 2
}
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
-e RADIANCE_CALIB_TOKENS=12000000)
-e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
# The tap's own authorization. A pusher must see this marker before it sends
# a prompt (calib-push.py --tap-marker), it names the container so a push
# aimed anywhere else refuses, and a boot that fails removes it again below.
# Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
# into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
if [ "$DRY" = 1 ]; then
:
else
printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
"$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
fi
;;
esac
@@ -351,7 +446,27 @@ cp -f "$TEMPLATE" "$TPL" || {
[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
[ "$MODE" = calib ] && mkdir -p "$CALIB_DIR"
restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
# Stop the campaign container BEFORE handing the cards back: a killed run leaves
# its conmon alive (--rm only fires when podman itself exits), and production
# then boots beside a container still holding the GPUs — measured 2026-10-29,
# where the service reported `active` while its engine starved on the rig's VRAM.
restore() {
podman rm -f "$C" >/dev/null 2>&1 || true
# Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
# radiance kept /dev/kfd after `podman rm -f`, production's
# ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
# and Restart=always looped it 100 times with /health never answering. The
# service is stopped for the whole run, so any radiance still holding the
# device here is an orphan — kill it BY PID (pkill -f matches the caller's
# own command line, which is how a session gets killed instead of the orphan).
for p in $(fuser /dev/kfd 2>/dev/null); do
if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
kill -9 "$p" 2>/dev/null || true
fi
done
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
}
trap restore EXIT
echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
@@ -359,5 +474,12 @@ podman rm -f "$C" >/dev/null 2>&1 || true
"${CMD[@]}" 2>&1 | tee "$LOG"
rc=${PIPESTATUS[0]}
# A boot that failed must not leave the tap marked as rig-authorized: the marker
# is what calib-push.py trusts, and the point of it is that a dead rig can never
# be pushed into again.
if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
fi
echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
exit "$rc"
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
+3 -3
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
# vendor build 10777, CUDA sm_86).
# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
#
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
: "${ALIAS:=frank-zealot}"
: "${BIND_ADDR:=0.0.0.0}"
@@ -119,7 +119,7 @@ fi
# above are what pay for it. The draft cache dtype IS a further knob
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
# build's --help) and stays at its default here. Card gate: quantized V alongside
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
# prebuilt; the type-142 loader lock keeps it pinned regardless.
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
: "${SPEC_TYPE:=draft-mtp}"
+16 -7
View File
@@ -337,8 +337,12 @@ def main():
if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
log("ERROR: PLE shards disagree on HD")
return 2
if hd % 128 != 0 and hd != 160:
log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
if hd != 160 and hd % 128 != 0:
# The pack below writes lanes of 128 codes; any other HD would produce a
# table that decodes to the wrong rows. The old warning let it continue.
log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
"128-lane pack cannot express it")
return 2
# ---- idempotence gate -------------------------------------------------
stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
}
for name, (dt, shp) in want.items():
e = hdr[name]
assert e["dtype"] == dt and list(e["shape"]) == shp, name
assert e["data_offsets"][1] - e["data_offsets"][0] == \
int(np.prod(shp)) * (1 if dt == F8 else 2), name
if e["dtype"] != dt or list(e["shape"]) != shp:
raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
_, phdr = parse_header(out_path)
e = phdr[next(iter(phdr))]
assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
f"expected {BF16}[{sum(rows)}, {hd}]")
# sampled PLE rows: recompute and compare bytes exactly
lut = e4m3_lut()
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
with open(out_path, "rb") as ofh:
ofh.seek(out_off + got_row * hd * 2)
got = ofh.read(hd * 2)
assert got == want, f"PLE row {got_row} mismatch"
if got != want:
raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
mm.close()
log("verify: new-file headers, sizes and sampled PLE rows all exact; "
"expert slabs were byte-checked in-stack")
+10
View File
@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
dankserver/.env.example|dankclaw/.env.example)
add_test tests/env_consistency.bats
;;
# The SSD-backed metadata view: the installer, the host mount unit, and the
# two media units that bind the view rather than the raw array.
scripts/media-view.sh|dankserver/host/*)
add_test tests/media_overlay.bats
add_test tests/dankserver_quadlet.bats
add_test tests/deployment_scoping.bats
;;
dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
add_test tests/media_overlay.bats
;;
shared/homepage/*)
add_test tests/homepage_caddy.bats
add_test tests/homepage_config.bats
@@ -48,6 +48,14 @@ Use the `Authorization: token` header for everything.
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
- **Comment on a PR**: the conversation thread is `POST $API/issues/<PR-number>/comments`
(201, and the answer's `html_url` points at `pulls/<n>`; Gitea's issue index is
unified, so the PR's number is the same number). `$API/pulls/<n>/comments` is the
**review**-comment store and answers `405` to a plain body.
- **A body built by `jq --rawfile b /dev/stdin` inside `$(…)` is empty**: the command
substitution does not inherit the heredoc, so Gitea receives `{}` and answers 422 —
and `jq -r '.id'` on the error prints nothing, which reads as success. Write the
body to a file and pass `--rawfile` that file.
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
@@ -195,8 +195,9 @@ deploy.sh, never a live env poke.
check every namespace (`sudo podman ps -a`) and native units
(`systemctl list-units`) before declaring a service absent. Compare
compose mount sources against the real filesystem — compose files name
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
`/mnt/entropy`). Fix compose through git first, deploy second.
phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
compose through git first, deploy second.
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
or creation fails `statfs …: no such file or directory`.
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+120 -7
View File
@@ -1,9 +1,9 @@
---
type: reference
title: Jellyfin
description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
tags: [jellyfin, dankserver, media-server]
timestamp: 2026-09-03T00:00:00Z
timestamp: 2026-10-30T00:00:00Z
---
# Jellyfin
@@ -18,7 +18,7 @@ clients.
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
## Facts (verified from the live unit, 2026-09-03)
## Facts (the repo quadlet; the live unit is still the pre-view generation — see Data layout)
| Item | Value |
|------|-------|
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
| Image | `ghcr.io/jellyfin/jellyfin:latest` |
| Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
| Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it |
| Media | `Volume=/var/lib/media-meta/view:/data` in git — the [[Media Metadata View]] overlay, **writable**, sidecars land on the NVMe and the [[Entropy Volume]] array underneath stays untouched. **The live unit is still pre-view: `/mnt/entropy:/data:ro`** — nothing is written into the library until the view is deployed |
| GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
| Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
| Health | `curl -f http://localhost:8096/health` every 30 s |
@@ -36,11 +36,27 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
```
/home/dank/Containers/jellyfin/config -> /config (JELLYFIN_DATA_DIR)
/home/dank/Containers/jellyfin/cache -> /cache (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
/mnt/entropy -> /data (read-only media volume)
/var/lib/media-meta/view -> /data (git: the overlay view)
/mnt/entropy -> /data (live today: the array, read-only)
```
Library folders point at `/data` paths; because the mount uses the same
absolute path as on the host, library paths survive container recreation.
Library folders point at `/data` paths: stable across container recreation and
identical to the raw array layout below. Which side of that path the container
is on is the open question on this host — the installed unit binds the read-only
array, and `scripts/media-view.sh status` reports `NOT MOUNTED` (2026-10-31).
`Local metadata` — save metadata to media folders — is enabled, and it is the
reason the view exists: `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork
written once on the NVMe upper layer are files [[Plex]] can read too. While
`/data` is bound read-only every item update logs the error below and no sidecar
is written; Jellyfin becomes the fleet's only metadata writer with the deploy
that raises the view.
**Failure signature:** `System.IO.IOException: Read-only file system :
'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
instead of the view. On Dankserver that is today's expected state — the view has
never been installed (`scripts/media-view.sh status`). Once it is deployed, the
same line is a fault: the quadlet drifted or the mount died.
## Ports & access
@@ -105,12 +121,109 @@ where the LDAP plugin runs. Verified 2026-09-25: `alex` authenticates over
`AuthenticateByName` (HTTP 200; full plugin flow logged — service bind,
search, `ldapUid`, user bind, token).
## Metadata providers and their keys (2026-10-30)
TMDb, OMDb, StudioImages and MusicBrainz are **core plugins** in Jellyfin 12 —
no DLL, just a configuration file the running container owns under the data
root. TVDB is not core (its metadata provider was removed in 10.9), so it is an
official-catalog plugin.
| Provider | Shape on disk | Key lives in |
|---|---|---|
| TMDb | `config/plugins/configurations/Jellyfin.Plugin.Tmdb.xml` → `<TmdbApiKey>` | `/home/dank/Containers/jellyfin/provider-keys` → `TMDB_API_KEY=` (mode 600) |
| TVDB | `config/plugins/thetvdb_24.0.0.0/` + `…Jellyfin.Plugin.Tvdb.xml` → `<ApiKey>` | the same file → `TVDB_API_KEY=` |
| OMDb | `…Jellyfin.Plugin.Omdb.xml` | **not configured** — its key is issued by email, 1 000 req/day, and its ToS forbids storing the returned data; ratings stay Jellyfin's own lookup |
**None of it is tracked in git.** Those XML files belong to the container and
hold credentials, and `tests/jellyfin_providers.bats` fails the suite if one is
ever committed. The keys themselves sit in the per-service host env file that is
the fleet's convention for container credentials (grafana, caddy, pihole,
copyparty, romm, seerr): `provider-keys`, mode 600, beside the data root.
`scripts/jellyfin-providers.sh` is the only thing that writes the XML:
```
scripts/jellyfin-providers.sh plan # what would change; prints no key
scripts/jellyfin-providers.sh install # plugin + keys, previous copy to trash
scripts/jellyfin-providers.sh verify # plugin + key LENGTHS, never key bytes
```
- **A key never enters git, argv, or the tool's output.** The installer reads
that 600-perm file itself (a value in argv is world-readable through
`/proc/<pid>/cmdline`) and refuses it when it is group- or world-readable —
an exposed key is an error, not a warning. A key that shows up in a transcript
or `ps` output has leaked: rotate it with the provider, rewrite
`provider-keys`, re-run `install`.
- **The TVDB artifact is pinned three ways**: sha256
`6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319`, the
official plugin guid `a677c0da-fac5-4cde-941a-7134223f14c8`, and version
`24.0.0.0` (`targetAbi 12.0.0.0`, which fits the deployed 12.4.0). A sha
match alone only proves the bytes are the bytes fetched — the guid check is
what says it is the official plugin. Rollback: move `thetvdb_24.0.0.0/` into
the trash and revert the pin; a second `thetvdb_*` directory is a refusal,
never a quiet second copy.
- **The homelab TVDB key is a licensed key** (its JWT claims say
`community_supported: false`), so it needs **no PIN** — the optional
`TVDB_PIN=` line exists only for a subscriber-shaped key.
- **TVDB is installed and keyed but is not a metadata agent**: TMDb still
identifies everything. Enabling TVDB as a provider would re-fetch show
metadata across the library, so it is a deliberate, separate change.
- **The deployer places the keys before it restarts jellyfin**
(`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
failing when `provider-keys` is absent, so a host without it still deploys.
- **Keys and plugins are read at start-up**: after a manual `install` the
change is inert until `./scripts/deploy.sh dankserver --service jellyfin`.
- TMDb's terms are the reason this stays narrow: attribution required, no
caching beyond 6 months, no derivative works. Fine for a personal homelab,
and the reason no bulk enrichment pass runs against it.
## Configuration workflow
1. Create libraries pointing at `/data` folders (movies, shows, audio).
2. Set metadata/language preferences (`LANG=en_US.UTF-8` pinned in the unit).
3. Configure clients to use `http://192.168.0.235:8096` or the guarded vhost.
## Triggering a scan
The scan key lives in `~/.config/secrets/jellyfin` (mode 600; the same value is
`JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a mode-600 curl
config, never a header argument, in the `MediaBrowser` form: 12.4 authenticates
only `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token`
for every key, valid or not. Before believing a 401, prove the key against the
server's own table — `select count(*) from ApiKeys where AccessToken = ?` on
`jellyfin.db` opened `mode=ro&immutable=1`. The backslashes below matter:
```bash
umask 077
{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
"$(cat ~/.config/secrets/jellyfin)"
echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
echo 'request = "POST"'; } >/tmp/jf.hdr
curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
shred -u /tmp/jf.hdr
```
`204` is acceptance; anything else queued no scan, so re-issue the identical
request instead of diagnosing. A `401` from a key that answers `200` on a
hand-typed request is this block's quoting, not the server: bash `printf`
collapses `\"` to `"`, curl ends the quoted value at that quote, and Jellyfin
receives `Authorization: MediaBrowser Token=` with nothing after it (`\\"` answers
204, `\"` answers 401 — both run on this host).
One accepted refresh is the whole procedure. A second one while a scan runs moves
`Scan Media Library` to `Cancelling`, and once the runner holds `Cancelling` every
later `204` is accepted and then discarded — the status code stops meaning
anything. The fix is `systemctl --user restart jellyfin.service` (never
`deploy.sh`: that runs the never-tested metadata-view cutover), one POST after
health answers, and then believe `ScheduledTasks` State at 130 % CPU, not the
response code.
**A scan that must see a new folder has to be recursive.** `POST /Library/Refresh`
is the incremental pass, and inotify does not cross a rootless podman bind, so
folders created on the host stay invisible to it. The move is
`POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id from
`GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and that same
record's `RefreshStatus` is the progress read.
## See Also
- [[Plex]] — co-resident media server on the same volume
@@ -0,0 +1,130 @@
---
type: reference
title: Media Metadata View
description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
tags: [dankserver, media-server, jellyfin, plex, storage]
timestamp: 2026-10-30T00:00:00Z
---
# Media Metadata View
`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
file, and the array is read-only and slow — so the media bytes stay there and the
*writes* land on SSD. One tree to both servers, one copy of the media.
The mount point is not a volume and holds nothing while unmounted. It sits inside
`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
merged door are one directory — and so the mount table grows no new `/mnt` entry
next to the array.
**Not deployed on Dankserver — verified 2026-10-31.** `scripts/media-view.sh
status` answers `view: /var/lib/media-meta/view NOT MOUNTED`,
`/var/lib/media-meta` does not exist, and the installed quadlets under
`~/.config/containers/systemd/` still carry the pre-view binds — `/mnt/entropy:/data:ro`
for [[Jellyfin]], `/mnt/entropy:/mnt/entropy:ro` for [[Plex]]. Everything below
is the shape git describes, not the shape the servers run. The cutover is an
ordinary deploy: `deploy.sh dankserver --service jellyfin|plex` runs
`ensure_media_view` (`install` then `verify`) before it restarts the container,
so it must not be fired through the middle of a library scan. Until that deploy
lands there is no upper layer, so no relayout can leave ghost metadata and
`reap` has nothing to do — `media-apply.py` prints that follow-up only when the
view really is a mountpoint.
## Shape
| Item | Value |
|---|---|
| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
Both consumers guard the view with
`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
install` + `verify` before it restarts either container.
| Consumer | Bind | Mode | Why that path |
|---|---|---|---|
| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
Putting the sidecars inside Plex's reach is not the same as making Plex use them:
a library parses `.nfo` only with its *Local media information* agent enabled, and
this change touches no library setting.
## Commands
```bash
scripts/media-view.sh plan # print the resolved shape, touch nothing
sudo scripts/media-view.sh install # create dirs, install the unit, mount, enable
scripts/media-view.sh verify # is the view really the overlay? (deploy gate)
scripts/media-view.sh status # mount options + upper-layer size/file count
scripts/media-view.sh reap # list ghost metadata (media moved, sidecars left)
sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
```
`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
the defaults are the host values above.
## Laws
- **The upper layer must sit on non-rotational storage.** `install` and `verify`
refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
whose upper is the spinners is the original problem with extra steps, and it
silently turns every metadata write back into a seek.
- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
presents an *empty library* as truth, and a media scan would then clear the
database of everything. systemd does not create a mount point, so `install`
creates `view` itself; without it the mount simply fails.
- **Never delete through the view.** A delete writes an overlayfs whiteout
(`.wh.<name>`) in the upper layer, which hides the array's copy without
removing it — the file reappears the moment the view is unmounted, and the
upper layer now carries a landmine. Deletions belong on the raw path
(`/mnt/entropy`), which the relayout tool already treats as the only writer.
`reap` refuses to remove whiteouts and only warns about them.
- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
sidecars the view already wrote stay at the old coordinates: a ghost folder
the servers browse and find nothing in. `reap` lists upper-layer directories
with no counterpart below and moves them (with their subtree) into a dated
`.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
that exists only above is normal, so only *directories* are candidates.
- **File-watcher events do not cross the overlay.** Plex's NFO agent and
Jellyfin's real-time monitor see inotify events for writes that land *in the
upper layer*, not for changes on the array underneath. After any relayout or
externally-written metadata, trigger a library refresh — see
[[Media Relayout]] for the refresh procedure.
## Failure signatures
| Symptom | Meaning |
|---|---|
| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
## Rollback
Revert the commit that pointed the two quadlets at the view and re-deploy both
services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
The NVMe upper layer can stay: nothing reads it once the binds are back, and it
holds no state either server needs. Unmounting alone
(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
binds it, and the array is never modified by any of this.
## Links
- [[Jellyfin]] — the metadata writer; its data layout section.
- [[Plex]] — the read-only consumer; in-place bind-mount history.
- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
- [[Entropy Volume]] — the array underneath.
@@ -0,0 +1,497 @@
---
type: reference
title: Media Relayout
description: "Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback."
tags: [dankserver, media-server, jellyfin, plex, procedure]
timestamp: 2026-10-30T00:00:00Z
---
# Media Relayout
The movie library on [[Entropy Volume]] arrived as release folders —
`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
the release. [[Jellyfin]] matches a movie by folder and file name, so the
release noise cost posters, extras and versions. These scripts turn that
library into the shape its media server reads — the movies, the TV packs, the TV
that turns up in the movie share, the music library, and the audiobooks, whose
only reader is Plex — and nothing else in the repo touches a media file.
| Script | Touches media? | Output |
|---|---|---|
| `scripts/media-census.py` | reads headers and the servers' own DBs, never writes | `enum`, `probed`, and one authority TSV per library |
| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
| `scripts/media-shows.py` | no — TSV out | a plan that re-files the TV library itself |
| `scripts/media-music.py` | no — TSV out | a plan that re-files the music library by its tags |
| `scripts/media-audiobooks.py` | no — TSV out | a plan that re-files the audiobooks by the book Plex matched |
| `scripts/media-verify.py` | no — reads the servers' DBs read-only | the verdict that every journaled move landed |
The seams are `tests/media_relayout.bats`, `tests/media_shows.bats`,
`tests/media_music.bats`, `tests/media_audiobooks.bats` and `tests/media_verify.bats`;
the movie and TV target contracts are quoted from
[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
in `media-plan.py`'s docstring.
## Procedure
Run the census on the machine that mounts the library. `probe` is the slow step
(a header read per container) and is `nice`d — it is safe beside a live service
because it never competes for the GPU and never writes inside the library.
```bash
C=scripts/media-census.py
$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
$C probe --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
$C plex --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
```
`plex` reads the Plex metadata cache (`--db` defaults to the path the
[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
`mode=ro&immutable=1`: a live server is never locked or written.
Then plan — this reads only the three TSVs, so it costs seconds and can run
anywhere, including a laptop:
```bash
scripts/media-plan.py --root /mnt/entropy/Movies \
--enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
--plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
```
**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
is a file the plan could not name a film for, a pile that needs a README, or a
collision it refused. The printed gate lines are the go/no-go:
`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
0 and `unplanned=0` must appear in the report. Any non-zero number means a file
that would be filed wrong, and the plan is not run until it is understood.
```bash
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies # dry, writes nothing
scripts/media-apply.py apply --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
scripts/media-apply.py verify --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies # dry
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
```
An apply also leaves the release folders it emptied standing where they were —
hundreds of them at the library root — which is what `sweep` removes, dry-run
first, from the same journal.
After it lands, both servers need a scan or the change is invisible. [[Plex]] and
[[Jellyfin]] own the exact requests; two laws hold at this step: `204` is
acceptance and anything else queued nothing — re-issue the identical request
rather than diagnosing — and ONE accepted refresh is the whole procedure, because
a second while a scan runs moves `Scan Media Library` to `Cancelling` and every
later `204` is accepted and then discarded.
### Proving the move landed
A scan answers `204` whether or not it finds anything, so proof is the journal
replayed against both servers' own tables: every `move` row has a row at its
`dst`, and **zero** rows still name its `src`. `scripts/media-verify.py` is that
replay: it opens each database read-only (there is no `sqlite3` binary on these
hosts or inside either container) and exits non-zero while a row is stale.
```bash
scripts/media-verify.py --journal /mnt/entropy/media-relayout-music.jsonl \
--db-kind jellyfin --ignore-ext jpg,jpeg,png,log,cue,toc,nfo,txt,m3u,accurip
```
Ignore only what a server cannot index. A missing `.flac` or `.mkv` is the
fault; a row still naming its `src` means the scan has not followed the rename
yet, so re-run the check after the scan, never before.
Three doors that look right and are not: `GET /search?query=` matches metadata
titles, never file paths; `?path=` on `/library/sections/N/all` makes Plex's
HTTPD close without a single byte; and the same HTTPD drops requests that carry
no `Accept` and client-identifier header (see the scan block above for the form
that works).
## TV in the movie library
A box set that arrives in the movie share stays there until someone moves it:
`media-plan.py` refuses to file TV and parks each series in
`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
walks the movie library itself (no census needed), groups episode-shaped names
by series, and writes a plan in the executor's own format:
```bash
scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
--out /mnt/entropy/media-tv-plan.tsv
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
--root /mnt/entropy --apply
```
The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
because the publisher's spelling beats a re-derivation, with the year taken
from the box-set folder. What it refuses it prints and leaves alone:
- fewer than `--min` (5) files under one series name: one file named `Part 2`
is a film, not a series;
- an episode whose name states no season number — no invented `Season 01`;
- a destination two files both claim: the executor refuses the whole plan
before its first rename, and only Frank decides which copy is the keeper.
- an episode the destination library already holds at the same byte count. The
movie share carried a second copy of the whole Monk box set — 124 episodes
both sides, the same episode list, every byte count identical, 31.5 GiB each
way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
such a series in place and says so; a duplicate is deleted or kept by Frank,
never copied into one library by a tool.
A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
Below the threshold the mover prints and leaves, so a lone downloaded episode of
a real show stays in the movie library on purpose — six did on 2026-10-30.
Filing one by hand is the same executor on a hand-written plan and the same
journal; the trap is the series year. Read it from the file's own name or the
folder it owns, never from the title whose bonus folder it happens to be parked
in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
show. A name whose marker is `Part 2` is a film and stays put.
## The TV library's own relayout
`/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
already matched and numbered most of it. `scripts/media-shows.py` re-files that
library into the shape Jellyfin reads, with **tvdb as the name authority**:
<Series> (<year>)/Season NN/<Series> (<year>) - SxxEyy - <Episode Title>.ext
It never guesses: every series name, episode number and title it writes traces to
an authority, and what has none is parked for a human to read.
```bash
# 1. The authority: Jellyfin's own matched rows — each carries the series' TMDb
# and Tvdb ids and the title the row holds (the DB opens read-only).
python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
--out /tmp/tv_titles.tsv
# 2. What is on disk (stat only), and the runtimes the film arm needs.
python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
--out /tmp/tv_enum.tsv
python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
--enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
# The probe is the slow step: ffprobe reads both ends of every file. So re-use
# the last one — a duration belongs to the file, not to its name: re-key the
# previous census through that pass's own journal (`src` -> `dst`) instead of
# seeking the array again. It is also the only column the film arm reads, and
# only for a file with no marker and no provider row (4,794 of 58,551 today,
# all of them covered by a re-key that costs seconds, not an hour).
# 3. The plan. --tmdb needs TMDB_API_KEY — the v3 key in ~/.config/secrets/tmdb,
# which travels as the `api_key` query parameter. Only a v4 request token is
# accepted as a Bearer header, and the wrong shape answers 401 on EVERY call,
# which the summary line flags as NOT provider-complete. --tvdb reads its
# `api_key=` line from ~/.config/secrets/tvdb, never argv. Both write the SAME
# cache file, so the plan is reproducible offline afterwards: run it once to
# fill the cache, and quote the second run.
# Paths are only strings to a dry run, so this step can run on the workstation
# against scp'd TSVs — which is where the two provider secrets live.
python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
--titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
--tmdb-cache /mnt/entropy/media-tv-names.json --tmdb --tvdb \
--park /mnt/entropy/media-park/TVShows \
--out /mnt/entropy/media-relayout-shows2-plan.tsv \
--exceptions /mnt/entropy/media-relayout-shows2-exceptions.txt
# 4. Read the exceptions file FIRST — it is the review, not a log. Same executor
# as above, but on this arm's OWN journal, NAMED PER PASS: the journal of an
# applied pass is its unwind record, so a second pass that reuses the name
# destroys the only map back.
scripts/media-apply.py apply --plan /mnt/entropy/media-relayout-shows2-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-shows2.jsonl --apply
```
Evidence, strongest first — the order the planner believes things:
- the **episode marker the file's own name states** (`S01E02-E03`, `1x04`);
- the **`Season NN` folder it sits in and the number its name leads with inside
it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
- the **provider row**: which episode the file IS, and the numbering of a file
whose own name says nothing;
- the **row's tvdb id, else its TMDb id carried through TMDb's external ids, else
an exact tvdb name search**: that id names the SERIES FOLDER and supplies the
episode title, because a name the server read off a folder is not a show name;
- a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
series' canonical extra folder;
- a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies` only
when the movie provider's title is a run of words the FILE itself states and its
year agrees with the file's: the query is only what `title_of` salvaged from a
release name, and the first hit for a short one is a near miss — a 1963 Bond film
for the word "russia", *Special Edition I* for a file saying *Edition III*. Else it
parks.
Two refusals carry the design, and a dry run on the real library found both:
- **Folders resolve a series; they never invent one.** A folder's words may
resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
`…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
- **A scraped row does not outrank the filesystem.** Jellyfin numbers
multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
season 16 episode 20, and one Gundam row says season 2004. The folder, then
the file's own leading number, win; a row that still insists on season 1989
parks its file instead of creating `Season 1989`.
**A provider name is evidence only when an id backs it.** Jellyfin names a Series
after the folder it scanned, so a library filed by release folders ends up with
one Series item per box set — `24 (2001) Season 1`, `Modern Family S05` — and
copying those names verbatim launders folder junk into the contract. The planner
resolves each series to its tvdb id and writes tvdb's name and year; the merged
identity is also what turns a box set split across five release folders into ONE
folder, and what makes two files claiming one episode MEET in the duplicate rule.
A series no id resolves keeps the provider's words and is listed under *no tvdb
identity* in the exceptions — that list, not a rename, is the review.
**The review pile lives outside the library.** Files parked inside
`/mnt/entropy/TVShows` were indexed as one show named `_unsorted`, and that
show's rows then looked like evidence. So a row whose series is the hidden
top-level folder the file itself sits in is dropped — and the folder is matched
with its leading underscore stripped, because the server *displays* `_unsorted` as
`unsorted`: compare the spelled names and the pile survives the relayout as a
visible series folder that real files get renamed into. `--park` outside the root
makes the next plan re-file the pile instead of skipping it. Inside a hidden
top-level folder a file's own words may MATCH a registered show, never invent one.
## The music library's own relayout
`/mnt/entropy/Music` is rips and discography packs, and Jellyfin has matched and
tagged almost all of it. `scripts/media-music.py` re-files it into the shape
Jellyfin's music scanner reads —
Music/<album artist>/<Album> (<year>)/NN - <Title>.ext
— and the authority is the tag row of the track, never the name the ripper gave
the file.
```bash
# 1. The authority: Jellyfin's audio rows, one per track, with the album and
# artist ids its own database carries (the DB opens read-only).
python3 scripts/media-census.py music --root /mnt/entropy/Music \
--out /tmp/music_rows.tsv
# 2. What is on disk (stat only).
python3 scripts/media-census.py enumerate --root /mnt/entropy/Music \
--out /tmp/music_enum.tsv
# 3. The plan.
python3 scripts/media-music.py --root /mnt/entropy/Music --enum /tmp/music_enum.tsv \
--rows /tmp/music_rows.tsv --out /tmp/music_plan.tsv \
--exceptions /tmp/music_exceptions.txt
# 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
# executor, journal and rollback as above, on its own journal.
scripts/media-apply.py apply --plan /tmp/music_plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-music.jsonl --apply
```
The group is what the tags assert, not where the files sit: **(album-artist tag,
album, year)**. One album arrives as three disc folders and as two copies of one
rip, and one MusicBrainz release group is one album. A compilation is one album
whose tracks name many performers, so it files once under `Various Artists`:
keying on the performer would file one album once per artist. Two different
album-artist tags on one album name are two albums, which is what Jellyfin's two
album items say.
What it refuses:
+- **A sidecar is placed by the audio around it, never by its own name.** A
`.lrc`, `.txt` or picture that shares a track's stem follows that track and
takes the name the track now has, or Jellyfin cannot find it; anything else
goes to the album its own folder was placed in, or to the folder whose discs
sit one level below it. A folder whose audio belongs to two albums parks its
sidecar rather than guessing.
+- **An image album keeps its name.** A folder holding one audio file and a
`.cue` is a whole album in one file; numbering it `01 - …` tells Jellyfin to
index it as one track.
+- **A video container parks**, because Jellyfin will not index it as music and a
rename to `.m4a` would claim it is; **audio no provider row names parks** too,
because unmatched audio is a tagging job, not a renaming job.
ONE writer at a time on this array: two rename passes race, and neither journal
tells — which is why each arm above carries its own journal name.
## The audiobook library's own relayout
`/mnt/entropy/Audiobooks` is the one library here that [[Jellyfin]] does not read
at all — its music scanner has no audiobook concept — while [[Plex]] indexes all
of it as Artist → Album → Track: the album is the BOOK, that book's single artist
is the NARRATOR, both matched out of the files' own tags. Today one book is split
across `… [Part 1]`, `… [Part 2]` folders three or four deep inside a collection
pack, so a scan serves three quarters of a novel; `scripts/media-audiobooks.py`
re-files the tree so the property both servers' book scanners assume is true —
Audiobooks/<Narrator>/<Book> (<year>)/<the file keeps its own name>
```bash
# 1. The authority: Plex's chapter rows for this root (the DB opens read-only).
python3 scripts/media-census.py audiobooks --root /mnt/entropy/Audiobooks \
--out /tmp/ab_rows.tsv
# 2. What is on disk (stat only).
python3 scripts/media-census.py enumerate --root /mnt/entropy/Audiobooks \
--out /tmp/ab_enum.tsv
# 3. The plan.
python3 scripts/media-audiobooks.py --root /mnt/entropy/Audiobooks \
--enum /tmp/ab_enum.tsv --rows /tmp/ab_rows.tsv --out /tmp/ab_plan.tsv \
--exceptions /tmp/ab_exceptions.txt
# 4. Read the exceptions file FIRST. Then the same executor, journal and rollback
# as above, on its own journal, and refresh the Plex section as in step 5 of
# the movie Procedure — from inside the container, never over the host port.
scripts/media-apply.py apply --plan /tmp/ab_plan.tsv --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-audiobooks.jsonl --apply
```
The group is the identity Plex asserts in its own database — **(narrator, book,
year)** — so one book arriving in four part folders becomes one folder, and a
1965 and a 2020 recording that share a title stay two books.
Two laws are forced by what the rows actually hold:
+- **A chapter is never renumbered.** Most books contain chapters whose index Plex
does not know, so a number written into a name would be invented — and it would
buy nothing, because both servers order playback by the embedded track tag, not
by the file name. A wrong number in a name is the number a human later trusts.
Fix order in the tags: a Plex scan re-reads tags and never renames a file.
+- **The shelf is the narrator, not the author.** Every book has exactly one artist
row and it is the narrator; the author survives only inside some album titles
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving it would
mean parsing titles where a ripper wrote one and inventing it everywhere else.
The author stays visible in the folder name Plex itself shows.
What it refuses, the music arm's list with one addition: a video container, audio
no chapter row names, and a loose file in a folder holding two books all park under
`_unsorted` with the path they have today — and so does a chapter whose row names
**no book or no narrator**: a placeholder (`[Unknown Album]`, `[Unknown Artist]`)
that unrelated books share, or a chapter number mistagged as the artist (`0.4`).
`Various Artists` is not a placeholder — that is a multi-cast read.
This arm applies behind the music writer — one writer at a time — on its own
journal, and then refreshes the Plex section.
## Laws
- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
database keeps the release-folder path, so playback fails `Couldn't find the
file to stream` while the scanner works through the section. Never read such a
404 as a lost file: the file is where the journal says it is.
- **Plex's matched title outranks the filename**, and a Plex year of `0` means
*indexed but never matched* — trusting it named folders `Title (0)`, which is
a Jellyfin item with no year, no poster and no identity. A year that is not a
plausible year is no information.
- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
`V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
levels deep that neither matcher reads as a movie.
- **A folder inside a movie's folder is bonus territory even when its name is
misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
`Extras`; the vocabulary rule never fired, the featurette became a movie
folder, and its words then out-scored the real film and drew that box set's
other extras into it. The rule that replaced it needs no metadata: the folder
holding the longest film owns the shorter video filed under it, unless that
file's own name carries a different year — which is another film speaking.
- **Matching words are split on everything that is not a letter or a digit.**
Splitting on separators but not on `/` left tokens like `(1960)/extras` that
no other name could ever agree with, so a film could not be found by the
folder holding it, and extras drifted into a neighbour. A bare number counts
only when it could be a release year, so `1080` and `2160` never pose as
agreement between two names.
- **Compare two media libraries with the series engine, never a path
substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
letting whichever file a walk happens to reach first stand for an episode
paired two libraries into a fake duplicate — the same sloppiness pointed the
other way deletes a file that is unique. The comparison is `media-titles.ep_of`
on both sides: series, season, episode, bytes, and a hash before anything is
trashed.
- **A deletion is staged on the item's own filesystem.** The trash rule names
`~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
stage there is a copy rather than a rename, it has to fit in the 187 GiB of
free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
`/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
rolls back exactly like the move that made it.
- **A file whose name carries no words is bonus material, not a folder.**
`.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
is not a movie. It becomes a review line telling a human to name the file.
- **The executor renames and nothing else.** `rename(2)` cannot cross a
filesystem, so a plan that would is refused rather than silently copying tens
of terabytes.
- **The journal is a write-ahead log: the intent is fsynced before the rename.**
An interrupted run therefore leaves only two states — an intent whose move
happened, and an intent whose move never ran — and the next run finishes the
job without moving anything twice. A file that moved with no journal line did
not move through this tool, and the tool refuses to guess on its behalf.
- **The journal lives BESIDE the library root, never inside it** (default:
`media-relayout.jsonl` next to the root): a log inside the library is a file
Jellyfin indexes.
- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
duplicate destination stops the run *before the first rename*, so the operator
fixes one thing and re-runs instead of untangling a half-applied plan.
`--force` relaxes only the opinion guards; it never permits overwriting.
- **A sweep deletes only what the journal proves the run emptied.** A folder is
eligible when a journalled file moved out of it or out of something under it,
it now holds nothing but empty directories, no journalled destination lives
beneath it, and it is not a folder the run created. A folder nobody moved —
someone's placeholder — cannot be selected at all, and a symlink anywhere
inside disqualifies the subtree, because the thing behind a link is not this
tool's to delete. Every removal is journalled as `rmdir` and `rollback`
recreates the folder, so the sweep is undone by the same command as the moves.
Proof carries downward: inside a folder the journal proves emptied, every
empty directory goes with it. A release's `Screens/` or `Sample/` is never a
move source itself, so it can hold an emptied parent until the next sweep.
A failed `rmdir` is not a removal: only a folder that is really gone leaves
scope, so the next sweep finishes what a permission problem stopped.
- **An episode-shaped name marks a SERIES, not a film — and the series is the
name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
`Monk`. Keyed on the whole title instead, every episode became a group of one
and never reached the five-episode size that calls something a series, so a
box set was filed as 124 movies in 124 folders. The threshold still earns its
keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
in `_TV needs a TV library - <series>` for a human to move to a TV source —
the movie tool never files TV.
- **One journal is one undo.** The TV move runs on its own journal, and with
`--root` set to the share that holds BOTH libraries: the executor refuses any
path outside `--root`, and a journal shared with the movie relayout would
undo both runs when only one of them is wanted.
- **Sidecar metadata does not follow a rename, because it lives on another
filesystem.** The media servers write `.nfo` and artwork through the
[[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
array path, so where that view is mounted a sweep or a move must be followed by
`scripts/media-view.sh reap --apply` — stranded sidecars become ghost folders
that both servers browse and report as content. Whether this host mounts the
view at all is stated on that page; `media-apply.py` stays silent when it does not.
- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
whiteout in the upper layer, which hides the array's copy without removing it;
the file returns when the view is unmounted. Deletions in this pipeline happen
on the raw path only, which is what the journal can undo.
- **The music album key is case-folded; the folder name is not.** `Master Of
Puppets` and `Master of Puppets` are one album to the matcher, so the spelling
that reaches the folder is the one most tracks carry. Taking it from whichever
row sorted first made the folder name depend on row order, and re-planning an
already-filed library renamed albums that were already in place.
- **An occupied path is never a destination**, not even when its occupant is
scheduled to leave: the executor refuses an occupied target, so a plan that
aims at one fails after the renames that were supposed to clear it. A file
standing where the contract puts it is a keep, and a file already carrying a
collision suffix keeps that suffix — otherwise the plan is a shuffle, not a
fixed point, and a re-run moves albums that are already right.
- **`media-census.py music` reads the `BaseItemProviders` table, not the item's
JSON.** This Jellyfin build carries an empty `ProviderIds` inside
`BaseItems.Data` for albums, so a census that parses the JSON finds no
MusicBrainz or Spotify id on any album and reads as "the library is untagged".
## Rollback
One command, from the journal of the run:
`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
any folder a sweep removed, restores every source path, removes only the folders
that run created *and that are now empty*, and refuses everything if any moved
file cannot be accounted for. A rolled-back journal refuses to roll back twice.
## See Also
[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
## Citations
- Jellyfin Movie documentation — folder/file naming, extras subfolders,
`Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
<https://jellyfin.org/docs/general/server/media/movies>
+34 -4
View File
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
tags:
- dankserver
- plex
timestamp: 2026-10-21
timestamp: 2026-10-30
---
# Plex
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
quadlet (`plex.service`) under the systemd **user** generator, generated from
`dankserver/plex/quadlet/plex.container`. It shares the host with a second live
media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
from the same read-only Entropy Volume mount.
from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
**Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
only port PMS needs for client streaming; LAN clients connect directly,
@@ -26,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
| Media | git: `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only**; live today: `/mnt/entropy` itself, read-only — the view has never been installed. Either way every library path in the PMS database stays valid, zero re-scan |
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
@@ -38,7 +39,9 @@ public traffic can be fronted by Caddy like every other vhost.
The migration moved zero bytes: the container bind-mounts the native data
root where it already lived, and the media archive re-mounts at
`/mnt/entropy` inside the container, so every absolute library path in the
`/mnt/entropy` inside the container — only the HOST side of that bind has moved
(the array today; the overlay view in git has never been deployed), so every
absolute library path in the
PMS database keeps resolving. The one metadata change: rootless podman
cannot map the real host uid 997, so the fleet model applies — container
root runs as the `dank` user, and the deployer adopts the data root with
@@ -84,3 +87,30 @@ root runs as the `dank` user, and the deployer adopts the data root with
while the HTTP layer finishes loading; the first boot of a years-old
library took ~10 minutes (2026-09 cutover). The deployer gate now allows
600 s before failing; later boots are much faster (warm page cache).
## Triggering a scan
A rename on disk is invisible until the section is rescanned, and a moved file is
a `Couldn't find the file to stream` 404 until that scan passes it. Run this from
the host shell through `podman exec`, never from a neighbour container: PMS's
HTTPD closes without a single byte on any request that arrives from another
container's netns (10.89.x.x reads as "empty reply", not as a 401). The token
rides stdin, never argv, where any process on the host could read it out of
`/proc/<pid>/cmdline`; it lives in `Preferences.xml` inside the RW data bind.
```bash
umask 077
podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
'/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
podman exec -i plex sh -c 'read -r T
for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
-H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
done' </tmp/px.tok
```
Every section answers `204` on success. Proof that a rename landed is not a scan
response — `scripts/media-verify.py` replays the run's journal against the
database, which is the only read that distinguishes "not scanned yet" from "gone";
see [[Media Relayout]].
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
## Server Overview
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
## Hardware Specifications
### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
@@ -0,0 +1,287 @@
---
type: "howto"
title: "Radiance Architecture Plugins — declaring a model"
description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
tags:
- llm
- radiance
- plugins
- howto
timestamp: 2026-10-30
---
# Radiance architecture plugins
An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
fallback — so one family at two weight formats is two `.so` files, and branching on a quant
descriptor inside one plugin is the anti-pattern the split exists to prevent.
The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
beside the ABI and the in-tree plugins include them by that spelling.
## Exports
```c
const char* rad_arch_id(void); /* "llama", "qwen35", ... */
const char* rad_arch_quant(void); /* "" unquantised, else the descriptor served */
int rad_arch_serves_imports(void); /* optional: 0 declines an imported quantisation */
int rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out); /* optional */
int rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
void rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
const RadChatFormat* rad_arch_chat_format(const RadModelMeta* meta); /* optional */
const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta); /* optional */
```
Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
only its own containers returns 0.
## Declare: the metadata
`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
in a family.
**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
a derived value silently breaks projection widths, KV page size and the attention kernel's band.
`RadBuildCtx` is the deployment the plugin must declare against:
| field | meaning |
|---|---|
| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
| `cvec` | the loaded control vector, or null |
| `tp_attn_rows` | row-split prefill attention requested |
| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
| `weight_prefetch_off` | declare no weight prefetch beside collectives |
## Declare: weights
```c
static const RadWeightDecl d = {
.dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
.access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
.group = grp_layer(l), .optional = 0,
.n_row_parts = 0, .source = nullptr, .planes = nullptr,
};
rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
```
- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
`PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
`RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
all-reduce). The plugin says; the core does not infer. Uneven splits use
`rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
delta net's `[q|k|v|z]` of unequal part widths).
- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
because the unit of transfer is an expert, not a tensor.
- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
`[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
contiguous slice produces fluent wrong text.
- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
`planes` takes every plane.
- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
depends on the format (fp8 GEMM vs plain, rotated expert kernels).
## Declare: buffers and KV groups
`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
step.
`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
the whole history every step, so it cannot stream) are the three refinements.
## Declare: ops
```c
rad_op h = RAD_OP(b, "gemm_w4a8_nt",
RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
RAD_WEIGHTS(w));
rad_op_reads (b, h, RAD_BUFS(b_x), n);
rad_op_writes(b, h, RAD_BUFS(b_y), n);
```
- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
vocabulary has none, and the first plugin to declare it fixes its schema.
- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
for the whole program: sound but wasteful.
- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
fusion's `replaces` hook is a checkable equivalence, not a rewrite.
- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
emitting a fused form only an optional library defines.
- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
## Declare: drafters, encoders, generators
- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
`rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
before declare.
- Optional format exports let a model state what it knows and the server learn nothing new:
`rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
`rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
`rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
`struct_size`, so adding a field does not move the ABI.
**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
pipeline needs a core change.
## Step: issuing the graph
`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
(`n_spec`, `draft_pass`, …).
```c
static void step(RadCtx* c, const RadBatch* batch) {
if (batch->draft_pass > 0) { m.ds.step(c, batch); return; } /* query pass */
if (batch->draft_pass < 0) { m.ds.context(c, batch); return; } /* context pass */
RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
/* ...heads; the lm_head only over batch->n_out rows... */
}
```
- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
`RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
rolled back, and the model stays fluent while drifting. Issue the recurrent path over
`RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
`cu + n_seq_decode` with the token operands at their base).
- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
declared one (runtime LoRA) counts at that op's position.
- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
architectures' decode and prefill halves do.
- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
## Probe: what the plugin measured
`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
| field | effect |
|---|---|
| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
That is the design: a deployment switch the plugin cannot honour is an error, never a silent
substitution.
## Worked pattern: the smallest complete plugin
`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
the whole shape:
1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
op handles.
2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
`declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
`rad_note` lines explaining the shape.
3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
drafter.
5. `RAD_ARCH_PLUGIN(...)` at the bottom.
Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
ops that format needs.
## Verifying a new plugin
```sh
radiance --model m.rad --debug-graph # every op, its resolved kernel, and why
radiance --model m.rad --debug-placement # the plan
rad-convert $SRC -o m.rad --plan-only -v # a convert that plans will boot
rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
rad-tune -m m.rad --tp 2 # only at the shapes you serve
```
`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
same check `rad-convert` runs before writing.
## Related
- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.
@@ -0,0 +1,354 @@
---
type: "howto"
title: "Radiance Conversion — checkpoints into .rad containers"
description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
tags:
- llm
- radiance
- quantization
- howto
timestamp: 2026-10-30
---
# Radiance conversion
`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
running server.
Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
## What a conversion does
1. Read `config.json` (or GGUF metadata), and the shard index if present.
2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
recipe no kernel can serve is refused **here**, before any I/O.
4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
planes the matching quantiser writes, a block of rows at a time.
5. Measure every quantised weight against its source through the core's decoder and print the worst.
6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
```sh
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
--recipe data/recipes/q38-27b-bf16.recipe
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
```
## Every `rad-convert` option
| option | meaning |
|---|---|
| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
| `-o, --out FILE` | the container to write |
| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
| `--arch ID` | override the architecture id read from the checkpoint |
| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
| `--lora-strength F` | strength for every `--lora` (default 1) |
| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
| `--report N` | print the N worst quantisation errors (default 20) |
| `-v, --verbose` | per-weight detail |
`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
bytes** — and carries whatever calibration made the old planes. Use it only when source and
calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
change a weight the container holds or to add an expert.
## The recipe language
```
# PATTERN QUANTISER OPTIONS
blk.*.ffn_*_exps.*.weight gptq codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
blk.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
```
- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
`X.weight_scale_inv`). Anything else is named and refused.
- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
— `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
- An option the quantiser does not declare is refused, by name, before anything is quantised.
- Two plugins naming the same quantiser are refused, both named.
## Quantisers
| quantiser | what it does |
|---|---|
| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
| option | values |
|---|---|
| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
| `zero` | an integer zero point per block — an asymmetric grid |
| `transform=fwhtN` | rotate each row by the Hadamard first |
| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
| `clamp` | a signed code's range: `full` or `sym` |
Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
encoding model.
## Importing a release someone else quantised
These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
re-quantises from the exact dequantisation where the family's kernels want another form.
| `quantization_config` | read as | notes |
|---|---|---|
| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
The release's `quantization_config` names the tool, not a served form, so it is kept as
`imported_quantization_config.*`.
`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
a file size.
## The descriptor law (`--set`)
The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
writes it into the container. Storage format is not such a choice (the encoding is in the container
and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
re-derives the descriptor.
```sh
rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
--recipe data/recipes/q38-27b-fp8block-df2.recipe \
--set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
--set "quantization_config.weight_block_size=128 128" \
--set quantization_config.activation_scheme=dynamic
```
A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
## Inspect and validate the result
```sh
rad-info out.rad # architecture, source quant, recipe, sections
rad-info --recipe out.rad # the expanded rules — did the intended one win?
rad-info -v out.rad | head -40 # every tensor, its encoding, and what made it
rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
radiance --model out.rad --debug-graph # every op resolves
```
The converter's own error report (the worst `--report N` weights) measures each quantised weight
against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
[[Radiance Tools]].
## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
(default 65536) the run writes the files.
Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
push real prompts through prefill → convert again with the Hessians. The second moment is robust to
the first container's small quantisation error. A weight with no file is quantised uncalibrated and
the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
is given.
- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
the launches it sums.
- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
`H` by world size — GPTQ is invariant to a positive scale.
- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
`gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
cross-rank products zero. They must cover K and be in the model's own domain.
- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
per-column importances a quantiser uses, and the error report's weighting.
## Reuse, drafters, merging
```sh
# re-quantise only what changed
rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
# merge a PEFT adapter at convert time
rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
```
- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
no `lm_head` of its own; it uses the target's.
- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
(`qkv`) takes several targets and splits the adapter's rows in order.
- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
(Anima: diffusion model, text encoder, VAE). A name in two files is refused.
## `rad-export` — finetunes without hand-writing the command
```sh
rad-export ~/outputs/my-run/checkpoint-400 --dry-run # what it would run
rad-export ~/outputs/my-run/checkpoint-400 # adapter -> <name>.rad
rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
```
| flag | default | meaning |
|---|---|---|
| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
| `--memory M` | — | docker memory limit for the convert |
| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
| `--keep-merged` | — | keep the merged checkpoint beside OUT |
| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
(`adapter_config.json` + `adapter_model.safetensors`).
- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
anything is written. The adapter's tokeniser/template replace the base's where shipped.
- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
`rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
not the stock model's.
- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
`OUT.log`.
- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
`recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
drafter needs `huggingface_hub`. The image carries no Python; extract the script with
`docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
## Scoring a container (the KL mode)
```sh
# once, on the bf16 container
radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
# each candidate, same flags otherwise
radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
```
- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
disjoint from the calibration corpus.
- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
exists, which is written last.
- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
`--max-num-batched-tokens` costs pinned host memory as well as VRAM.
- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
the rest as the base, in the same steps — each document's rows should be byte-identical to the
all-adapter and all-base runs.
The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
[[Quantization Production]]; the mode here is only the instrument.
## Convert laws paid for in full
- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
of I/O on a large checkpoint.
- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
generator's largest pass may override `--max-tok`.
- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
each file physically holds, first-wins. A merged PLE shard must sort first
(`00-ple.merged.safetensors`) or a raw shard wins the name.
- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
table; a split tier (down=fp8, gate_up=w4) is unservable.
- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
## Related
- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
- [[frank-archon]] — the deployment whose recipes and convert laws these are.
@@ -0,0 +1,140 @@
---
type: "reference"
title: "Radiance Engine — the inference server behind frank-archon"
description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
tags:
- llm
- radiance
- reference
timestamp: 2026-10-30
---
# Radiance Engine — shape, plugin model, container
Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
llama-server's native one. Nothing in the serving path is Python.
The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
and its siblings own the engine reference; the live deployment shape, its Eras and its measured
laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
itself retired relative to its own rollback target.
Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
family, `OPS.md` for the op vocabulary, `spec.md` for the design).
## The four moving parts
```mermaid
graph LR
A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
B --> C[radiance engine]
C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
P1[kernel library .so<br/>libr4d, libavx, libref] --> C
P2[architecture .so<br/>one per arch id × quant descriptor] --> C
P3[quantiser .so<br/>libquant] -->|rad-convert| A
```
- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
tokeniser, HTTP server, media preprocessing.
- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
`gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
slow reference implementation every other library is checked against.
- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
quantisation descriptor) pair, with no fallback**.
- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
`ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
`rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
## Two phases, one static graph
`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
The core resolves every op against the kernel hierarchy per band, per domain, and completes the
whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
plan-only check possible.
Consequences a plugin author must respect:
- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
ask for the fused op, emit the chain when `rad_op_resolved` is false.
- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
it runs behind, or trunk and drafter buffers silently share bytes.
- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
## The `.rad` container (RAD2)
A container is one file holding everything the server needs: weights as **canonical encoding
planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
was made with, and any draft head. It names no kernel library.
- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
hooks), losslessly, so a kernel library can change layouts without reconverting the model.
- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
`givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
rotations and llama.cpp grids.
- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
encoding and what made it.
See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
## Selection key and version law
- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
--arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
unquantised checkpoint).
- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
number is refused at load, with both numbers printed. It moves on any shape change, not on a new
optional export; rebuild when it changes. Release version and ABI are independent: `/version`
reports the release, `rad-info --plugins` reports what loaded.
## Environment
| variable | meaning |
|---|---|
| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
the source's `getenv` calls; none is a deployment setting.
## Where the pages are
| Page | Owns |
|---|---|
| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
## Sources
- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
`docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
`core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).
@@ -0,0 +1,198 @@
---
type: "reference"
title: "Radiance HTTP API — endpoints, fields, refusals"
description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
tags:
- llm
- radiance
- api
- reference
timestamp: 2026-10-30
---
# Radiance HTTP API
One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
`model` in a request is accepted and **not enforced**.
## Endpoints
| route | what it serves |
|---|---|
| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
| `POST /v1/completions` | raw text completion |
| `POST /v1/embeddings` | an embedding head, else 501 |
| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
| `GET /v1/realtime` (WebSocket) | streaming transcription session |
| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
| `GET /metrics` | Prometheus |
| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
audio speech, models, responses and completions, for a client given the bare host as base URL.
`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
404.
A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
The model's declared BOS is prepended unless the prompt already opens with it.
## Chat
```sh
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
```
- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
replaces it). Use `/v1/completions` instead.
- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
**Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
`finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
model that answered nothing.
- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
`reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
under speculation and at any temperature. It needs a one-token end marker; otherwise the response
has no `completion_tokens_details` and a budget is a 400.
- `continue_final_message` continues a conversation ending in an assistant message instead of
answering it; `add_generation_prompt` opens a new turn.
- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
`parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
cannot combine with `response_format` or `grammar`.
- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
refusal.
## Streaming
```sh
curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
"max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
```
Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
token, so a client timing its first event measures time-to-first-token. `return_progress` adds
llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
chunk.
## Completions, embeddings, images, audio, video, tokens
- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
`preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
`dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
prompt. Pooling is the model's; the engine does not renormalise.
- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
(repaint only where transparent). `partial_images` streams previews as SSE.
- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
(Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
streams text in and audio out as each sentence completes.
- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
`GET /get_tokenizer_info` reports the tokeniser configuration.
## Request fields
Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
| field | notes |
|---|---|
| `model` | accepted, not enforced |
| `messages` / `prompt` / `input` | chat / completion / embedding |
| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
| `seed` | absent means "pick one", not 0 |
| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
| `priority` | scheduler hint |
| `ignore_eos` | generate to `max_tokens` whatever the model emits |
| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
| `echo`, `preserved_tokens` | `/v1/completions` only |
| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
| `images_config` | image models only: a named processor mode or an object restating served sizes |
## What is refused, ignored, and clamped
The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
into four classes.
- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
`speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
`dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
(empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
`n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
`grammar_triggers`, `preserved_tokens` (empty on chat).
- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
extension than a misunderstood request.
- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
`n_prompt_tokens`, `n_ctx`, and how many tokens must go.
## Anthropic, Responses, llama-server
- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
`ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
returns the prompt length.
- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
`function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
`reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
`response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
`namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
left out.
- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
## Auth, CORS, errors
- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
`/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
- CORS headers are sent unless `--no-cors`.
- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
`Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
accepted off value.
## Related
- [[Radiance Serve Options]] — the defaults every omitted field takes.
- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
- [[frank-archon]] — the harness contract built on this surface.
@@ -0,0 +1,142 @@
---
type: "reference"
title: "Radiance Model Families — arch ids, descriptors and recipes"
description: "Which radiance plugin claims which model family, and the conversion command and recipe of record for each."
tags:
- llm
- radiance
- models
- reference
timestamp: 2026-10-30
---
# Radiance model families
The selection key is the exact **(architecture id, quantisation descriptor)** pair, so a family at
two weight formats is two `.so` files. A checkpoint's id is its `config.json` `model_type` without
underscores (`qwen3_5` → `qwen35`); `rad-convert --arch ID` overrides it. `data/recipes/` is the
authority: **read the recipe's header** — it names the exact `rad-convert` line, its `--set` keys
and whether it needs calibration.
## The plugin table
| plugin | id | quant | what it serves |
|---|---|---|---|
| `llama_fp8` | `llama` | `fp8_e4m3` | plain LLaMA decoder (MiniCPM5, Llama 3, derivatives reporting `model_type: llama`) from a checkpoint holding E4M3 codes + `weight_scale_inv` |
| `llama_dense_fp8` | `llama` | *(empty)* | the same graph from a bf16 checkpoint, linears made block fp8 by a recipe |
| `qwen35_bf16` | `qwen35` | *(empty)* | Qwen3.5/3.6/3.8 dense hybrid (GDN + gated attention), bf16 |
| `qwen35_fp8` | `qwen35` | `fp8_e4m3` | the same, block-scaled FP8 weights and activations |
| `qwen35_w8a8` | `qwen35` | `w8a8_fwht128` | int8 trunk: i8 weights, bf16 scale per 128 of K, 128-wide Hadamard, int8 activations |
| `qwen35moe_fp8` | `qwen35moe` | `fp8_e4m3` | `qwen35_fp8` with a routed feed-forward |
| `qwen4exp_fp8` | `qwen4exp` | *(empty)* | Qwen4-Exp / Qwen3.8-Flash-Next: a bf16 checkpoint quantised at convert, or Qwen's FP8 release as it ships |
| `glm5next` | `glm5next` | *(empty)* | GLM-5.3-Flash: four mHC residual streams, 34 KDA + 11 sparse MLA layers with a DSA indexer, 288 MXFP4 experts top-8, block-FP8 attention/dense at f32 scales, MTP drafter, vision tower |
| `hyv3_bf16` | `hyv3` | *(empty)* | Tencent Hy-Memory-30B-A3B bf16; bf16/W4/W8A8/block-FP8/MXFP4 by recipe |
| `hyv3_fp8` | `hyv3` | `fp8` | Hy-Memory-30B-A3B-FP8: imported per-tensor E4M3, one dense layer then sigmoid-routed MoE |
| `hyv3_gguf` | `hyv3` | `gguf_ftype_15` | a llama.cpp Hy-Memory Q4_K_M GGUF, dequantised and recipe-converted |
| `gemma4_bf16` | `gemma4` | `compressed-tensors` | Gemma 4 31B dense (and E2B/E4B at one rank): sliding-window + proportional-RoPE global attention, GELU, softcapped logits; int4/Q8 codes per 128; assistant (shared-KV), DFlash or DFlash2 drafter; vision |
| `gemma4` | `gemma4` | *(empty)* | Gemma 4 text from Google's bf16 checkpoint (26B-A4B MoE, dense sizes) with z-lab's DFlash |
| `gemma3` | `gemma3` | *(empty)* | Gemma 3 dense (TranslateGemma 27B): sliding-window + global attention, GemmaRMSNorm, GELU; bf16 or int8/MXFP4/asym-int4 linears; DFlash2 drafter |
| `lfm2_bf16` | `lfm2` | *(empty)* | LiquidAI LFM2 / LFM2-VL, bf16, one rank |
| `embgemma2` | `embeddinggemma2` | *(empty)* | EmbeddingGemma 2: bidirectional encoder with Gemma 4 vision tower and audio conformer, pooled |
| `nomicbert` | `nomicbert` | *(empty)* | nomic-embed-text-v2-moe: bidirectional post-norm encoder, 8 experts top-2 on every second layer, mean-pooled |
| `nemotron35_embed` | `mistral3` | *(empty)* | Nemotron 3.5 Embed 1B/8B and Rerank 8B: bf16 text/image retrieval with a Pixtral tower |
| `qwenimage21` | `qwenimage21` | *(empty)* | Qwen-Image 2.1 / 2.1-Turbo image generator and editor (diffusers pipeline) |
| `krea2` | `krea2` | *(empty)* | Krea 2 Raw/Turbo: Qwen3-VL-4B tapped layers, 12.9B single-stream DiT, Wan 2.1 VAE |
| `sdxl` | `sdxl` | *(empty)* | Stable Diffusion XL and finetunes (Illustrious, NoobAI, Pony), single-file checkpoint |
| `flux3` | `flux3` | *(empty)* | FLUX.3 [klein] and base: image generation and edits from up to ten pictures |
| `flux3distilled` | `flux3distilled` | *(empty)* | the same plugin under the step-distilled release's own pipeline name |
| `anima` | `anima` | *(empty)* | Anima Base/Aesthetic/Turbo from the release's three single files |
| `qwen3asr` | `qwen3asr` | *(empty)* | Qwen3-ASR: Whisper-style audio encoder over a dense Qwen3 decoder, speech to text |
| `minimaxmusic3` | `minimaxmusic3` | *(empty)* | MiniMax-Music3: Qwen3-8B frame model, depth decoder, flow-matching DiT, DAC vocoder |
| `breeze` | `breeze` | *(empty)* | Breeze-TTS-2: T5Gemma 2 text encoder, Qwen3 backbone, 16-codebook depth decoder, Qwen3-TTS 12 Hz codec |
| `fishomni` | `fishomni` | `bsgemm-moe` | Fish Audio drama-3: Qwen3.5-35B-A3B backbone, block-FP8 experts at f32 scales, 13-layer codebook tower, Fish S2 DAC/codec |
| `minimaxh3` | `minimaxh3` | *(empty)* | MiniMax-H3 video with stereo sound: Qwen3-VL-32B text encoder, 33B DiT, tiled ViT decoder, BigVGAN audio |
| `ltx2` | `ltx2` | *(empty)* | LTX-2.5 base/distilled: Gemma 4 LM + text connectors, duration head, audio-video DiT, causal video VAE, audio VAE |
## Recipes of record, by family
`$SRC` is a downloaded checkpoint; `$CALIB` is a calibration directory (see
[[Radiance Conversion]]). Paths are shortened to the recipe name.
| target | command |
|---|---|
| Qwen3.8-27B bf16 | `rad-convert $SRC -o q38-27b-bf16.rad --recipe data/recipes/q38-27b-bf16.recipe` |
| Qwen3.8-27B block fp8 + DFlash2 | `rad-convert $SRC -o q38-27b-fp8.rad --recipe data/recipes/q38-27b-fp8block-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 --set "quantization_config.weight_block_size=128 128" --set quantization_config.activation_scheme=dynamic` |
| Qwen3.8-27B MXFP4 + DFlash2 | `rad-convert $SRC -o q38-27b-mxfp4.rad --recipe data/recipes/q38-27b-mxfp4-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3` |
| Qwen3.8-27B w4a16 (an int4 release) | `rad-convert $SRC -o q38-27b-w4a16.rad --recipe data/recipes/q38-27b-w4a16.recipe` |
| Qwen3.8-27B Quark AWQ-MXFP4 | `rad-convert $SRC -o q38-27b-quark.rad --recipe data/recipes/q38-27b-quark-mxfp4-df2.recipe --draft-model $DFLASH2` |
| Qwen3.8-27B ParoQuant int4/int5/MXFP4 | `rad-convert $SRC -o q38-27b-paro.rad --recipe data/recipes/q38-27b-paro-mxfp4-df2.recipe --draft-model $DFLASH2` |
| Qwen3.8-27B NVFP4 (compressed-tensors or ModelOpt) | `--recipe data/recipes/q38-27b-nvfp4-df2.recipe` or `q38-27b-modelopt-nvfp4.recipe`; every NVFP4 weight must take a rule (no kernel serves NVFP4) |
| Qwen3.8-Flash-Next bf16 reference | `rad-convert $SRC -o flashnext-bf16.rad --max-ctx 4096` (no recipe; served with `--weights-disk-tier`) |
| Qwen3.8-Flash-Next four-bit production | `CALIB=$CALIB rad-convert $SRC -o w4nl64-i8lm.rad --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe` (GPTQ experts + int8 trunk) |
| Qwen3.8-Flash-Next uncalibrated four-bit | `--recipe data/recipes/qwen4exp-w4.recipe` (rtn, no calibration) |
| Flash-Next from an AutoRound int4 release | `--recipe data/recipes/qwen4exp-autoround-i8-hc8m.recipe` |
| Gemma 4 31B w4a8 (no calibration) | `--recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe --set quantization_config.quant_method=compressed-tensors --draft-model $ASSISTANT` |
| Gemma 4 31B w4a8 + DFlash / DFlash2 | `gemma4-31b-dense-bf16-w4a8-df.recipe` / `-w4a8-df2.recipe`, same `--set` |
| Gemma 4 31B Q8 grid | `gemma4-31b-dense-bf16-q8-df.recipe` |
| Gemma 4 26B-A4B + DFlash | `gemma4-26b-a4b-df.recipe`; `-pad768` when the experts were padded to 768 (down may then be block fp8) |
| Gemma 4 E4B (Nemotron Bolt) | `gemma4-e4b-bf16-df.recipe`, `-i8g-`, `-a4-`, `-w4a8-`, `-w8a8-` variants |
| TranslateGemma 27B | `translategemma-27b-i8a8.recipe` (calibration container), then `CALIB=$CALIB translategemma-27b-mxfp4-gptq.recipe` (or `-df2`) |
| GLM-5.3-Flash | `rad-convert $AMD_QUARK -o glm53.rad --recipe data/recipes/glm53-mxfp4.recipe` |
| Hy-Memory-30B-A3B | `hy-memory-30b-a3b-bf16-fp8.recipe`, `-mxfp4`, `-w4`, `-w8`, `hy-memory-30b-a3b-fp8.recipe`, `hy-memory-30b-a3b-gguf-q4k-to-w4.recipe` |
| MiniCPM5-2B | `minicpm5-2b-dspark.recipe` (or `-bf16-dspark`) with `--draft-model $DSPARK` |
| Qwen3.6-35B-A3B | `q36-35b-a3b-fp8.recipe` |
| Qwen-Image 2.1 | `qwen-image-2.1-fp8.recipe` (Turbo uses the same file) or `-bf16` |
| Krea 2 | `krea-2-fp8.recipe`, `krea-2-bf16.recipe`, `krea-2-fp8-unscaled.recipe` for a plain-FP8 finetune file |
| FLUX.3 [klein] | `flux-3-klein-fp8.recipe` |
| SDXL / finetunes | `sdxl-bf16.recipe --tokenizer <openai/clip-vit-large-patch14>/tokenizer.json` |
| Anima | `anima-fp8.recipe` (or `-bf16`) + `--add` ×2 + `--config` + `--tokenizer2` + `--arch anima --set anima.variant=aesthetic` |
| LTX-2.5 | `ltx-2.5-fp8.recipe` (or `-bf16`); full DiT: symlink a folder whose `transformer/` is `transformer_full/` and `--set scheduler.use_dynamic_shifting=1 --set scheduler.shift_terminal=0.1` |
| MiniMax-H3 / FastH3 | `minimax-h3-fp8.recipe`, `minimax-h3-w8a8.recipe`, `minimax-h3-curve-*` for curve-form checkpoints |
| MiniMax-Music3 | `minimax-music3-bf16.recipe` |
| Breeze-TTS-2 | `breeze-tts2-bf16.recipe` |
| drama-3 | `drama3-fp8.recipe` |
| nomic-embed-v2-moe | `nomic-embed-v2-moe-bf16.recipe` |
| Unlimited-OCR | `unlimited-ocr-bf16.recipe` |
| Qwen3-ASR | no container: serve the checkpoint directory as it ships |
| EmbeddingGemma 2, Nemotron Embed/Rerank | serve the checkpoint, or convert bf16 with no recipe |
A checkpoint served at its own precision needs no container at all:
`radiance --model /srv/models/hf/Qwen/Qwen3.8-27B-FP8`. Convert when you want a tier only a container
has (quantised encodings, a drafter, a second vocab) or a smaller file.
## Family-specific traps
- **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden_size /
num_attention_heads` is 213; a derived value silently breaks projection widths, KV page size and the
attention kernel's band.
- **A fused projection shards by part, not by contiguous rows.** A tensor-parallel rank owns a slice
of *each* stacked projection in a fused `[gate|up]` or `[q|k|v|z]`; a contiguous slice would hand
rank 0 the whole gate, and the model would produce fluent wrong text. This is declared through
`RadWeightDecl::row_parts`, never inferred from the container.
- **Gemma 4 26B-A4B:** the routed experts' `gate_up` may be block fp8, but `down` stays bf16 unless the
experts were padded to 768 — hence the `-pad768` recipe.
- **GLM-5.3-Flash:** every weight is served as stored except the latent attention's small `kv_b`,
decoded to bf16. Its KV cache is bf16 only (the plugin refuses `fp8` at startup), and at 172 GiB it
needs `--tp 8`.
- **TranslateGemma 27B:** the checkpoint's template takes structured turns naming two languages and
refuses the plain turn the server renders at startup. The container therefore serves a plain Gemma 3
template, and a client sends on `/v1/completions` the prompt the checkpoint's own template renders.
- **ParoQuant containers made before 1.5.0 are refused at startup.** Convert again with the old
container as `--reuse`; every tensor that did not change is copied.
- **SDXL carries no tokeniser.** Pass CLIP's `tokenizer.json` beside the single file. A Krea 2
finetune shipped as one DiT file in Krea's key names is rewritten into a directory whose
`transformer/` holds it (`scripts/krea2_native.py`), with the release's other parts copied in.
- **Anima** ships as three single files: `--add` the text encoder and VAE, `--config` a Qwen3-0.6B-Base
directory (config + tokenizer), `--tokenizer2` ComfyUI's T5 tokenizer, `--arch anima`,
`--set anima.variant=…`.
- **drama-3's codec is `codec.pth`**, a `torch.save` file read as data, never unpickled.
- **Generators that load LoRA files declare their module aliases**; the checkpoint and a ComfyUI port
disagree in ways that load without error and generate garbage. The per-plugin alias table is in
`docs/ARCHITECTURES.md` ("Adapter key names of the generators").
- **Licences travel with the checkpoint, not the engine.** Qwen-Image 2.1 is non-commercial;
MiniMax-H3's licence excludes the EU, UK, Republic of Korea and USA; drama-3's community licence
needs a commercial licence to serve speech to others, and cloning a real person needs consent;
Breeze-TTS-2 is research/non-commercial.
## Related
- [[Radiance Conversion]] — the tool, recipe language and quantisers used above.
- [[Radiance Serve Options]] — the flags each family's compose file uses.
- [[Radiance Architecture Plugins]] — writing a plugin for a family not in this table.
- [[Quantization Production]] — the fleet's campaign and gates.
@@ -0,0 +1,240 @@
---
type: "howto"
title: "Radiance Operations — running, reading, restarting"
description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
tags:
- llm
- radiance
- operations
- howto
timestamp: 2026-10-30
---
# Radiance operations
This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
## Install
```sh
git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
./build.sh # configure, build for the cards present, run fast tests
./build.sh --no-hip # host backend only
cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201' # cards this machine lacks
cmake --install build # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
```
Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
for its plugins elsewhere.
- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
`share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
(faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
unaffected.
Run without Docker:
```sh
export RADIANCE_HOME=$PWD/build/radiance_home # plugins; an install finds its own
build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
```
`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
## Container
The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
```sh
docker run -d --name radiance --network host --init --ulimit memlock=-1 \
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
--group-add "$(getent group render | cut -d: -f3)" \
--group-add "$(getent group video | cut -d: -f3)" \
-v /srv/models/rad:/models:ro \
radiance --model /models/m.rad --tp 2 --host :: --port 8000
```
Each option is load-bearing:
| option | why |
|---|---|
| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
allocation and the heap below what the placement plan hands out.
Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
a chroot of the tree.
Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
model listens on the same port, and most use both cards, so **run one at a time**. The models
directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
work.
A plugin of your own is mounted and put first on the search path; nothing in the image changes:
```sh
-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
```
## Host prerequisites (upstream's two-R9700 checklist)
| item | setting |
|---|---|
| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
different box and its posture is on [[Dankserver System Specs]].
## Readiness: the receipt, then `/health`
Startup states every budget **before anything is allocated**, then loads. `/health` (and
`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
without an API key** — with every option and its source and the hardware read again now.
Read the receipt in this order:
1. `elastic` — what the experts and KV cache share after non-expert weights.
2. The split: `experts … / kv cache …`, and `unclaimed` if any.
3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
6. `device: the N cards reach each other's memory directly` or the failed pairs;
`expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
names the `--gpu-headroom-mib` that clears it.
A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
container stops at once and the unit's restart policy re-runs the same refusal. A unit in
`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
refusal**, not an engine fault: read the journal line and the argv, not the GPU.
## Lifecycle and restart
- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
`podman stop -t N`/`docker stop` is the graceful path.
- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
worker fault.
- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
container restart loop during a legitimate load is worse than waiting.
## Two engines on one set of cards
- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
step when another server shares the cards.
- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
(`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
mtime and the table's place, so a file never serves another model. The first engine fills it; a
second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
page against every process that pins it, so the pinned total is twice the table.
- The engine prefers discrete cards (all of the first one's architecture), integrated only when
there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
eighth of host RAM to the host.
## Logs and diagnostics
- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
step's token count (how a chunk geometry is confirmed in practice).
- The engine writes to stderr. Under a container, the journal holds only the container's own output;
a rootless `podman logs` covers only the current instance across recreations.
- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
evictions, preemptions and queue depth. `/stats`,
`/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
time and gap, for a machine with no scraper (its key comes from `--api-key` or
`RADIANCE_API_KEY`).
- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
`RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
## Verifying a deploy (fleet law)
`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
identity can serve a different model at a different shape while `/health` is 200. After deploying
any recently-benched service:
1. `systemctl show <svc> -p DropInPaths` — must be empty.
2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
meant.
3. The boot ledger — the shape you meant.
4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
probe with a solid colour and an exact expected word.
`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
## Security posture
- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
`/ping` and the dashboard; the key is a command-line argument, so it is readable in
`/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
boundary is the real control; do not run one where untrusted local users share the host.
- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
## Failure signatures at startup
| what you see | what it is |
|---|---|
| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
| "reports ABI N, engine M" | rebuild the plugin against this engine |
| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
## Related
- [[Radiance Serve Options]] — every knob and its law.
- [[Radiance HTTP API]] — readiness and the wire.
- [[Radiance Tools]] — how to corroborate a receipt.
- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
- [[Deployment Flow]] — how a committed change reaches the host.
@@ -0,0 +1,371 @@
---
type: "howto"
title: "Radiance Plugins — kernel libraries and quantisers"
description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
tags:
- llm
- radiance
- plugins
- howto
timestamp: 2026-10-30
---
# Radiance plugins — kernels and quantisers
Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
radiance's** (they compile against the installed `abi/` headers):
| kind | value | contract |
|---|---|---|
| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |
In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
plugin out of tree against an install.
## Identity and ABI
```c
#include "rad_abi.h" /* required; pulls rad_types.h and rad_encoding.h */
uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }
const RadPluginInfo* rad_plugin_info(void) {
static const RadPluginInfo i = {
RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
};
return &i;
}
```
- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
against. Return the macro; a mismatch is refused at load with both numbers printed.
- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
`"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
checks the library's actual code objects instead (§"A library for a new card").
- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
`rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.
## What a kernel library exports
```c
int rad_kernel_schema_count(void);
const RadOpSchema* rad_kernel_schema_at(int i);
int rad_kernel_count(void);
const RadKernelInfo* rad_kernel_at(int i);
```
A row declares one op implementation and what it can serve:
```c
static const RadConstraint c_gemm[] = {
RAD_CLE("M", 64), /* M <= 64 */
RAD_CDIV("K", 128), /* K divisible by 128 */
RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set */
};
static const RadKernelInfo k[] = {{
.name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
.computes = "row-major A times column-major B, fp8 in, bf16 out",
.shape = "M<=64, K%128==0",
.dtypes = "fp8a8, w4a8",
.domain = RAD_DOMAIN_DEVICE,
.priority = 50,
.constraints = c_gemm, .n_constraints = 3,
.launch = my_launch,
}};
```
- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
but is correct to 64 is wrong at concurrency.
- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
machine.
- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
supplies the op, however specialised a later library's row is. That makes a user's library an
override, not a competitor.
- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
`RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.
## Operands and extents
`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.
An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
everyday helpers in `rad_plugin.h`.
## Optional hooks
| hook | meaning |
|---|---|
| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
| `replaces` | the chain of conventional ops this kernel computes in one launch |
| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |
`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
ordinary parameters (`rad_args_geti("sk")`), with no variant index.
`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
is — the common case, and the only form that can be read in place from the file/mapped tier),
`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
touch is stored as it is.
## What a launch may do
On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
defines and serves these HIP entry points:
- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
- `hipMemset` (runs on the engine's queue and waits);
- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.
**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
and `hipStreamWaitEvent` on an engine stream are refused by name.
The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.
`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
`rad_store_bits` read and write any core dtype from a plane.
## Colours of agreement
- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
launches may overlap.
- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
did.
- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
`RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
device row). A reference library ranks below every unnamed library, is refused for serving unless
`--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.
## Defining a new op
Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
runs only on a library that implements it.
A schema is positional data, not code:
```c
static const RadParamSpec params[] = {
{"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE}, /* the ranged extent */
{"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
{"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
{"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
};
static const RadOperandSpec opds[] = {
{"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
};
static const RadOpSchema schemas[] = {
{"my_op", params, 4, opds, 3, "what it computes, for a human"},
};
```
`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
constraints and launch do the rest.
## Quantiser plugins
`abi/rad_quant.h` is the whole contract:
```c
int rad_quant_count(void);
const RadQuantizerInfo* rad_quant_at(int i);
typedef struct RadQuantizerInfo {
const char* name; /* what a recipe names */
const char* doc;
const RadQuantOption* options; /* key, RAD_P_INT|F64|STR, default text, doc */
int n_options;
RadQuantEncodingFn encoding; /* the encoding this quantiser would write */
RadQuantRowBlockFn row_block; /* rows per quantize() call; 0 = whole weight */
RadQuantizeFn quantize; /* logical rows -> planes */
RadQuantDecodeFn decode; /* optional, only for a scheme of its own */
} RadQuantizerInfo;
```
The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
no movement group), let the architecture declare against that encoding, then hand the weight over in
row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
the core's decoder can read it — that is how the converter measures error without knowing the method.
A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.
Two plugins naming the same quantiser are refused, both named.
### The encoding model (`abi/rad_encoding.h`)
An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:
```
q = code codes TILED, block [1x1], or [1xV] with a grid
q = grid[code][k] grid TABLE [entries x V]
q = table[code] table TABLE [1 x entries]
q = -q if the sign bit is set signs TILED u1
q = q - zero zero TILED
v = q * scale * scale.1 * ... every scale level
v = v - min * min.1 * ... a minimum in the value's domain
w = v . T^-1 the transform, per row over columns
```
So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
value and two scale levels.
- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
`perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
optional `t.scale`, per-part `t.part`, composed `t.dense`).
A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
is why a container never has to be reconverted when a kernel library changes its layout.
## Build, install, search path
```cmake
cmake_minimum_required(VERSION 3.21)
project(mykernels LANGUAGES CXX HIP) # CXX alone for a host-only plugin
find_package(radiance REQUIRED)
rad_plugin_defaults() # C++20, optimising build unless asked otherwise
rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
```
The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
against; `RAD_PLUGIN_INSTALL_HOME` overrides that.
**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
one. Your plugins go ahead of an installation without copying it:
```sh
RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
# in the container:
-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
```
The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
path). An entry matching nothing is warned about by name. With no list, every library found loads,
reference last.
## A library for a new card
The engine's own device code is compiled for every AMD family's generic target
(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
there depends on the installed kernel library.
- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
its host rows stay, and the log says what it carries versus what the card runs. One installation
can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
device memory is host memory and the budget leaves an eighth of host RAM to the host.
- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
`relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
rows like any other; the engine inserts none and assumes no link.
## Checking your work
```sh
rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
```
- It replays a recorded fixture of `libref`'s answers over real model geometries
(`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
completeness is the op-by-library matrix; speed is timed on the same bytes.
- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
`memcmp`.
- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
at every ISA level and with parallel regions forced onto several threads (for a library exporting
`avx_isa_force`/`avx_par_force`).
- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
works on a plugin the engine cannot load.
- `--debug-selection` prints which row took each band and which constraint refused each other
candidate; `/graph` on a running server shows the declared graph and which declarations the run
phase issued.
- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
declarations, reference status, whether its device code runs on this card, and its file.
## A worked example to copy
`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:
- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
`rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
refuse the device rows.
- `spechub.cpp` — the rows, their constraints and their argument readers.
- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
`-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
reference.
## Related
- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.
@@ -0,0 +1,240 @@
---
type: "reference"
title: "Radiance Serve Options — every flag and the law behind it"
description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
tags:
- llm
- radiance
- reference
- serving
timestamp: 2026-10-30
---
# Radiance serve options
`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
the fleet is pinned older — see the pin note on [[Radiance Engine]].
The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
pool.** The startup receipt prints every term before anything is allocated.
## Model, plugins, container
| flag | default | meaning |
|---|---|---|
| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
## Batch shape, context, scheduler
| flag | default | meaning |
|---|---|---|
| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
## Memory and placement
| flag | default | meaning |
|---|---|---|
| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
## KV cache, host tier, checkpoints
| flag | default | meaning |
|---|---|---|
| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
| `--no-prefix-cache` | cache on | disable prefix caching |
| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
## Speculation
| flag | default | meaning |
|---|---|---|
| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
## Tensor parallelism and collectives
| flag | default | meaning |
|---|---|---|
| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
## LoRA adapters
| flag | default | meaning |
|---|---|---|
| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
## Request defaults and bounds
A request that names a field wins. These set defaults for requests that do not.
| flag | default | meaning |
|---|---|---|
| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
| `--typical-p F` | 1 | typical sampling |
| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
## Server and HTTP
| flag | default | meaning |
|---|---|---|
| `--host ADDR` | 0.0.0.0 | listen address |
| `--port N` | 8000 | listen port |
| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
| `--read-timeout S` | 30 | a client may go silent this long while sending |
| `--write-timeout S` | 600 | one write to a client may block this long |
| `--keep-alive-timeout S` | 5 | idle connection hold |
| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
| `--no-cors` | CORS on | send no CORS headers |
## Media preprocessing
Each replaces what the container's preprocessor config states, and does nothing for a model that
takes no media.
| flag | default | meaning |
|---|---|---|
| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
| `--video-fps F` | container's | frames sampled per second |
| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
## Quality measurement (instead of serving)
| flag | meaning |
|---|---|
| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
See [[Radiance Conversion]] for the corpus and the scoring discipline.
## Diagnostics
| flag | meaning |
|---|---|
| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
| `--debug-selection` | the selection table only |
| `--debug-placement` | the placement plan only |
| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
| `--profile-ops` | per-op device timing |
| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
| `--live` | the terminal live view; redirected, one `key=value` line |
| `-v`, `-vv` | verbose, very verbose |
## The laws behind the knobs
**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
ceiling, where the extra host room shows up as unclaimed. Read the boot line
`kv: the pool backs N % of the worst case …`, never arithmetic.
**The activation arena comes off the budget before the split**, so both sides pay for
`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
prints which bound applies.
**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
kernels exist only for particular attention geometries, and the server refuses them elsewhere by
plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
funded is the resolver's answer.
**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
steps and a 2 % gain, because a new window is a new pass shape.
**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
across one.
**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
## Related
- [[Radiance HTTP API]] — what the flags expose on the wire.
- [[Radiance Operations]] — the container contract and readiness.
- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
- [[frank-archon]] — the measured deployment of this resolver on two R9700s.
+322
View File
@@ -0,0 +1,322 @@
---
type: "reference"
title: "Radiance Tools — inspect, check, tune, tokenise"
description: "The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports."
tags:
- llm
- radiance
- tools
- reference
timestamp: 2026-10-30
---
# Radiance tools
Six binaries and `rad-export` are installed beside `radiance`; `rad-avx-check`, `r4d_selftest` and
`rad-chatparse` live in the build tree. Every tool that loads plugins honours `RADIANCE_HOME`,
`RADIANCE_KERNELS` and `--kernels` as the engine does. A numeric option takes a whole number: `12k`,
`auto` (where the option does not name it) or an empty value exits 2.
`rad-convert`, `rad-kbench` and `rad-tune` run the engine's startup with a different last step: the
same plugins, in the same hierarchy, through the same declare phase, so they resolve kernels exactly
as a serve would.
## `rad-info` — what a container holds
```
rad-info [options] <model.rad>
rad-info --plugins [--home DIRS]
-v, --verbose list every tensor, not just the per-unit summary
--meta metadata table, then stop
--vocab vocab summary and normaliser chain, then stop
--profile N top N expert-profile entries (default 20, 0 = all)
--recipe the recipe the container was converted with, then stop
--plugins every plugin the search path offers, as the engine loads it
--home DIRS search path for --plugins (default $RADIANCE_HOME)
```
It opens the container read-only and loads no plugins (except for `--plugins`): a container names no
kernel library. Output is aligned; byte counts are binary units and each count says what it counts.
```sh
rad-info m.rad # format, architecture, model, source quant, recipe, sections
rad-info --recipe m.rad # the recipe, expanded, one rule a line
rad-info -v m.rad | head -40 # every tensor, encoding, shape and what made it
rad-info --plugins # kind, version, declarations, reference, device fit, file
```
The section table (strings, metadata, directory, planes, encodings, vocab, profile, data) is the
container's layout; the metadata table, the weight directory grouped by layer and movement unit, the
vocab, and the expert profile follow.
## `rad-kbench` — correctness, completeness, speed
```
rad-kbench [options] replay the recorded fixture (no model)
rad-kbench -m <model.rad> [options] check against a real container
rad-kbench -m <model.rad> --record FILE record a new fixture
rad-kbench -m <model.rad> --skeleton FILE write a metadata-only copy to record from
```
**Replay** needs no model, weights or card. Geometries, seeds and `libref`'s reduced answers are
checked in as `data/kernels.rkb` (dense model, block drafter, gqa 6) and `data/kernels_moe.rkb`
(routed MoE, MTP head, gqa 8). `--fixture` is repeatable; `ctest` runs three replays — the other
libraries against each recording, and `libref` against both at tolerance zero (its regression test,
since a live comparison moves both sides together).
Each library is asked for its row as the engine would ask it with that library alone: same
constraints, derived parameters and weight-encoding filter, so a row the engine would pass over does
not run here either (`--all-rows` runs every row whose constraints hold).
**Against a container** (`-m`), weight operands are read from it: each weight's canonical planes,
which the kernel gets through its own `layout`/`relayout` hooks and the oracle gets decoded, and
every expert of a weight table. This is the only test of the artifact and of the layout hooks.
**Recording** takes the reference's answer once at every point of every declared op, whichever
library resolved it. Weights are drawn from a seed, not read from the container, so any machine can
reproduce the reference and no card is needed. A skeleton (tables intact, data a hole) records the
same fixture in a few megabytes.
```sh
rad-kbench -m model.rad --skeleton model.skel.rad
rad-kbench -m model.skel.rad --max-cases 1 --record data/kernels.rkb
rad-kbench -m moe.rad --max-cases 1 --record data/kernels_moe.rkb
```
### Verdicts
- An output is compared on sampled positions, the full-tensor norm, and 32 seeded random-sign
projections (which estimate rel_l2 to about 25 %). The largest of the three is the error.
- Tolerances are per op and activation dtype (`dtype`, else `q_dtype`, else `kv_dtype`). Copies,
gathers, scatters and cache stores are exact. Every op in the fixtures has a row of its own.
- Ids are compared exactly; where the contract leaves a tie open (`sample_argmax`, `router_topk`,
`sample_topk`), an id indexing a value equal to the reference id's is accepted.
- A case whose reference and output are both identically zero checked nothing: **vacuous**, not
passed.
- A failure, a red-zone overrun, a fusion that differs from its chain, or a host kernel that wrote
into an input exits 1. `--strict` also fails a stale fixture.
- `--coverage DIR` keeps one file per fixture/library of the rows a replay reaches
(`data/kbench_rows/`); a listed row not reached fails, unless a memory skip kept its cases out.
- Every pass ends with a line saying where its time went (draw, hashing, layout, upload, launch,
read-back, comparison, cache, timing).
### Timing
`--bench` times every case that passes, on the bytes it was checked on.
- Device cases are timed **after** the checks, operands still on the device, after one sustained
warm-up. A time is device timestamps around a batch of back-to-back launches divided by the launch
count; the batch is queued behind a gate the host opens once submission is done, so neither the
host rate nor a sync is in the interval. Two untimed launches precede every batch; batches repeat
until the median's 95 % interval is within 1 % or `--bench-reps` is reached.
- **Cold by rotation**: weights and inputs of 1/32 of the footprint or more are read from copies
spanning the footprint (`--bench-flush`, default 256 MiB, at least twice an R9700's caches), so
each launch reads them from memory as streamed weights do. Smaller inputs stay warm, as the
previous op leaves them.
- In-place operands are restored from a pristine copy before every launch; the restores are timed
alone and subtracted.
- Timing draws expert offsets/routing skewed (Zipf 0.5) and block tables as permutations; the
checked launch keeps the recorded draw.
- The header prints the calibration: launch floor and streaming-read bandwidth warm and cold. If a
smaller cold probe runs >1.5× the largest's rate the rotation was cache-served and device times
are withheld. Throttler bits that came on during a batch repeat it, up to three times.
- Host cases are timed in their own column, after an untimed launch, weights rotated over at least
256 MiB (twice LLC), for at most two seconds a case.
### Flags
| flag | meaning |
|---|---|
| `--fixture FILE` | replay recording (default `$RADIANCE_HOME/kernels.rkb`); repeatable |
| `--record FILE` | with `-m`: write a fixture |
| `--skeleton FILE` | with `-m`: write a sparse container copy |
| `--rerecord FILE` | replay: write the fixtures' cases again with the reference's answers |
| `--record-rows FILE` | replay: record a row-driven fixture from the cases |
| `-m, --model FILE` | the `.rad` to read weight operands from |
| `--home DIR`, `--kernels A:B` | plugin home; kernel libraries (default every one found) |
| `--op NAME` | only this op (repeatable prefix match) |
| `--ref NAME` | the oracle plugin's name (default: the library declaring itself a reference) |
| `--no-ref` | replay without the reference among the tested libraries |
| `--strict` | replay: a stale fixture fails |
| `--all-rows` | run every row whose constraints hold |
| `--coverage DIR`, `--coverage-write` | the row-coverage ratchet |
| `--no-cache`, `--cache-dir DIR` | run the reference instead of reusing cached answers; cache location |
| `--seed N` | base seed; every case derives its own |
| `--tol X` | override every tolerance (bisecting only) |
| `--max-ctx N` | deepest context considered (declare and the attention sweep) |
| `--max-tok N` | `max_tok` declare runs at (default 512, small on purpose) |
| `--mm-max-patches N` | encoder pass declare runs at (default 2048 when a tower is present; 0 leaves it out) |
| `--kv-cache-dtype T` | `bf16` or `fp8`, as on the engine |
| `--num-speculative-tokens N` | draft depth to declare at (default: the container's) |
| `--max-cases N` | runs per distinct case (default 4; 0 = every one) |
| `--max-bytes MIB` | skip a case needing more than this, by name (default: half available memory, 512–8192) |
| `--no-fusion` | skip the fusion oracle |
| `--bench`, `--bench-iters N`, `--bench-reps N`, `--bench-warm-ms N`, `--bench-flush MIB` | timing |
| `--timing-only` | time every device kernel, check nothing (refuses `--record`) |
| `--report FILE`, `--analysis FILE` | markdown report; splice in `tools/kbench-analysis.sh` output |
| `-v` | every case, not only failures |
Notes: `--max-bytes` matters on a loaded machine — one case holds each operand as drawn, as the
kernel's copy, and on the device; without the cap a small machine is OOM-killed with no indication of
the op. `--bench` times cold on purpose: an R9700's 52 MB bf16 weight read warm comes from cache at
1.45 TB/s against 565 GB/s from memory, so warm times rank shapes but do not size a roofline. A
kernel that cannot lay out an encoding is a **skip**, not a failure (`RAD_E_DTYPE`), and the report
names the kernel, operand and encoding.
## `rad-tune` — measure the instantiation space
```
rad-tune -m <model.rad> [options]
-m, --model FILE the .rad whose declared graph supplies the shapes
--home DIR $RADIANCE_HOME; cache at <home>/tune/<machine>.tune
--kernels A:B kernel libraries to load, first wins
--op NAME only ops whose name starts with NAME
--machine KEY override the machine key (writing a cache for another machine)
--reps N timing reps, median reported (default 5)
--iters N launches per rep (default 32)
--max-tok N max_tok declare runs at (default 8192)
--tp N tensor-parallel width to declare at (default 1)
--num-speculative-tokens N|auto draft depth to declare at
--dry-run list what would be measured
-v, --verbose every candidate, not just the winner
```
Kernels declare tunable axes, not named variants; `rad-tune` walks the cross product at this model's
shapes and writes the winner per (op, geometry) to the cache the selector reads at startup.
- **Declare at the shapes you serve**: `--tp N` divides every sharded N; a declare at world size 1
measures a program no `--tp 2` serve runs. `--num-speculative-tokens` puts the drafter's kernels in
the measured graph.
- At `--tp > 1` every rank declares at once, each on its own thread and device: a collective's
`init` is a rendezvous, so declaring rank 0 alone hangs.
- A collective is measured as the exchange: per-rank operands (own seeds, so identical halves cannot
pass), own scratch and stream, own instance; every rank fires at once and the clock stops when the
last stream drains. Non-collective rows run on rank 0 alone.
- `RADIANCE_TUNE` pins a kernel's axes to compare a tuned cache against the defaults — a measurement
aid, not a deployment setting.
## `rad-schemas` — the op vocabulary diff
```
rad-schemas libref.so print one plugin's vocabulary
rad-schemas libr4d.so libref.so diff them, in hierarchy order
```
The first plugin in hierarchy order to declare an op fixes its schema; a later plugin declaring the
same op differently is refused at load, both named (arguments are positional, so disagreeing schemas
would make one call mean different things). `rad-schemas` prints every disagreement at once, so a new
plugin can be conformed before it is loaded beside another. It exits 1 on any disagreement. It links
no core — it `dlopen`s plugins and reads static tables, so it works on a plugin the engine cannot
load.
## `rad-avx-check` — host libraries without a card
```
rad-avx-check --ref libref.so --avx LIB.so [options]
--ref PATH the oracle plugin (default libref.so)
--avx PATH the plugin under test (default libavx.so)
--op NAME only this op
--level N only this ISA level (0 scalar, 1 avx, 2 avx2, 3 avx512)
--threads N rerun every case with each parallel region on N threads (default 4; 0 skips)
--tol X override every tolerance
-v every case
```
Checks a host library op by op against `libref`, on the CPU, with neither engine nor card: two
`dlopen`s, a geometry table (`libavx/avx_cases.cpp`), libref's operand descriptions and hand-built
cases. For every case: schemas as the loader compares them, outputs within `rad-kbench`'s tolerances,
guard bytes, inputs unchanged, and an oracle output that is not just the draw. `libavx` is also
checked at every ISA level the CPU executes and with parallel regions forced onto several threads.
A new host library is checked with the same command; it is treated as a host library (not `libavx`),
so libref ops it has no row for are listed rather than failed, and the ISA/thread sweeps run only if
it exports `avx_isa_force`/`avx_par_force`. A row returning `RAD_E_UNSUPPORTED` for a geometry libref
serves fails: the selector would resolve to it and the step would fail at issue.
## `r4d_selftest` — libr4d's own check and timer
```
r4d_selftest <libr4d.so> [--case <row>]
r4d_selftest <libr4d.so> --perf N K M [iters] (and the other --perf* modes in its usage)
```
With no mode it runs the catalogue and static checks, then each row's smoke and numeric cases in a
child process of its own, with fusion claims compared byte for byte (`ctest` labels `slow`, `gpu`).
`--case` runs one row in-process as the reproducer. `--perf` covers the fp8 decode GEMMs and head;
`--perfmxd` the dense OCP-MXFP4 rows with the weight bandwidth each reaches; `--perfmx` the MXFP4
expert GEMM; `--cold MIB` times each launch alone behind a cache flush. `--perfar8` prints the
eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at every piece count;
`R4D_SELFTEST_ONE_CARD=1` runs them on one card for their bits only.
## `rad-tokcheck` — tokeniser against its references
```
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus basename>.ids
(little-endian int32, encode(text, add_special_tokens=False))
--no-ref skip the reference encoder (slow)
--user encode with parse_special off
--turns N turns in the multi-turn measurement (default 40; 0 skips)
--repeat N timing repeats, best taken (default 3)
--context N ids shown either side of a difference (default 6)
```
Per corpus file: ids match `tools/tokref`'s reference encoder (llama.cpp's splitters and
`std::regex`); ids match HuggingFace `tokenizers` read from files made elsewhere; and speed — MB/s
for both encoders, plus the cost of re-tokenising a growing conversation with and without the
segment cache. Any difference prints the ids and surrounding text and exits nonzero, so it gates a
change.
## `rad-chatparse` — where a tool call goes wrong
```
rad-chatparse --request REQ.json --rad MODEL.rad --reply R.txt
rad-chatparse --render-raw T.jinja --context CTX.json
```
Renders one OpenAI chat request as the server would (container template or
`--override-chat-template`, `--chat-template-kwargs`, the plugin's declared reply format, or the
template's own with `--no-plugin`), then reads each reply file as the model's output: parsed whole,
parsed through the SSE stream as a client reassembles it, and run through the request's tool-call
grammar. It prints one JSON object; exit 1 when the two parses disagree or the grammar refuses a
token. No weights are loaded.
## The KL mode
`radiance --kld-record/--kld-ref/--kld-corpus/--kld-out/--kld-lora` is a mode of the engine, not a
separate tool: same placement and kernels, prefill instead of listening, then exit. The flags and the
corpus format are on [[Radiance Conversion]].
## Shell harnesses (`scripts/`)
They drive a running engine: bring a model up, measure it, assert on what came back. Each header
records why it measures the way it does.
| script | what it does |
|---|---|
| `mcserve.sh`, `fnserve.sh` | bring MiniCPM5-2B+DSpark / Flash-Next up and leave it running (env knobs: `P TP MLEN SEQS M KVD HR MNBT EXTRA …`) |
| `fnstop.sh` | stop whichever is running and wait for exit |
| `convdspark.sh`, `convfn.sh` | rebuild the two reference containers (`convfn.sh` needs `CALIB=`) |
| `hfget.sh` | resumable Hugging Face fetch with a content manifest |
| `fnpf.sh`, `fnconc.sh`, `cstep2.sh`, `ident.sh`, `cardclock.py` | attach to a live server: prefill rate; prefill-vs-decode contention; steady-state decode ms/step per concurrency; run-to-run hashes; which clock limit binds and which rank the TP step waits for |
| `fndrop.sh` | disconnect a client mid-request and check every other answer against a reference |
| `tierident.sh`, `tiercross.sh` | whether a session that left VRAM returns byte for byte (attention and recurrent state), including two sessions in flight |
| `mctools.sh`, `mcstream.sh`, `mcagent.sh`, `mcdecl.sh` | tool-call parsing shapes; streamed tool calls on raw SSE; speculation on an agentic workload; whether the drafter proposes |
| `dprofd.sh`, `fntrace.sh`, `mctrace.sh`, `oracle.sh` | per-op device timing; rocprofv3 traces; every op against libref on the operands the model issued |
| `kv-parity.sh`, `kv-stage-parity-64k.sh` | KV streaming: covering ring and staged prefill are byte-identical to streaming off |
**Not all of them leave a server alone.** `fnserve.sh`, `mcserve.sh`, `fnstop.sh`, `fntrace.sh`,
`dprofd.sh`, `mctrace.sh`, `mcagent.sh` and `mcdecl.sh` kill any running engine (`pkill` on the
binary), whatever port it is on; `oracle.sh` stops whatever listens on its own port. `cstep2.sh`,
`ident.sh`, `tierident.sh`, `tiercross.sh`, `mctools.sh`, `mcstream.sh`, `hfget.sh`, `convfn.sh`,
`convdspark.sh`, `fnpf.sh`, `fnconc.sh`, `fndrop.sh` and `cardclock.py` leave it running
(`fndrop.sh` deliberately disconnects clients). Never point the stopping ones at a production port.
## Related
- [[Radiance Conversion]] — `rad-convert`, `rad-export`, the KL mode.
- [[Radiance Plugins]] — what `rad-kbench`/`rad-tune`/`rad-schemas` check.
- [[Radiance Operations]] — the startup receipt these tools corroborate.
+88 -23
View File
@@ -91,27 +91,36 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=66560` **per rank** (raised 2026-10-29 from 49152 — see *Eras*) — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** (raised to 66560 on 2026-10-29, folded back the same day — the A/B under *Laws* shows the raise bought no cache) — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
| TTM gate | `TTM_MIN_MIB=197000`, `TTM_TARGET_MIB=224000` |
| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
the bigger cache, paid over PCIe when those experts are hit.
Boot ledger for the live payload. Last boot: 2026-10-29 at `MAX_SEQS=4`,
`HOST_POOL_MIB=66560`, `MAX_BATCHED_TOKENS=2048`; the live pool is `49152` again, and the
two boots are **identical above `host_pool`**, so one ledger serves both shapes:
`elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
`activation arena 990.36 MiB` (the price of the 2,048-token step: `256 tokens use
155.60 MiB, 1024 tokens use 472.86 MiB of 896.20 MiB`), `prefill staging 373.65 MiB`,
`vram_weights 24.30 GiB`, `host_pool 65.59 GiB` (opened with `0 B used`; `48.59 GiB` at
the 49,152 boot), `unclaimed 96.89 MiB`; referee line `kv: the pool backs 93% of the worst
case (4 sequences all at 200000 tokens), 3.7 full-length sessions`; `device: the 2 cards
reach each other's memory directly (peer to peer)`; sampler **12 rows**
(= `seqs × (1 + speculative depth)`), 4 hardware queues a card (133.45 MiB), `already held
256.67 MiB`, mover `17519 slab slots, 7624 pinned pool slots, 3.58 GiB static vram`,
`load: rank 0, mover took 24.30 GiB`.
The A/B that ended the raise: the 2026-10-20 boot at `HOST_POOL_MIB=49152` printed the
**same** `elastic 26.73 / experts 20.72 / kv cache 5.64 / vram_weights 24.30` (Laws: the
lever saturates at the expert floor), and the campaign rig — the other payload that
inherits this env — moved `vram_kv 269.99 → 271.25 MiB`, a rounding step, because its
window is set by its own n-gram table (95.37 GiB) and expert plane rather than by the
pool. The refused calibration documents are fixed by cutting them to the window, not by
widening the pool.
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -150,7 +159,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-20 → 10-29 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 4 × 200,000 fp8 KV — resolver cache 5.64 GiB = 747,492 tokens, `prefill staging 373.65 MiB`, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`; host pool 49,152 MiB a rank, `TTM_MIN_MIB=162000` / `TTM_TARGET_MIB=204800`, `queue_cap 32`, sampler 12 rows, `--prefill-interleave 1` | the `qwen38-w5nl64` campaign rig booted this pool at the shared 1,024 step and reported `kv cache 269.99 MiB a card: 7436 tokens`, so 682 of its 1,294 calibration documents were refused; Frank ordered the pool **and** the step raised to production's widest legal step (2,048) as one shape | revert the `feat/frank-archon-pool-step` commit and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `HOST_POOL_MIB=49152`, `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
| 2026-10-20 → 10-29 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 4 × 200,000 fp8 KV — resolver cache 5.64 GiB = 747,492 tokens, `prefill staging 373.65 MiB`, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`; host pool 49,152 MiB a rank, `TTM_MIN_MIB=162000` / `TTM_TARGET_MIB=204800`, `queue_cap 32`, sampler 12 rows, `--prefill-interleave 1` | Frank ordered the campaign's step (2,048) onto production; the pool half of that order (66,560) booted, came back byte-identical to this row's cache, and was folded back the same day — the calibration documents it was meant to save were cut to the window instead (`scripts/rechunk-kld-corpus.py`) | the live shape is this row's pool with `MAX_BATCHED_TOKENS=2048`: the pair is the rollback unit, not the merge commit (`scripts/frank-archon.env`) |
| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
@@ -244,7 +253,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
| 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
| 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
| 4 × 200,000, host pool 49,152 (live) | 800,000 tok | 5.64 GiB = 747,492 tok | 93 % |
| 4 × 200,000, host pool 66,560 (2026-10-29 A/B, folded back) | 800,000 tok | **5.64 GiB — byte-identical to 49,152** | 93 % |
The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
`ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
@@ -552,6 +562,52 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
### The quantization pipeline
- **A calibration tap is only a tap when the rig that opened it is running.** The runner writes
`.rig-boot` into the tap dir (container, image, pid) and retires it when its boot exits non-zero;
`calib-push.py` refuses without that marker, refuses when it names another container, and refuses
when the named container is not running; `qwen38-calib-window.sh` stops on a failed phase instead
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
- **A calibration step has an OPTIMUM, and it is not the widest step that boots.** The tap's cost is
expert re-reads per document (a 7,168-token document cut into four 2,048-token steps re-streams
each layer's routed expert union four times), which argues for the widest step — but the measured
curve turns over: at the 32,768 calibration window one 2-way prefill of the same document ran
**106 tok/s at an 8,192-token step and 76 tok/s at 16,384**, with the computed activation arena
growing 3.39 → 4.50 GiB, so the wider step pays more compute than it saves in reads. The
200,000-window `QSA score plane` refusal that made the wide step look impossible is an artifact of
serving the production window: the same 16,384-token step needs a 512 MiB plane at 32,768.
`Q38_CALIB_MNBT` is the only knob — the KL modes keep production's step, because their numbers are
only comparable at one shape.
- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
at second zero), and the identity check written to catch that compared two different number spaces:
`podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
The binding is `podman top <ctr> hpid` against the listener from
`ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
`scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
`No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
while the reference recording and every score boot succeeded at the same pool and slot count.
Calibration only needs the routed weights prefilled, so it reads the table from the container's
file tier (`--ngram-placement disk`); kldref and score keep production's placement, because their
numbers are only comparable at an identical shape.
- **Acceptance is three metrics and every slice, not a mean.** `scripts/kld-gate-qwen38.sh` grades
mean KL, the p99.9 tail, top-1 agreement and every `by_source` slice (code, prose, tools):
a mean can hide a few positions taking the whole hit, and an overall win can be bought by
regressing one distribution. The FP8-native anchor is an interval, not a requirement — without
the 172.8 GiB fetch the gate runs in two-report mode (candidate vs the served payload) and says
so on its first line.
## Contract (the wire)
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
@@ -628,11 +684,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
`scripts/qwen38-froggeric-v22.5.jinja`.
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
- The campaign (qwen38-w5nl64, live): `scripts/run-qwen38-bf16.sh` (kldref / calib / score — the
ONE measuring shape), `scripts/qwen38-calib-window.sh` (the gated calibration window),
`scripts/convert-qwen38-w5nl64.sh` with
`scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe` (the arm driver and its template),
`scripts/kld-gate-qwen38.sh` (acceptance), `scripts/calib-push.py`,
`scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
`scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
corpus tools share — one `n_tokens`, so every tool counts the same way),
`scripts/derive-protected-experts.py`, `scripts/apply-protected-cast.sh`.
- Retired swift15-era campaign scripts, superseded by the above and kept only as reference:
`scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
`scripts/run-swift15-bf16.sh`, `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`,
`scripts/verify-swift15-ple.py`.
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
`scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
@@ -642,5 +706,6 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
`scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
---
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], and the engine reference —
[[Radiance Engine]], [[Radiance Serve Options]], [[Radiance Conversion]], [[Radiance Plugins]],
[[Radiance Operations]].
+21 -14
View File
@@ -1,6 +1,6 @@
---
type: reference
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
tags:
- llm
@@ -10,10 +10,10 @@ tags:
- gfx1151
- halogen
- endpoint
timestamp: 2026-10-21
timestamp: 2026-10-30
---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
# frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
(0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
@@ -190,9 +191,11 @@ created it — earlier topologies in .
## Contract (wire)
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
wall-clock plus `usage.prompt_tokens`.
- Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
`/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
`Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
`usage.prompt_tokens`.
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
@@ -232,11 +235,15 @@ created it — earlier topologies in .
XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`,
`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts
away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2`
(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3.
is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
`qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
`embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
`d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
size and sha256 against the catalog and the payload mount carries the tree in.
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,
+11 -3
View File
@@ -1,6 +1,6 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 12118), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
tags:
- llm
@@ -31,7 +31,7 @@ is the bug.
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
| Runtime | `/home/dank/GitPackages/llama-prism-0244744/bin/llama-server` — PrismML **fork**, `0.7.0-dev (build 12118, commit 024474448)`; the only loader for this payload |
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
@@ -136,6 +136,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-30 → | same 9B MTP file, Q8_0 mmproj and refusal adapter (payload untouched) | PrismML fork **12118** (`0.7.0-dev`, `024474448`) | unchanged (3 × 200,704; f16 K + q8_0 V; MTP depth 2) | fork rebuilt from `origin/prism`; the payload gate held — 202 `pq2_0` tensors and 202 Hadamard-folded weights loaded, MTP drafting engaged, chat/tools/vision answered | `LLAMA_BIN` back to `llama-prism-b10777/llama-server` (kept on disk) + redeploy |
| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
@@ -201,7 +202,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter.
- **A fork rebuild needs an install RPATH and a bin/lib-aware `LLAMA_BIN`.** `cmake --install` strips the
build-tree rpath, so the installed `llama-server` dies on `libllama-server-impl.so` unless the configure
carries `-DCMAKE_INSTALL_RPATH='$ORIGIN;$ORIGIN/../lib'`; the installed layout is `bin/` + `lib/`, not the
flat directory b10777 shipped, so the env's `LLAMA_BIN` has to name `…/bin/llama-server`. Build 12118 was
made from `origin/prism` with the 10777 configure (Release, `GGML_CUDA=ON`, `sm_86`,
`CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13`, Ninja) — gcc 15 is the box default and nvcc 12.4 rejects it.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. Build 12118 clears all
three.
## Contract
+14
View File
@@ -13,6 +13,20 @@ the endpoint's **Eras** table, and the numbers a retired era measured are gone.
| [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. |
| [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. |
## Radiance engine reference
| Page | Owns |
|---|---|
| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
## Retired endpoints
| Page | Owns |
+12 -1
View File
@@ -49,8 +49,10 @@ is a locator: one row, one clause, no numbers.
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. |
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. |
| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status. |
| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
| [[Media Metadata View]] | The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback. |
| [[Media Relayout]] | Census, plan and apply the movie, TV, music and audiobook libraries: the gates, the refusals, the rollback. |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
@@ -162,6 +164,15 @@ is a locator: one row, one clause, no numbers.
| [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. |
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
| [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. |
| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. |
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
+10
View File
@@ -133,6 +133,16 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
[ -z "$output" ]
}
@test "a compose edit never restarts a Quadlet-owned media server" {
# dankserver/compose.yaml is an inventory; the media servers' Quadlets own
# their containers. A compose edit that restarted them would cut real viewers
# for a change that cannot affect them.
run python3 "$PLANNER" --target dankserver --paths dankserver/compose.yaml
[ "$status" -eq 0 ]
! printf '%s\n' "$output" | grep -qx jellyfin
! printf '%s\n' "$output" | grep -qx plex
}
@test "compose planner narrows a compose edit to the changed service" {
local base head
base=$(git -C "$REPO_ROOT" rev-parse HEAD)
+1 -1
View File
@@ -158,7 +158,7 @@ nosudo_bin() {
# so the pool's token count is a boot-ledger line — verified live by the
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=66560$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache: pinned host RAM only (the disk tier was removed), and the
+1 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bats
# Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11)
# of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.18.20,
# digest-pinned), driven by the root unit's serve script — 2 slots over ONE
# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
# and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
+15 -4
View File
@@ -126,11 +126,22 @@ run_stage() {
@test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" {
# The payload path IS the endpoint's identity, and the runtime is part of
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata
# make the PrismML build the only loader, and this build must be >= b10776
# make the PrismML build the only loader, and that build must be >= b10776
# because MTP beside q8_0 V is illegal below it. A retarget that moves the
# model but leaves the old build behind ships an endpoint that cannot load
# (type 142) or cannot run the documented MTP arm (the build gate).
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE"
# model but leaves a stock llama.cpp behind ships an endpoint that cannot
# load (type 142) or cannot run the documented MTP arm (the build gate).
#
# Assert the fork FAMILY, not the directory one rebuild happens to install:
# a literal path turns every runtime move into a test edit, and the suite
# runs on a box that has no /home/dank at all, so existence cannot be
# checked here either.
local bin
bin="$(sed -n 's/^LLAMA_BIN=//p' "$ENV_FILE")"
if [[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/llama-server$ ]] &&
[[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/bin/llama-server$ ]]; then
echo "LLAMA_BIN does not name a PrismML fork build: '$bin'" >&2
return 1
fi
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE"
+297
View File
@@ -0,0 +1,297 @@
#!/usr/bin/env bats
# Seam: scripts/jellyfin-providers.sh — the only mechanism that puts a
# metadata-provider key into Jellyfin's plugin configuration on Dankserver.
#
# Output vocabulary the tests depend on:
# "+ <action>" something this run will change
# "= <fact>" something already true, so nothing changes
#
# The contracts, in the order the seam tests them:
# 1. plan reports every action and never prints a key;
# 2. a secret file that is not mode 600 is refused by name and nothing is
# written — a key readable by other users is an error, not a warning;
# 3. a downloaded plugin artifact is accepted only on its pinned sha256, its
# official guid and its pinned version (a swapped or downgraded artifact
# is refused before anything lands in plugins/);
# 4. the TMDb key is written into the EXISTING config with every other
# setting preserved — Jellyfin owns that file, and clobbering it would
# silently reset country, language and adult settings;
# 5. a TVDB config the operator already saved through the UI is kept;
# 6. install is idempotent: a second run changes no byte;
# 7. whatever is replaced is copied to the trash first (fleet law: never
# overwrite without a recoverable copy);
# 8. verify answers whether the endpoint is configured, key LENGTH only;
# 9. no plugin configuration XML is tracked in git, the deployer places the
# keys BEFORE it restarts Jellyfin, and the planner routes a change to this
# installer to jellyfin.
load helpers
PROV="$REPO_ROOT/scripts/jellyfin-providers.sh"
TMDB_KEY="tmdbk3a9Zq7Rtestkey"
TVDB_KEY="tvdbk8b2Wt4Ptestkey"
GUID_OFFICIAL="a677c0da-fac5-4cde-941a-7134223f14c8"
# make_plugin <file> <guid> <version> — a zip shaped like the official TheTVDB
# artifact (meta.json + its two DLLs), so extraction is exercised for real.
make_plugin() {
python3 - "$1" "$2" "$3" <<'PYEOF'
import json, sys, zipfile
path, guid, version = sys.argv[1], sys.argv[2], sys.argv[3]
meta = {
"category": "MoviesAndShows",
"guid": guid,
"name": "TheTVDB",
"owner": "jellyfin",
"targetAbi": "12.0.0.0",
"version": version,
}
with zipfile.ZipFile(path, "w") as z:
z.writestr("meta.json", json.dumps(meta))
z.writestr("Jellyfin.Plugin.Tvdb.dll", "dll-bytes")
z.writestr("Tvdb.Sdk.dll", "dll-bytes")
PYEOF
sha256sum "$1" | cut -d' ' -f1
}
setup() {
JF="$BATS_TEST_TMPDIR/jf"
TRASH="$BATS_TEST_TMPDIR/trash"
SEC="$BATS_TEST_TMPDIR/secrets"
mkdir -p "$JF/plugins/configurations" "$TRASH" "$SEC"
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml" <<XML
<?xml version="1.0" encoding="utf-8"?>
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<MetadataCountryCode>US</MetadataCountryCode>
<PreferOriginalLanguage />
<Language>en</Language>
<EnableCinematicForTrailers>false</EnableCinematicForTrailers>
<IncludeAdult>false</IncludeAdult>
<MaxCastMembers>25</MaxCastMembers>
<AddGenresFromKeywords>true</AddGenresFromKeywords>
<TmdbApiKey />
</Configuration>
XML
KEYS="$SEC/provider-keys"
printf 'TMDB_API_KEY=%s\nTVDB_API_KEY=%s\n' "$TMDB_KEY" "$TVDB_KEY" >"$KEYS"
chmod 600 "$KEYS"
ZIP="$BATS_TEST_TMPDIR/thetvdb.zip"
PIN_SHA="$(make_plugin "$ZIP" "$GUID_OFFICIAL" 24.0.0.0)"
PROV_ENV=(
JF_ROOT="$JF"
PROV_KEYS="$KEYS"
TRASH_DIR="$TRASH"
TVDB_PLUGIN_URL="file://$ZIP"
TVDB_PLUGIN_SHA256="$PIN_SHA"
TVDB_PLUGIN_GUID="$GUID_OFFICIAL"
TVDB_PLUGIN_VERSION=24.0.0.0
)
}
prov() { env "${PROV_ENV[@]}" bash "$PROV" "$@"; }
tree_fingerprint() { find "$JF" -type f | sort | xargs -r sha256sum | sha256sum; }
@test "plan names the plugin and both config files, and prints no key" {
run prov plan
[ "$status" -eq 0 ]
printf '%s\n' "${lines[@]}" | grep -q '^+ install thetvdb_24\.0\.0\.0'
printf '%s\n' "${lines[@]}" | grep -q '^+ set TmdbApiKey'
printf '%s\n' "${lines[@]}" | grep -q '^+ create Jellyfin\.Plugin\.Tvdb\.xml'
# The whole point of running this over a shoulder: a key never appears.
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
}
@test "plan writes nothing" {
before="$(tree_fingerprint)"
run prov plan
[ "$status" -eq 0 ]
[ "$(tree_fingerprint)" = "$before" ]
}
@test "an already-configured endpoint reports no pending action" {
prov install >/dev/null
run prov plan
[ "$status" -eq 0 ]
! printf '%s\n' "${lines[@]}" | grep -q '^+ '
printf '%s\n' "${lines[@]}" | grep -q '= thetvdb_24\.0\.0\.0 already installed'
printf '%s\n' "${lines[@]}" | grep -q '= TmdbApiKey already set'
printf '%s\n' "${lines[@]}" | grep -q '= Jellyfin\.Plugin\.Tvdb\.xml already has an ApiKey'
}
@test "a keys file that is not mode 600 is refused and nothing is written" {
chmod 644 "$KEYS"
before="$(tree_fingerprint)"
run prov install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qF '600'
[ "$(tree_fingerprint)" = "$before" ]
}
@test "a missing TVDB key is refused by name before anything is fetched" {
printf 'TMDB_API_KEY=%s\n' "$TMDB_KEY" >"$KEYS"
chmod 600 "$KEYS"
run prov install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qF 'TVDB_API_KEY'
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
}
@test "a root with no plugin directory is refused instead of fabricated" {
# Creating plugins/ inside a wrong path would install a plugin Jellyfin
# never loads, and would hide the operator's typo.
run env "${PROV_ENV[@]}" JF_ROOT="$BATS_TEST_TMPDIR/nope" bash "$PROV" install
[ "$status" -ne 0 ]
[ ! -d "$BATS_TEST_TMPDIR/nope/plugins" ]
}
@test "an artifact whose sha256 is not the pinned one is refused" {
run env "${PROV_ENV[@]}" TVDB_PLUGIN_SHA256="$(printf 'f%.0s' {1..64})" \
bash "$PROV" install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qiF 'sha256'
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
}
@test "an artifact that is not the official TVDB plugin is refused" {
# A matching sha proves only that the bytes are the bytes we fetched.
EVIL="$BATS_TEST_TMPDIR/evil.zip"
EVIL_SHA="$(make_plugin "$EVIL" "deadbeef-0000-0000-0000-000000000000" 24.0.0.0)"
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$EVIL" TVDB_PLUGIN_SHA256="$EVIL_SHA" \
bash "$PROV" install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qiF 'guid'
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
}
@test "an artifact that is not the pinned version is refused" {
OLD="$BATS_TEST_TMPDIR/old.zip"
OLD_SHA="$(make_plugin "$OLD" "$GUID_OFFICIAL" 22.0.0.0)"
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$OLD" TVDB_PLUGIN_SHA256="$OLD_SHA" \
bash "$PROV" install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qiF 'version'
[ ! -d "$JF/plugins/thetvdb_24.0.0.0" ]
}
@test "a different plugin version already in plugins/ is refused, not doubled" {
# Two thetvdb_* directories means Jellyfin loads one plugin twice; the
# operator removes the old one (into the trash) deliberately, not by an
# installer quietly adding a second.
prov install >/dev/null
OLD="$BATS_TEST_TMPDIR/old.zip"
OLD_SHA="$(make_plugin "$OLD" "$GUID_OFFICIAL" 22.0.0.0)"
run env "${PROV_ENV[@]}" TVDB_PLUGIN_URL="file://$OLD" TVDB_PLUGIN_SHA256="$OLD_SHA" \
TVDB_PLUGIN_VERSION=22.0.0.0 bash "$PROV" install
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qF 'thetvdb_24.0.0.0'
[ ! -d "$JF/plugins/thetvdb_22.0.0.0" ]
}
@test "install places the plugin, keys TMDb, and preserves every other setting" {
run prov install
[ "$status" -eq 0 ]
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
[ -f "$JF/plugins/thetvdb_24.0.0.0/meta.json" ]
[ -f "$JF/plugins/thetvdb_24.0.0.0/Jellyfin.Plugin.Tvdb.dll" ]
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tmdb.xml"
# Jellyfin owns this file: the settings it wrote must survive the edit.
for setting in MetadataCountryCode Language IncludeAdult MaxCastMembers AddGenresFromKeywords; do
grep -qF "<$setting>" "$cfg" || {
echo "lost $setting"
return 1
}
done
grep -qF '<IncludeAdult>false</IncludeAdult>' "$cfg"
grep -qF "<TmdbApiKey>$TMDB_KEY</TmdbApiKey>" "$cfg"
grep -qF "<ApiKey>$TVDB_KEY</ApiKey>" "$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
}
@test "a TVDB config the operator saved through the UI is kept" {
cat >"$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml" <<XML
<?xml version="1.0" encoding="utf-8"?>
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ApiKey>ui-saved-key</ApiKey>
<EnableSeasonPathMatching>true</EnableSeasonPathMatching>
</Configuration>
XML
run prov install
[ "$status" -eq 0 ]
printf '%s\n' "${lines[@]}" | grep -qiF 'kept'
local cfg="$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
grep -qF '<ApiKey>ui-saved-key</ApiKey>' "$cfg"
grep -qF '<EnableSeasonPathMatching>true</EnableSeasonPathMatching>' "$cfg"
}
@test "a pin in the TVDB secret lands in the plugin config" {
# A subscriber-shaped key needs the account's PIN beside it; a licensed key
# does not, so the element appears only when the secret carries one.
printf 'TMDB_API_KEY=%s\nTVDB_API_KEY=%s\nTVDB_PIN=123456\n' "$TMDB_KEY" "$TVDB_KEY" >"$KEYS"
chmod 600 "$KEYS"
prov install >/dev/null
grep -qF '<Pin>123456</Pin>' "$JF/plugins/configurations/Jellyfin.Plugin.Tvdb.xml"
}
@test "install is idempotent: a second run changes no byte" {
prov install >/dev/null
before="$(tree_fingerprint)"
run prov install
[ "$status" -eq 0 ]
[ "$(tree_fingerprint)" = "$before" ]
printf '%s\n' "${lines[@]}" | grep -qiF 'no change'
# One plugin directory only: two versions in plugins/ means two loads.
[ "$(find "$JF/plugins" -maxdepth 1 -type d -name 'thetvdb_*' | wc -l)" -eq 1 ]
}
@test "what install replaces is copied to the trash first" {
prov install >/dev/null
# The TMDb config existed before this tool touched it, so the pre-edit copy
# must stay recoverable.
[ -n "$(find "$TRASH" -name 'Jellyfin.Plugin.Tmdb.xml*' -print -quit)" ]
grep -qF '<TmdbApiKey />' "$(find "$TRASH" -name 'Jellyfin.Plugin.Tmdb.xml*' | head -1)"
}
@test "verify refuses an unconfigured endpoint and names what is missing" {
run prov verify
[ "$status" -ne 0 ]
printf '%s\n' "${lines[@]}" | grep -qiF 'thetvdb'
printf '%s\n' "${lines[@]}" | grep -qiF 'TmdbApiKey'
}
@test "verify passes once the plugin and both keys are in place" {
prov install >/dev/null
run prov verify
[ "$status" -eq 0 ]
! printf '%s\n' "${lines[@]}" | grep -qF "$TMDB_KEY"
! printf '%s\n' "${lines[@]}" | grep -qF "$TVDB_KEY"
# Length, not bytes: the operator sees the key loaded without reading it.
printf '%s\n' "${lines[@]}" | grep -qE 'TmdbApiKey .*[0-9]+ chars'
}
@test "no plugin configuration XML is tracked in this repo" {
# These files hold provider keys and are owned by the running container;
# tracking them would put credentials in git and guarantee drift.
run git -C "$REPO_ROOT" ls-files
[ "$status" -eq 0 ]
! printf '%s\n' "${lines[@]}" | grep -qF 'plugins/configurations'
! printf '%s\n' "${lines[@]}" | grep -qiF 'Plugin.Tvdb.xml'
}
@test "the planner routes this installer to jellyfin" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
--paths scripts/jellyfin-providers.sh
[ "$status" -eq 0 ]
[ "$output" = "jellyfin" ]
}
@test "the deployer places the keys before it restarts jellyfin" {
# The plugin and both keys are read at start-up, so a restart that runs
# ahead of the placement boots the old, keyless state.
body="$(sed -n '/^deploy_jellyfin()/,/^\}/p' "$REPO_ROOT/scripts/deploy-remote.sh")"
[ -n "$body" ]
up="$(printf '%s\n' "$body" | grep -n 'ensure_provider_keys' | head -1 | cut -d: -f1)"
restart="$(printf '%s\n' "$body" | grep -n 'systemctl --user restart jellyfin.service' | head -1 | cut -d: -f1)"
[ "${up:-999}" -lt "${restart:-0}" ]
}
+438
View File
@@ -0,0 +1,438 @@
#!/usr/bin/env bats
# Seam: the audiobook-library relayout — scripts/media-census.py audiobooks (the
# export of the rows Plex matched) and scripts/media-audiobooks.py (the plan).
#
# The audiobook tree is the one library on this box that Jellyfin does not read
# at all and Plex reads completely: every chapter here is a Plex Track row under
# an Album row (the BOOK) under one Artist row (the NARRATOR). That is the
# authority this seam files by — never the collection folder a release pack made.
#
# Two laws here are the opposite of the music tool's, and both come from what the
# rows actually contain. Files are NEVER renumbered: 7,556 of 10,485 books carry
# at least one chapter whose index Plex does not know, so a `00 -` invented from
# a missing index would order a book wrong, and playback order comes from the
# embedded track tag either way — the name buys nothing and can only lie. And the
# folder artist is the NARRATOR even though a reader thinks in authors: every
# book has exactly one artist row, while the author survives only in 10% of the
# album titles (`Kryptos by Graham McNeill`) and in no column Plex stores, so one
# rule beats a shelf that is half authors and half narrators.
#
# Every test states which authority a file has and asserts where the PLAN files
# it, never how the planner decided.
load helpers
PLAN="$REPO_ROOT/scripts/media-audiobooks.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# row FILE NARRATOR BOOK YEAR TRACK CHAPTER
# One row of the authority export scripts/media-census.py audiobooks writes.
row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
"$1" "$2" "$3" "${4:-}" "${5:-}" "$6" >> "$ROWS"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
dst() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
}
# kind SRC — the kind the plan gave SRC's move row.
kind() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $4}' "$OUT"
}
setup() {
AB="$BATS_TEST_TMPDIR/Audiobooks"
PARK="$AB/_unsorted"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
# The header is part of the authority export, so the fixture carries it.
printf 'file\tnarrator\tbook\tyear\ttrack\tchapter\n' > "$ROWS"
# One book of one author, split across the two part folders a pack shipped it
# in, with a cover above part 1 and a playlist beside the chapters.
A="$AB/Stephen King/It (1986)"
at "$A [Part 1]/01 - Chapter 1.mp3" 9000
at "$A [Part 1]/02 - Chapter 2.mp3" 9000
at "$A [Part 1]/cover.jpg" 20
at "$A [Part 1]/It.m3u" 30
at "$A [Part 2]/03 - Chapter 3.mp3" 9000
row "$A [Part 1]/01 - Chapter 1.mp3" "George Guidall" "It" 1986 1 "Chapter 1"
row "$A [Part 1]/02 - Chapter 2.mp3" "George Guidall" "It" 1986 2 "Chapter 2"
row "$A [Part 2]/03 - Chapter 3.mp3" "George Guidall" "It" 1986 3 "Chapter 3"
# A book whose album row carries no year at all.
B="$AB/Collected/Project Gold"
at "$B/01.ogg" 9000
row "$B/01.ogg" "Operator X" "Project Gold" "" 1 "Opening"
# One book title, two narrators: two different books.
C="$AB/Shared"
at "$C/Dune side a.m4b" 9000
at "$C/Dune side b.m4b" 9000
at "$C/Dune 2020.m4b" 9000
row "$C/Dune side a.m4b" "Scott Brick" "Dune" 1965 1 "Prologue"
row "$C/Dune side b.m4b" "Scott Brick" "Dune" 1965 2 "Dune"
row "$C/Dune 2020.m4b" "Malcolm Hillgartner" "Dune" 2020 1 "Book One"
# Two part folders of one book whose chapters share a file name.
D="$AB/Pulp"
at "$D/Book A/vol1.mp3" 9000
at "$D/Book A vol2/vol1.mp3" 9000
row "$D/Book A/vol1.mp3" "Nora Pace" "Hard Food" 2001 1 "Vol 1"
row "$D/Book A vol2/vol1.mp3" "Nora Pace" "Hard Food" 2001 2 "Vol 1"
# The same narrator spelled two ways across one book's chapters.
E="$AB/Spell"
at "$E/kharn 1.mp3" 9000
at "$E/kharn 2.mp3" 9000
at "$E/kharn 3.mp3" 9000
row "$E/kharn 1.mp3" "David Timson" "Kharn" 2013 1 "One"
row "$E/kharn 2.mp3" "David timson" "Kharn" 2013 2 "Two"
row "$E/kharn 3.mp3" "David Timson" "Kharn" 2013 3 "Three"
# A folder holding two different books: its loose note has no owner.
F="$AB/Mixed"
at "$F/book one chapter.mp3" 9000
at "$F/book two chapter.mp3" 9000
at "$F/notes.txt" 10
row "$F/book one chapter.mp3" "Ann Reader" "Book One" 2001 1 "One"
row "$F/book two chapter.mp3" "Bob Reader" "Book Two" 2002 1 "Two"
# A file already parked: the pile is for a human, so a row that would file it
# must not pull it back into the tree.
at "$PARK/Old/nobody.mp3" 9000
row "$PARK/Old/nobody.mp3" "Ann Reader" "Book One" 2001 1 "One"
# The odd cases: no row, a video container, a name carrying a character Plex
# says breaks playback, rows with a blank book and a blank narrator, and a
# matched chapter whose file has no extension.
G="$AB/Odd"
at "$G/loose.mp3" 9000
at "$G/concert.mp4" 9000
at "$G/Bad: Name/01 chapter.mp3" 9000
row "$G/Bad: Name/01 chapter.mp3" "Some Reader" "Trail: Marked" 1999 1 "Marked"
at "$G/blank book.mp3" 9000
row "$G/blank book.mp3" "Some Reader" "" 2000 1 "No Book"
at "$G/blank narrator.mp3" 9000
row "$G/blank narrator.mp3" "" "No Artist" 2000 1 "No Artist"
at "$G/NoExt/part01" 9000
row "$G/NoExt/part01" "Archivist" "Tapes" 1971 1 "Tape One"
# Tag-scan placeholders. Two unrelated books whose album rows both read
# `[Unknown Album]`: the name is shared, so filing on it would merge books
# that have nothing to do with each other. A rip timestamp can be glued to
# the placeholder, and a multi-cast read says `Various Artists` — which IS a
# narrator.
H="$AB/Packs"
at "$H/Stross/Ship/c1.mp3" 9000
at "$H/Leckie/State/c1.mp3" 9000
at "$H/Rip/tape.mp3" 9000
at "$H/NoWho/Real Book.mp3" 9000
at "$H/Multicast/Genesis.mp3" 9000
row "$H/Stross/Ship/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
row "$H/Leckie/State/c1.mp3" "Various Artists" "[Unknown Album]" "" 1 "Part 1"
row "$H/Rip/tape.mp3" "Ann Reader" "Unknown album (03/03/2015 21:39:55)" "" 1 "One"
row "$H/NoWho/Real Book.mp3" "[Unknown Artist]" "A Named Book" 2001 1 "One"
row "$H/Multicast/Genesis.mp3" "Various Artists" "The Bible" "" 1 "Genesis"
at "$H/Numbered/01 - Grimoire.mp3" 9000
at "$H/Digits/1984.mp3" 9000
row "$H/Numbered/01 - Grimoire.mp3" "0.4" "Grimoire of the Lamb" "" 1 "One"
row "$H/Digits/1984.mp3" "Ann Reader" "1984" 1949 1 "One"
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
# path, no header.
python3 - "$AB" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
}
plan() {
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS" \
--out "$OUT" --exceptions "$EXC"
if [ "$status" -ne 0 ]; then
echo "planner said: $output"
return 1
fi
}
# db_with_part DB FILE NARRATOR BOOK YEAR TRACK CHAPTER — one Plex Track row
# under an Album under an Artist, in the three tables the export joins.
db_with_part() {
local db="$1" file="$2" narrator="$3" book="$4" year="$5" track="$6" chapter="$7"
local del="${8:-0}"
python3 - "$db" "$file" "$narrator" "$book" "$year" "$track" "$chapter" "$del" <<'PY'
import sqlite3, sys
db, file, narrator, book, year, track, chapter, deleted = sys.argv[1:9]
c = sqlite3.connect(db)
c.execute("create table if not exists media_parts (id integer primary key,"
" media_item_id, file, deleted_at)")
c.execute("create table if not exists media_items (id integer primary key,"
" metadata_item_id, deleted_at)")
c.execute("""create table if not exists metadata_items (id integer primary key,
parent_id, metadata_type, title, year, "index", deleted_at)""")
now = None if deleted == "0" else "2026-01-01"
ar = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (NULL, 8, ?, NULL, NULL, NULL)",
(narrator,)).lastrowid
al = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (?, 9, ?, ?, NULL, NULL)",
(ar, book, int(year) if year else None)).lastrowid
mi = c.execute("insert into metadata_items (parent_id, metadata_type, title, year,"
" \"index\", deleted_at) VALUES (?, 10, ?, NULL, ?, NULL)",
(al, chapter, int(track) if track else -1)).lastrowid
it = c.execute("insert into media_items (metadata_item_id, deleted_at) VALUES (?, NULL)",
(mi,)).lastrowid
c.execute("insert into media_parts (media_item_id, file, deleted_at) VALUES (?,?,?)",
(it, file, now))
c.commit()
PY
}
# ------------------------------------------------------------------ the export
@test "the census exports the narrator, book and chapter Plex matched for a part" {
DB="$BATS_TEST_TMPDIR/plex.db"
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01.mp3" \
"George Guidall" "It" 1986 1 "Chapter 1"
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file narrator book year track chapter" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Audiobooks/Pack/It part 1/01\.mp3 George Guidall It 1986 1 Chapter 1$"
}
@test "the census exports nothing outside the root, and nothing Plex deleted" {
DB="$BATS_TEST_TMPDIR/plex.db"
db_with_part "$DB" "/mnt/other/Library/01.mp3" "Someone" "Elsewhere" 1990 1 "One"
db_with_part "$DB" "$BATS_TEST_TMPDIR/Audiobooks/Gone/01.mp3" "Someone" \
"Deleted" 1990 1 "One" 1
run python3 "$CENSUS" audiobooks --root "$BATS_TEST_TMPDIR/Audiobooks" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
[ "$(wc -l < "$BATS_TEST_TMPDIR/exp.tsv")" -eq 1 ] # header only
}
# -------------------------------------------------------------------- the plan
@test "a matched chapter files under its narrator and its book, name untouched" {
plan
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
}
@test "one book arrives in one folder however many part folders it shipped in" {
plan
[ "$(dst "$A [Part 2]/03 - Chapter 3.mp3")" = \
"$AB/George Guidall/It (1986)/03 - Chapter 3.mp3" ]
[ "$(dst "$A [Part 1]/02 - Chapter 2.mp3")" = \
"$AB/George Guidall/It (1986)/02 - Chapter 2.mp3" ]
}
@test "a chapter is never renumbered, whether or not Plex knows its index" {
# The law this guards: playback order comes from the embedded track tag, and
# three quarters of the books have chapters with no index at all, so a
# number written into a name would be invented and would lie.
plan
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
[ "$(dst "$A [Part 1]/01 - Chapter 1.mp3")" = \
"$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" ]
}
@test "a book whose album row has no year gets no parentheses" {
plan
[ "$(dst "$B/01.ogg")" = "$AB/Operator X/Project Gold/01.ogg" ]
}
@test "two chapters claiming one name inside one book are named apart" {
plan
first="$(dst "$D/Book A/vol1.mp3")"
second="$(dst "$D/Book A vol2/vol1.mp3")"
[ "$first" != "$second" ]
[ -n "$first" ] && [ -n "$second" ]
# and the plan still passes the executor's preflight, which refuses a target
# two sources claim.
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
[ "$status" -eq 0 ]
[[ "$output" != *"double-claim"* ]]
}
@test "a cover follows the book whose folder it lies in" {
plan
[ "$(dst "$A [Part 1]/cover.jpg")" = "$AB/George Guidall/It (1986)/cover.jpg" ]
}
@test "a playlist moves with its book and its bytes are never rewritten" {
plan
[ "$(dst "$A [Part 1]/It.m3u")" = "$AB/George Guidall/It (1986)/It.m3u" ]
before=$(md5sum "$A [Part 1]/It.m3u" | cut -d" " -f1)
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" --journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
[ "$status" -eq 0 ]
[ "$(md5sum "$AB/George Guidall/It (1986)/It.m3u" | cut -d" " -f1)" = "$before" ]
# the stored paths inside it still name the old file names, which the tool
# leaves alone rather than editing a file the operator owns.
[ "$(kind "$A [Part 1]/It.m3u")" != "chapter" ]
}
@test "a note in a folder holding two books has no owner and parks" {
plan
[ "$(kind "$F/notes.txt")" = "park" ]
[ "$(dst "$F/notes.txt")" = "$PARK/Mixed/notes.txt" ]
}
@test "an audio file no row names parks with the path it has today" {
plan
[ "$(kind "$G/loose.mp3")" = "park" ]
[ "$(dst "$G/loose.mp3")" = "$PARK/Odd/loose.mp3" ]
}
@test "a video container parks instead of being renamed into audio" {
plan
[ "$(kind "$G/concert.mp4")" = "park" ]
run grep -c "concert\.mp4" "$EXC"
[ "$output" != "0" ]
}
@test "a character Plex says breaks playback never reaches a destination" {
plan
[ "$(dst "$G/Bad: Name/01 chapter.mp3")" = \
"$AB/Some Reader/Trail Marked (1999)/01 chapter.mp3" ]
run cut -f3 "$OUT"
[[ "$output" != *":"* ]]
}
@test "one book title with two narrators stays two books" {
plan
[ "$(dst "$C/Dune side a.m4b")" = "$AB/Scott Brick/Dune (1965)/Dune side a.m4b" ]
[ "$(dst "$C/Dune 2020.m4b")" = \
"$AB/Malcolm Hillgartner/Dune (2020)/Dune 2020.m4b" ]
}
@test "the spelling most chapters carry names the folder, and they share one folder" {
plan
[ "$(dst "$E/kharn 1.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 1.mp3" ]
[ "$(dst "$E/kharn 2.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 2.mp3" ]
[ "$(dst "$E/kharn 3.mp3")" = "$AB/David Timson/Kharn (2013)/kharn 3.mp3" ]
}
@test "a row with no book title parks rather than inventing a folder" {
plan
[ "$(kind "$G/blank book.mp3")" = "park" ]
}
@test "a row with no narrator parks rather than inventing a shelf" {
plan
[ "$(kind "$G/blank narrator.mp3")" = "park" ]
}
@test "books that share a placeholder album name never share a shelf" {
plan
[ "$(dst "$H/Stross/Ship/c1.mp3")" = "$PARK/Packs/Stross/Ship/c1.mp3" ]
[ "$(dst "$H/Leckie/State/c1.mp3")" = "$PARK/Packs/Leckie/State/c1.mp3" ]
}
@test "a placeholder with a rip timestamp glued to it is still a placeholder" {
plan
[ "$(kind "$H/Rip/tape.mp3")" = "park" ]
}
@test "a narrator placeholder parks even when the book is named" {
plan
[ "$(kind "$H/NoWho/Real Book.mp3")" = "park" ]
}
@test "Various Artists is a multi-cast narrator and files its book" {
plan
[ "$(dst "$H/Multicast/Genesis.mp3")" = \
"$AB/Various Artists/The Bible/Genesis.mp3" ]
}
@test "a chapter number mistagged as the narrator parks" {
plan
[ "$(kind "$H/Numbered/01 - Grimoire.mp3")" = "park" ]
}
@test "a book whose title is digits still files, because only a name must have a letter" {
plan
[ "$(dst "$H/Digits/1984.mp3")" = "$AB/Ann Reader/1984 (1949)/1984.mp3" ]
}
@test "the row, not the extension, decides what a chapter is" {
plan
[ "$(kind "$G/NoExt/part01")" = "chapter" ]
[ "$(dst "$G/NoExt/part01")" = "$AB/Archivist/Tapes (1971)/part01" ]
}
@test "the plan replays through the executor with nothing refused" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
[ "$status" -eq 0 ]
[[ "$output" != *"refusing"* ]]
# The refusal this guards is two sources claiming one target: the executor
# stops the whole run on it, so the plan must never hand a path out twice.
[ -z "$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)" ]
}
@test "an occupied destination is refused rather than overwritten" {
plan
mkdir -p "$AB/George Guidall/It (1986)"
at "$AB/George Guidall/It (1986)/01 - Chapter 1.mp3" 1
run python3 "$APPLY" apply --plan "$OUT" --root "$AB"
[ "$status" -ne 0 ]
[[ "$output" == *"occupied"* ]]
[ -e "$A [Part 1]/01 - Chapter 1.mp3" ]
}
@test "applying the plan and re-planning moves nothing" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$AB" \
--journal "$BATS_TEST_TMPDIR/j.jsonl" --apply
[ "$status" -eq 0 ]
python3 - "$AB" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
# The rows still name the paths the chapters were placed at: that is what a
# server re-scan of the moved tree reports.
python3 - "$OUT" "$ROWS" <<'PY'
import csv, sys
out, rows = sys.argv[1], sys.argv[2]
moved = {r[1]: r[2] for r in csv.reader(open(out), delimiter="\t") if r[0] == "move"}
with open(rows) as fh, open(rows + ".2", "w") as fh2:
fh2.write(fh.readline().rstrip("\n") + "\n")
for line in fh:
f = line.split("\t")[0]
fh2.write("\t".join([moved.get(f, f)] + line.rstrip("\n").split("\t")[1:]) + "\n")
PY
run python3 "$PLAN" --root "$AB" --enum "$ENUM" --rows "$ROWS.2" \
--out "$OUT.2" --exceptions "$EXC.2"
[ "$status" -eq 0 ]
[ "$(awk -F '\t' '$1 == "move"' "$OUT.2" | wc -l)" -eq 0 ]
}
@test "nothing under the park root is planned again" {
plan
run awk -F '\t' '$1 == "move" && $2 ~ /_unsorted/' "$OUT"
[ -z "$output" ]
}
+373
View File
@@ -0,0 +1,373 @@
#!/usr/bin/env bats
# Seam: the music-library relayout — scripts/media-census.py music (the tag and
# provider authority export) and scripts/media-music.py (the plan).
#
# Jellyfin's music rule is the opposite of the movie and TV ones: the embedded
# tags are the metadata and the file name is only a fallback title, so the
# authority here is the Audio row Jellyfin already matched (its album-artist,
# album, year, disc and track columns) and NOT the path a ripper chose. Every
# test states which authority a file has — row only, row with no album item,
# sidecar of a file that has a row, nothing at all — and asserts where the PLAN
# files it, never how the planner decided.
#
# The three outcomes that cost a re-run of a 90k-file plan: a track filed under
# the wrong artist (it splits one album across two artists), a destination that
# two tracks claim (the executor refuses, and the operator loses the run), and a
# lyric file that stops matching the basename of the track it belongs to, which
# silently loses the lyrics. Everything below is written to catch those, plus the
# gate that matters most: the plan must replay through scripts/media-apply.py,
# whose preflight is the fleet's only overwrite guard.
load helpers
PLAN="$REPO_ROOT/scripts/media-music.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# row FILE ARTIST ALBUM YEAR DISC TRACK TITLE [MB] [PERFORMER]
# One row of the authority export scripts/media-census.py music writes.
row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$1" "$2" "$3" "${4:-}" "${5:-}" "${6:-}" "$7" "${8:-}" "${9:-}" >> "$ROWS"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
dst() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
}
setup() {
MUSIC="$BATS_TEST_TMPDIR/Music"
PARK="$MUSIC/_unsorted"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
ROWS="$BATS_TEST_TMPDIR/rows.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
# A discography pack: the artist's own albums buried under "01. Albums",
# with a lyric, a cover, an EAC log and a playlist beside the tracks.
A="$MUSIC/Artist One - Discography [FLAC]/01. Albums/Album X"
at "$A/01. First.flac" 9000
at "$A/02. Second.flac" 9000
at "$A/01. First.lrc" 10
at "$A/cover.jpg" 20
at "$A/rip.log" 10
at "$A/Album X.m3u" 10
# A multi-disc compilation: one album (the album artist is the compilation
# tag), two discs, and a different performer on each track.
B="$MUSIC/Box (2019)"
at "$B/Disc 1/01 Track A.flac" 9000
at "$B/Disc 2/01 Track B.flac" 9000
# The box cover lies above the two disc folders: the discs below it own it.
at "$B/box.jpg" 10
# One image plus its cuesheet: a whole album in one file.
C="$MUSIC/Image Cue"
at "$C/Image.flac" 9000
at "$C/Image.cue" 10
# The hard cases: no row at all, a video container, two tracks claiming one
# number and one title, a track with no number, and a track whose album
# artist tag is empty so only the performer names it.
D="$MUSIC/Odd"
at "$D/loose.flac" 9000
at "$D/concert.mkv" 9000
at "$D/dup1.flac" 9000
at "$D/dup2.flac" 9000
at "$D/unnumbered.flac" 9000
at "$D/lone.flac" 9000
# A sidecar in a folder that holds two different albums.
E="$MUSIC/Mixed"
at "$E/one.flac" 9000
at "$E/two.flac" 9000
at "$E/notes.pdf" 10
# An album title carrying a character Jellyfin says breaks playback.
F="$MUSIC/Question Mark"
at "$F/01. Bang.flac" 9000
# An album whose tags name no album artist at all, and two different
# performers; and one album title carried by two different album artists.
G="$MUSIC/Tagless"
at "$G/one.flac" 9000
at "$G/two.flac" 9000
H="$MUSIC/Same Title"
at "$H/a.flac" 9000
at "$H/b.flac" 9000
# One album whose tracks disagree on the spelling of its name; the minority
# spelling sorts first, so it must not win the folder.
I="$MUSIC/Puppets"
at "$I/one.flac" 9000
at "$I/two.flac" 9000
at "$I/three.flac" 9000
# The authority export carries a header; the planner reads it by column name.
printf 'file\tartist\talbum\tyear\tdisc\ttrack\ttitle\tmb\tperformer\n' > "$ROWS"
row "$A/01. First.flac" "Artist One" "Album X" 2015 1 1 "First Song"
row "$A/02. Second.flac" "Artist One" "Album X" 2015 1 2 "Second Song: Reprise"
row "$B/Disc 1/01 Track A.flac" "Various Artists" "Best Of" 2019 1 1 "Track A" \
"" "Aaa Singer"
row "$B/Disc 2/01 Track B.flac" "Various Artists" "Best Of" 2019 2 1 "Track B" \
"" "Bbb Singer"
row "$C/Image.flac" "Solo" "Single Album" 2001 1 1 "Single Album"
row "$D/dup1.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
row "$D/dup2.flac" "Dup Artist" "Dupes" "" "" 1 "Same"
row "$D/unnumbered.flac" "No Number" "No Number Album" "" "" "" "Untitled"
row "$D/lone.flac" "" "Session Works" "" "" 1 "Lone" "" "Session Singer"
row "$E/one.flac" "Mix Artist" "First Set" 2000 "" 1 "One"
row "$E/two.flac" "Mix Artist" "Second Set" 2001 "" 1 "Two"
row "$F/01. Bang.flac" "Billie" "WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?" \
2019 "" 1 "!!!!!!"
row "$G/one.flac" "" "Two Seasons" 2005 "" 1 "One" "" "Alpha Band"
row "$G/two.flac" "" "Two Seasons" 2005 "" 2 "Two" "" "Beta Band"
row "$H/a.flac" "Aaa Singer" "Same Title" 2011 "" 1 "A Song"
row "$H/b.flac" "Bbb Singer" "Same Title" 2011 "" 1 "B Song"
row "$I/one.flac" "Metallica" "Master Of Puppets" 2015 "" 1 "One"
row "$I/two.flac" "Metallica" "Master of Puppets" 2015 "" 2 "Two"
row "$I/three.flac" "Metallica" "Master of Puppets" 2015 "" 3 "Three"
# The enum TSV is what scripts/media-census.py enumerate writes: bytes, ext,
# path, no header.
python3 - "$MUSIC" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for dirpath, _dirs, files in os.walk(root):
for fn in files:
p = os.path.join(dirpath, fn)
fh.write(f"{os.path.getsize(p)}\t{os.path.splitext(fn)[1].lower()}\t{p}\n")
PY
}
plan() {
run python3 "$PLAN" --root "$MUSIC" --enum "$ENUM" --rows "$ROWS" \
--out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ] || echo "planner said: $output"
}
@test "the census exports the tag and provider row Jellyfin matched for a track" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" <<'PY'
import sqlite3, sys, uuid
c = sqlite3.connect(sys.argv[1])
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, Album,
AlbumArtists, Artists, Data)""")
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
AU = "MediaBrowser.Controller.Entities.Audio.Audio"
AL = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
aid, trk = str(uuid.uuid4()), "/data/Music/Pack/Album X/01. First.flac"
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(aid, None, AL, "Album X", "/data/Music/Pack/Album X", None, None, 2015,
None, "Artist One", None, '{}'))
c.execute("insert into BaseItemProviders values (?,?,?)",
(aid, "MusicBrainzReleaseGroup", "rg-1234"))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), aid, AU, "First Song", trk, 1, 1, 2015,
"album x (the tag)", "Artist One", "Artist One", '{}'))
c.commit()
PY
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file artist album year disc track title mb performer" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/Album X/01\. First\.flac Artist One Album X 2015 1 1 First Song rg-1234 Artist One$"
}
@test "a track whose folder is no album item still exports from its own tags" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" <<'PY'
import sqlite3, sys, uuid
c = sqlite3.connect(sys.argv[1])
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, Album,
AlbumArtists, Artists, Data)""")
c.execute("""create table BaseItemProviders (ItemId, ProviderId, ProviderValue)""")
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), None, "MediaBrowser.Controller.Entities.Audio.Audio",
"Lone", "/data/Music/Pack/lone.flac", 1, None, None,
"Session Works", None, "Session Singer", '{}'))
c.commit()
PY
run python3 "$CENSUS" music --root "$BATS_TEST_TMPDIR/Music" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$BATS_TEST_TMPDIR/Music/Pack/lone\.flac Session Singer Session Works 1 Lone Session Singer$"
}
@test "a tagged track files under artist, album and year with a numbered title" {
plan
[ "$status" -eq 0 ]
[ "$(dst "$A/01. First.flac")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.flac" ]
}
@test "the tag title wins over the name the ripper gave the file" {
plan
[ "$(dst "$A/02. Second.flac")" = "$MUSIC/Artist One/Album X (2015)/02 - Second Song Reprise.flac" ]
}
@test "a character Jellyfin says breaks playback never reaches a destination" {
plan
[ "$(dst "$F/01. Bang.flac")" = "$MUSIC/Billie/WHEN WE ALL FALL ASLEEP, WHERE DO WE GO (2019)/01 - !!!!!!.flac" ]
run awk -F '\t' '$1 == "move" && $3 ~ /[<>:"|?*]/ {print $3}' "$OUT"
[ -z "$output" ]
}
@test "every disc of a multi-disc album lands in one folder, numbered apart" {
plan
[ "$(dst "$B/Disc 1/01 Track A.flac")" = "$MUSIC/Various Artists/Best Of (2019)/1.01 - Track A.flac" ]
[ "$(dst "$B/Disc 2/01 Track B.flac")" = "$MUSIC/Various Artists/Best Of (2019)/2.01 - Track B.flac" ]
}
@test "an album whose tracks name several artists files under Various Artists" {
plan
run awk -F '\t' '$1 == "move" && $3 ~ /\/Best Of \(2019\)\/.*\.flac$/ {print $3}' "$OUT"
[ "${#lines[@]}" -eq 2 ]
[[ "${lines[0]}" == "$MUSIC/Various Artists/"* ]]
}
@test "an album with no album artist and two performers is a compilation" {
plan
[ "$(dst "$G/one.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/01 - One.flac" ]
[ "$(dst "$G/two.flac")" = "$MUSIC/Various Artists/Two Seasons (2005)/02 - Two.flac" ]
}
@test "the same album title under two album artists is two albums" {
plan
[ "$(dst "$H/a.flac")" = "$MUSIC/Aaa Singer/Same Title (2011)/01 - A Song.flac" ]
[ "$(dst "$H/b.flac")" = "$MUSIC/Bbb Singer/Same Title (2011)/01 - B Song.flac" ]
}
@test "the folder carries the album spelling most tracks carry" {
plan
[ "$(dst "$I/one.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/01 - One.flac" ]
[ "$(dst "$I/two.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/02 - Two.flac" ]
[ "$(dst "$I/three.flac")" = "$MUSIC/Metallica/Master of Puppets (2015)/03 - Three.flac" ]
}
@test "a lyric file is renamed to match the track it belongs to" {
plan
[ "$(dst "$A/01. First.lrc")" = "$MUSIC/Artist One/Album X (2015)/01 - First Song.lrc" ]
}
@test "the album image, the rip log and the playlist follow the album folder" {
plan
[ "$(dst "$A/cover.jpg")" = "$MUSIC/Artist One/Album X (2015)/cover.jpg" ]
[ "$(dst "$A/rip.log")" = "$MUSIC/Artist One/Album X (2015)/rip.log" ]
[ "$(dst "$A/Album X.m3u")" = "$MUSIC/Artist One/Album X (2015)/Album X.m3u" ]
}
@test "a cover above the disc folders of one album follows that album" {
plan
[ "$(dst "$B/box.jpg")" = "$MUSIC/Various Artists/Best Of (2019)/box.jpg" ]
}
@test "an image carrying a whole album keeps its own name beside its cuesheet" {
plan
[ "$(dst "$C/Image.flac")" = "$MUSIC/Solo/Single Album (2001)/Image.flac" ]
[ "$(dst "$C/Image.cue")" = "$MUSIC/Solo/Single Album (2001)/Image.cue" ]
}
@test "an audio file no provider row names is parked, not filed under an artist" {
plan
[[ "$(dst "$D/loose.flac")" == "$PARK/Odd/loose.flac" ]]
}
@test "a video container in the music library is parked, never renamed to an audio one" {
plan
[[ "$(dst "$D/concert.mkv")" == "$PARK/Odd/concert.mkv" ]]
run awk -F '\t' '$1 == "move" && $3 ~ /\.mka$|\.m4a$/ {print $3}' "$OUT"
[ -z "$output" ]
}
@test "two tracks claiming one number and one title get two destinations" {
plan
[ "$(dst "$D/dup1.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same.flac" ]
[ "$(dst "$D/dup2.flac")" = "$MUSIC/Dup Artist/Dupes/01 - Same (2).flac" ]
}
@test "an album with no year is named without one, a track with no number likewise" {
plan
[ "$(dst "$D/unnumbered.flac")" = "$MUSIC/No Number/No Number Album/Untitled.flac" ]
}
@test "a track naming only a performer files under that performer" {
plan
[ "$(dst "$D/lone.flac")" = "$MUSIC/Session Singer/Session Works/01 - Lone.flac" ]
}
@test "a sidecar in a folder holding two albums parks instead of guessing" {
plan
[[ "$(dst "$E/notes.pdf")" == "$PARK/Mixed/notes.pdf" ]]
}
@test "the plan creates every folder before the move that needs it" {
plan
python3 - "$OUT" "$MUSIC" <<'PY'
import os, sys
made = set()
for line in open(sys.argv[1]).read().splitlines()[1:]:
action, _src, dest, _kind = line.split("\t")
if action == "mkdir":
made.add(dest)
elif action == "move":
parent = os.path.dirname(dest)
while parent and parent != sys.argv[2]:
assert parent in made, f"{dest} needs {parent}, which is never made"
parent = os.path.dirname(parent)
PY
}
@test "no move lands outside the music library" {
plan
run awk -F '\t' -v l="$MUSIC" \
'$1 == "move" && $3 !~ "^" l "/" {print $3}' "$OUT"
[ -z "$output" ]
}
@test "the executor preflights the plan and no target is claimed twice" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$BATS_TEST_TMPDIR" \
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
[ "$status" -eq 0 ]
dup=$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)
[ -z "$dup" ]
}
@test "re-planning a library the plan already satisfies moves nothing" {
plan
# Applying the plan and re-censusing is what the operator does next: both
# authority files are keyed by path, so both are replayed through the plan
# the way a fresh census of the moved tree would see them. (The enum has no
# header; the rows export does, and its header row is not a path.)
python3 - "$OUT" "$ENUM" "$ROWS" <<'PY'
import os, sys
plan = sys.argv[1]
moves = {}
for line in open(plan).read().splitlines()[1:]:
action, src, dest, _kind = line.split("\t")
if action == "move":
moves[src] = dest
os.makedirs(os.path.dirname(dest), exist_ok=True)
os.rename(src, dest)
for path, col in ((sys.argv[2], 2), (sys.argv[3], 0)):
rows = [line.rstrip("\n").split("\t") for line in open(path)]
with open(path, "w") as fh:
for f in rows:
f[col] = moves.get(f[col], f[col])
fh.write("\t".join(f) + "\n")
PY
plan
run awk -F '\t' '$1 == "move" {print $2 " -> " $3}' "$OUT"
[ -z "$output" ] || { echo "moved again: $output"; false; }
}
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env bats
# Seam: the SSD-backed metadata view (OverlayFS) that lets Jellyfin write
# Kodi-format sidecars for Plex to read, without ever writing to the spindle
# array and without letting either media server start against a missing view.
#
# The contracts, in the order the seam tests them:
# 1. the plan puts the writable layer on the SSD and the media layer below it;
# 2. bad input is refused by name instead of mounting something surprising;
# 3. verify refuses a view that is not mounted (so nothing trusts a directory);
# 4. both media units bind the VIEW and refuse to start without it;
# 5. only those two units touch the view — the LLM and library tenants keep the
# raw volume, so the blast radius is two containers;
# 6. the deployer brings the view up BEFORE it restarts either server;
# 7. the planner routes a change to these files to both media services.
load helpers
VIEW="$REPO_ROOT/scripts/media-view.sh"
JELLY="$REPO_ROOT/dankserver/jellyfin/quadlet/jellyfin.container"
PLEX="$REPO_ROOT/dankserver/plex/quadlet/plex.container"
MOUNT=/var/lib/media-meta/view
UNIT="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
@test "the plan names an overlay with the spindle below and the SSD above" {
run bash "$VIEW" plan
[ "$status" -eq 0 ]
# Lower = the array; upper/work = the NVMe. A user asked for metadata to be
# SSD-resident, so a spindle upper layer would defeat the whole mechanism.
[ "${lines[0]}" = "lowerdir=/mnt/entropy" ]
printf '%s\n' "${lines[@]}" | grep -qx 'upperdir=/var/lib/media-meta/upper'
printf '%s\n' "${lines[@]}" | grep -qx 'workdir=/var/lib/media-meta/work'
printf '%s\n' "${lines[@]}" | grep -Fqx "mountpoint=$MOUNT"
# systemd names a mount unit after its escaped path, backslash and all.
printf '%s\n' "${lines[@]}" | grep -Fqx \
'unit=/etc/systemd/system/var-lib-media\x2dmeta-view.mount'
}
@test "install refuses a caller who cannot write the unit directory or mount" {
# Half an install is worse than none: the unit would exist without the
# directories behind it. Refuse by name before anything is created.
if [ "$(id -u)" -eq 0 ]; then
skip "this seam is running as root"
fi
run bash "$VIEW" install
[ "$status" -ne 0 ]
[[ "$output" == *"root"* ]]
}
@test "verify refuses a view that is not mounted" {
# Every consumer trusts the mount point to BE the library. On a host where it
# is a plain empty directory, verify must fail loudly: that is the difference
# between a scan that finds nothing and a scan that empties a library DB.
if mountpoint -q "$MOUNT"; then
skip "$MOUNT is a real mountpoint on this host"
fi
run bash "$VIEW" verify
[ "$status" -ne 0 ]
[[ "$output" == *"$MOUNT"* ]]
}
@test "both media servers bind the view and refuse to start without it" {
for unit in "$JELLY" "$PLEX"; do
grep -q "^Volume=$MOUNT:" "$unit" || return 1
grep -q "^UnitConfigItem=ConditionPathIsMountPoint=$MOUNT$" "$unit" || return 1
# The raw array must no longer be a container source: that bind is what
# made sidecar sharing impossible (read-only) and metadata slow (seeks).
! grep -q '^Volume=/mnt/entropy' "$unit"
done
# Jellyfin is the writer, so it needs the view writable; Plex only reads the
# sidecars, and keeping its bind read-only costs nothing.
grep -q "^Volume=$MOUNT:/data$" "$JELLY"
grep -q "^Volume=$MOUNT:/mnt/entropy:ro$" "$PLEX"
}
@test "only the two media units reference the view" {
run grep -rl --include='*.container' "$MOUNT" "$REPO_ROOT"/dankserver \
"$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro 2>/dev/null
[ "$status" -eq 0 ]
[ "$(printf '%s\n' "$output" | wc -l)" -eq 2 ]
printf '%s\n' "$output" | grep -q 'jellyfin/quadlet/jellyfin.container'
printf '%s\n' "$output" | grep -q 'plex/quadlet/plex.container'
}
@test "the mount unit is an overlay whose upper layer lives on the SSD" {
[ -f "$UNIT" ]
run grep -E '^Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work' "$UNIT"
[ "$status" -eq 0 ]
# The unit must mount exactly where the two containers bind, or the guard and
# the bind disagree and a media server browses an empty directory.
run grep -Fqx "Where=$MOUNT" "$UNIT"
[ "$status" -eq 0 ]
# It must wait for the array to be unlocked: mounting an overlay whose lower
# dir is an empty /mnt/entropy would present an empty library, not an error.
run grep -E '^RequiresMountsFor=/mnt/entropy$' "$UNIT"
[ "$status" -eq 0 ]
}
@test "the deployer raises the view before it restarts either media server" {
for svc in jellyfin plex; do
body="$(sed -n "/^deploy_${svc}()/,/^\}/p" "$REPO_ROOT/scripts/deploy-remote.sh")"
[ -n "$body" ]
up="$(printf '%s\n' "$body" | grep -n 'ensure_media_view' | head -1 | cut -d: -f1)"
up_line="${up:-999}"
restart="$(printf '%s\n' "$body" | grep -n "systemctl --user restart $svc.service" | head -1 | cut -d: -f1)"
restart_line="${restart:-0}"
[ "$up_line" -lt "$restart_line" ]
done
}
@test "the planner routes the view files to both media services" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
scripts/media-view.sh dankserver/host/systemd/media-meta-view.mount
[ "$status" -eq 0 ]
[ "$output" = "jellyfin
plex" ]
}
@test "a docs-only change still plans nothing for the view" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver --paths \
"serverAppConfigs-docs/Dankserver/Media Metadata View.md"
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "reap names the metadata whose media has moved and touches nothing" {
# A relayout moves media on the RAW path; the sidecars the view already wrote
# stay behind at the old coordinates. Left alone they become folders a media
# server browses and finds nothing in.
tmp="$(mktemp -d)"
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
"$tmp/meta/upper/TVShows/Big Little Lies/Season 01" "$tmp/meta/work" \
"$tmp/array/Movies/Keeper (2020)" "$tmp/array/TVShows"
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
: > "$tmp/meta/upper/TVShows/Big Little Lies/Season 01/show.nfo"
: > "$tmp/array/Movies/Keeper (2020)/Keeper (2020).mkv"
run bash "$VIEW" reap --upper-base "$tmp/meta" --lower "$tmp/array"
[ "$status" -eq 0 ]
printf '%s\n' "$output" | grep -qxF 'Movies/Ghost (2020)'
# The series folder is still below, so the ghost is the season inside it — and
# it is reported at the level that is actually missing, not deeper.
printf '%s\n' "$output" | grep -qxF 'TVShows/Big Little Lies'
! printf '%s\n' "$output" | grep -q 'Keeper'
[ -f "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo" ]
}
@test "reap --apply moves ghosts to the SSD trash and keeps live metadata" {
tmp="$(mktemp -d)"
mkdir -p "$tmp/meta/upper/Movies/Ghost (2020)" "$tmp/meta/upper/Movies/Keeper (2020)" \
"$tmp/array/Movies/Keeper (2020)"
: > "$tmp/meta/upper/Movies/Ghost (2020)/movie.nfo"
: > "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo"
run bash "$VIEW" reap --apply --upper-base "$tmp/meta" --lower "$tmp/array"
[ "$status" -eq 0 ]
[ ! -e "$tmp/meta/upper/Movies/Ghost (2020)" ]
[ -f "$tmp/meta/upper/Movies/Keeper (2020)/movie.nfo" ]
# Recoverable, not deleted: the item lands in a dated trash under the SSD base.
[ "$(find "$tmp/meta/.AGENTTRASH" -name movie.nfo | wc -l)" -eq 1 ]
}
+652
View File
@@ -0,0 +1,652 @@
#!/usr/bin/env bats
# Seam: the movie-library relayout — scripts/media-census.py (the read-only
# census), scripts/media-plan.py (the plan) and scripts/media-apply.py (the only
# thing that moves files).
#
# The library is 8k files on a 100 %-full array, so the two things this seam
# exists to catch are the ones that cost a re-run of a 20-minute census: a file
# that ends up in the WRONG movie folder, and a move that destroys something.
# Everything below runs on a synthetic library in the test's own tmpdir: no
# ffprobe, no network, no server. The probe step is the one part a fixture
# cannot run (it opens real containers), so the tests feed the planner the
# probed TSV in the column order scripts/media-census.py exports — which pins
# that contract as a side effect.
#
# The plan's own gate lines (unplanned, left-unmoved, double-claim, occupied)
# are asserted as output, because they are how the operator learns the plan is
# trustworthy before pointing the executor at 24 TB.
load helpers
PLAN="$REPO_ROOT/scripts/media-plan.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
TV="$REPO_ROOT/scripts/media-tv.py"
# probe_row FILE DUR [RES] [AUDIO] [SUBS] — one probed-TSV row: the twelve
# columns media-census.py probe writes (bytes, path, dur, vcodec, res, fps,
# vbit, pix, audio, subs, spare, fmt).
probe_row() {
local p="$1" dur="$2" res="${3:-1920x1080}" aud="${4:-aac+aac}" subs="${5:-0}"
printf '%s\t%s\t%s\thevc\t%s\t23.976\t5000000\t8bit\t%s\t%s\t0\tmatroska,webm\n' \
"$(stat -c %s "$p")" "$p" "$dur" "$res" "$aud" "$subs" >> "$PROBED"
}
# plex_row FILE TITLE YEAR — one Plex metadata row; the planner reads these
# three columns and treats a row without a plausible year as no match at all.
plex_row() {
printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$PLEX"
}
# at PATH SIZE — a file with deterministic content, big enough to matter.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
setup() {
LIB="$BATS_TEST_TMPDIR/Movies"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
PLEX="$BATS_TEST_TMPDIR/plex.tsv"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
TVLIB="$BATS_TEST_TMPDIR/TVShows"
TVPLAN="$BATS_TEST_TMPDIR/tv.tsv"
TVJ="$BATS_TEST_TMPDIR/tv.jsonl"
: > "$PROBED"
printf 'file\tleaf_title\tleaf_year\n' > "$PLEX"
# Two films that share a box set and a word, plus their bonus material:
# the shape that made one film's featurettes land in another's folder.
at "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4000
at "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 3300000
at "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 900000
at "$LIB/Alien (1979)/subs/3_Eng.srt" 40000
at "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 5000000
at "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 2400000
# A long extra: over 45 minutes, so only the folder may call it bonus.
at "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 2600000
# A misspelled bonus folder: no bonus vocabulary, so only the folder that
# holds the feature can decide that this is not a film.
at "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2500000
# A Plex title that carries "/" and a Plex row that was never matched.
at "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 4200000
at "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 3100000
# Junk the layout has to park somewhere other than a movie folder.
at "$LIB/.DS_Store" 6148
at "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv" 220
# A bonus file whose name is a number: never a movie folder named "1".
at "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 2700000
# A TV box set sitting in the movie library. This one names each file
# "Show - SxxEyy - Episode Title", so the episode title comes AFTER the
# marker; that convention is what made 124 episodes of one series look like
# 124 different films, each under its own threshold.
for e in 01 02 03 04 05; do
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
# The same trouble named the other way round, with no episode title at all.
for e in 01 02 03 04 05; do
at "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3000000
done
# A film whose own name carries a bare "Part 2": one episode-shaped token is
# not a series, and it must not be dragged into the TV pile.
at "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 9000000
probe_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4700
probe_row "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 2700
probe_row "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 30
probe_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 4800
probe_row "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 1200
probe_row "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 3000
probe_row "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2600
probe_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 5700
probe_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 6300
probe_row "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 5220
for e in 01 02 03 04 05; do
probe_row "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E$e - Mr Monk and the Rival.mkv" 2600
probe_row "$LIB/Big Little Lies Season 1/Big Little Lies S01E$e.mp4" 3300
done
probe_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" 6200
plex_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" "Alien" "1979"
plex_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" "Marnie" "1964"
plex_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" "V/H/S/2" "2013"
# An unmatched Plex row: the leaf title echoes the file and the year is 0.
plex_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" "AMERICAN POP" "0"
plex_row "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv" \
"The Godfather Part 2" "1974"
export MEDIA_ROOT="$LIB"
run python3 "$CENSUS" enumerate --root "$LIB" --out "$ENUM"
[ "$status" -eq 0 ]
}
# plan — run the planner over the fixture, keeping its plan TSV, its exception
# report and the gate lines it prints (that stdout is the operator's evidence
# that the plan accounts for every file, so the tests read it too).
plan() {
PLANOUT="$BATS_TEST_TMPDIR/plan.out"
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --probed "$PROBED" \
--plex "$PLEX" --out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$PLANOUT"
}
# dst_of SRC — the destination the plan gives one file.
dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$OUT" "$1"
}
# tv_plan — the TV mover over the fixture library; its stdout is the operator's
# census of what it called a series and what it refused to touch.
tv_plan() {
run python3 "$TV" --root "$LIB" --library "$TVLIB" --out "$TVPLAN"
[ "$status" -eq 0 ]
printf '%s\n' "$output" > "$BATS_TEST_TMPDIR/tv.out"
}
# tv_apply — that plan through the same executor. The destination is a sibling
# share, so the plan is applied at the root that holds both libraries.
tv_apply() {
run python3 "$APPLY" apply --plan "$TVPLAN" --root "$BATS_TEST_TMPDIR" \
--journal "$TVJ" --apply
[ "$status" -eq 0 ]
}
# tv_dst_of SRC — the destination the TV plan gives one file, empty if none.
tv_dst_of() {
python3 -c '
import sys
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[1] == sys.argv[2]:
print(f[2]); break
' "$TVPLAN" "$1"
}
@test "a film lands in Title (year)/Title (year).mkv, named by Plex" {
plan
[ "$(dst_of "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv")" \
= "$LIB/Alien (1979)/Alien (1979).mkv" ]
}
@test "a Plex title that carries a slash becomes one folder, not a nested tree" {
plan
local d; d="$(dst_of "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv")"
[ "$(dirname "$d")" = "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$LIB/V" ]
# the folder must be ONE mkdir: an un-sanitized name would make the
# executor build Movies/V/H/S/2 (2013) instead
run grep -F "mkdir" "$OUT"
[[ "$output" == *"$(printf '%s' "$LIB/V-H-S-2 (2013)")"* ]]
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != *"$LIB/V/"* ]]
}
@test "an unmatched Plex row never names a folder with a zero year" {
plan
run awk -F'\t' '{print $3}' "$OUT"
[[ "$output" != *"(0)"* ]]
# the filename's own year carries the folder, the folder and the file share
# the name (Jellyfin's rule), and the film sits at the library root rather
# than in an unsorted pile
local d; d="$(dst_of "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB" ]
[[ "$(basename "$(dirname "$d")")" == *" (2023)" ]]
[ "$(basename "$d")" = "$(basename "$(dirname "$d")").mkv" ]
}
@test "a bonus folder keeps its featurettes inside its own film, not a neighbour's" {
plan
local d
d="$(dst_of "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
d="$(dst_of "$LIB/Alien (1979)/Extras/Extra - Logo.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
# the feature-length extra is bonus too: it sits in a bonus folder and no
# metadata matched it, so it may not become a movie of its own
d="$(dst_of "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a folder inside a movie folder is bonus even when its name is misspelled" {
plan
local d; d="$(dst_of "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
}
@test "a subtitle follows the film of the folder it sits in" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/subs/3_Eng.srt")"
[ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
}
@test "macOS sidecars are parked in one pile, never inside a movie folder" {
plan
local d; d="$(dst_of "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
d="$(dst_of "$LIB/.DS_Store")"
[[ "$(dirname "$d")" == "$LIB"/_* ]]
grep -q "AppleDouble" "$EXC"
}
@test "a file whose name is a number becomes a review line, not a folder named 1" {
plan
run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
[[ "$output" != "$LIB/1"$'\n'* ]] && [[ "$output" != "$LIB/1" ]]
[[ "$output" != *"$LIB/1/"* ]]
grep -q "UNNAMED" "$EXC"
}
@test "a box set in the movie library is quarantined, not filed as movies" {
plan
local d
d="$(dst_of "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ monk* ]]
d="$(dst_of "$LIB/Big Little Lies Season 1/Big Little Lies S01E01.mp4")"
[[ "$d" == "$LIB"/_TV\ needs\ a\ TV\ library\ -\ big\ little\ lies* ]]
# one series is one pile, whatever its episodes are called
run python3 -c '
import re, sys
pile = {}
for line in open(sys.argv[1]):
f = line.rstrip("\n").split("\t")
if len(f) > 2 and f[0] == "move":
m = re.search(r"/_TV needs a TV library - ([^/]*)/", f[2] + "/")
if m:
pile[m.group(1)] = pile.get(m.group(1), 0) + 1
for k in sorted(pile):
print(f"{k}={pile[k]}")
' "$OUT"
[ "${#lines[@]}" -eq 2 ]
[[ "${lines[0]}" == big\ little\ lies*=5 ]]
[[ "${lines[1]}" == monk*=5 ]]
}
@test "a film named Part 2 is not called a series" {
plan
[ "$(dst_of "$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv")" \
= "$LIB/The Godfather Part 2 (1974)/The Godfather Part 2 (1974).mkv" ]
}
@test "the TV mover files a box set into the TV library in Jellyfin shape" {
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ ! -e "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv" ]
# films are not the TV mover's business
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "the TV mover leaves a film whose name only looks like an episode" {
tv_plan
tv_apply
local gf="$LIB/The Godfather Part 2 (1974)/The.Godfather.Part.2.1974.1080p.BluRay.mkv"
[ "$(tv_dst_of "$gf")" = "" ]
[ -e "$gf" ]
grep -q "Godfather.*not a series" "$BATS_TEST_TMPDIR/tv.out"
}
@test "the TV mover keeps an episode's own subtitle with the episode" {
at "$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.srt" 40000
tv_plan
tv_apply
[ -e "$TVLIB/Monk (2002)/Season 01/Monk - S01E01 - Mr Monk and the Rival.srt" ]
}
@test "a second TV move finds nothing left to move" {
tv_plan
tv_apply
tv_plan
grep -q "0 moves" "$BATS_TEST_TMPDIR/tv.out"
}
@test "a show whose own name carries the year is not given it twice" {
local d="$LIB/Leah Remini Scientology and the Aftermath (2016)/Season 1"
for e in 01 02 03 04 05; do
at "$d/Leah Remini Scientology and the Aftermath (2016) - S01E$e - Episode.mkv" 3000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016)/Season 01/Leah Remini Scientology and the Aftermath (2016) - S01E01 - Episode.mkv" ]
[ ! -e "$TVLIB/Leah Remini Scientology and the Aftermath (2016) (2016)" ]
}
@test "an all-lowercase series name is written the way the library writes names" {
for e in 01 02 03 04 05; do
at "$LIB/downton abbey 2010/Season 1/downton abbey - s01e$e - episode title.mkv" 3000000
done
tv_plan
tv_apply
# the FOLDER is titled; the file keeps the name the library already holds
[ -e "$TVLIB/Downton Abbey (2010)/Season 01/downton abbey - s01e01 - episode title.mkv" ]
}
@test "the mover leaves an episode the TV library already holds" {
local src="$LIB/Monk 2002 Seasons 1 to 8/Season 1/Monk - S01E01 - Mr Monk and the Rival.mkv"
for e in 01 02 03 04 05; do
at "$TVLIB/Monk (2002)/Season 01/Monk - S01E$e - Mr Monk and the Rival.mkv" 3000000
done
tv_plan
[ -z "$(tv_dst_of "$src")" ]
[ -e "$src" ]
grep -q "Monk.*already in" "$BATS_TEST_TMPDIR/tv.out"
}
@test "an episode the TV library holds in another encode is still moved" {
for e in 01 02 03 04 05; do
at "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E$e 1080p.mp4" 4000000
done
tv_plan
tv_apply
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01.mp4" ]
[ -e "$TVLIB/Big Little Lies/Season 01/Big Little Lies S01E01 1080p.mp4" ]
}
@test "every file in the census is placed or kept — nothing is dropped" {
plan
python3 - "$ENUM" "$OUT" <<'PY'
import sys
enum = {l.rstrip("\n").split("\t")[2] for l in open(sys.argv[1])}
placed = set()
for line in open(sys.argv[2]):
f = line.rstrip("\n").split("\t")
if f[0] in ("move", "keep"):
placed.add(f[1])
missing = enum - placed
assert not missing, f"{len(missing)} census files with no plan row: {sorted(missing)[:3]}"
PY
grep -q "left-unmoved=0" "$PLANOUT"
grep -q "double-claim=0" "$PLANOUT"
grep -q "occupied=0" "$PLANOUT"
grep -q "bad-names=0" "$PLANOUT"
grep -q "unplanned=0" "$EXC"
}
# ---------------------------------------------------------------- executor
# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
apply_plan() {
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
# `/` is always a mountpoint, so the overlay note behaves as it does on a
# host that runs the metadata view.
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" \
--journal "$JOURNAL" --metadata-view / --apply
}
@test "a run without --apply says what it would do and touches nothing" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# a dry run may not even build the folders: an operator must be able to
# run it against the live library with no --apply and change nothing
[ ! -d "$LIB/V-H-S-2 (2013)" ]
[ ! -e "$JOURNAL" ]
}
@test "apply moves every planned file, keeps its bytes, and journals each move" {
plan
apply_plan
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
[ ! -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
local planned moved
planned=$(awk -F'\t' '$1=="move"' "$OUT" | wc -l)
moved=$(python3 -c 'import json,sys
print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "$JOURNAL")
[ "$moved" -eq "$planned" ]
# The media servers' sidecars live in the metadata overlay and did not follow
# these moves, so the run names that cleanup at the moment it is needed.
[[ "$output" == *"media-view.sh reap"* ]]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
}
# The overlay note is only true where the metadata view is really mounted. On a
# host whose media servers bind the array directly (dankserver today: both
# quadlets mount /mnt/entropy, /var/lib/media-meta does not exist) the sidecars
# moved with the media, and "next: run reap" is an order that cannot be carried
# out. --metadata-view is how the caller names that path; the default is the
# host's view path, and the note rides only when it is a mountpoint.
@test "an apply with no mounted metadata view does not order a reap" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"moved"* ]]
[[ "$output" != *"media-view.sh reap"* ]]
}
@test "a rollback with no mounted metadata view does not order a reap" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
run python3 "$APPLY" rollback --journal "$JOURNAL" --root "$LIB" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"rolled back"* ]]
[[ "$output" != *"media-view.sh reap"* ]]
}
@test "a target already holding a different file stops the whole run" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
at "$LIB/Alien (1979)/Alien (1979).mkv" 7
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"occupied"* ]]
# nothing moved at all: the other films are still where the census saw them
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ ! -e "$JOURNAL" ]
}
@test "a plan row whose source has gone stops the run instead of inventing a move" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
rm "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv"
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
}
@test "a run interrupted between the journal write and the move finishes the job" {
plan
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
# The journal is written before each move, so an interruption leaves two
# states and only two: an intent whose move happened, and an intent whose
# move never ran. Both are simulated here by hand.
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"][:3]
with open(sys.argv[2], "w") as fh:
for i, r in enumerate(moves):
fh.write(json.dumps({"op": "move", "src": r[1], "dst": r[2],
"bytes": os.path.getsize(r[1])}) + "\n")
if i < 2: # the first two moves actually ran
os.makedirs(os.path.dirname(r[2]), exist_ok=True)
os.rename(r[1], r[2])
PY
run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
python3 - "$OUT" "$JOURNAL" <<'PY'
import json, os, sys
moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
if r[0] == "move"]
journal = [json.loads(l) for l in open(sys.argv[2])]
seen = [j["dst"] for j in journal]
assert len(seen) == len(set(seen)), "a destination was journalled twice"
for r in moves:
assert os.path.exists(r[2]) and not os.path.exists(r[1]), r[2]
PY
}
@test "rollback puts every file back where the census found it" {
plan
apply_plan
[ "$status" -eq 0 ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
[ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
[ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
# the folders the run created are empty afterwards and must not litter the
# library: this one never existed before the plan ran
[ -d "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/V-H-S-2 (2013)" ]
}
@test "rollback refuses when a moved file has gone missing" {
plan
apply_plan
[ "$status" -eq 0 ]
rm "$LIB/Alien (1979)/Alien (1979).mkv"
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"missing"* ]]
# the refusal is all-or-nothing: another film did not move back
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
}
# ---------------------------------------------------------------- sweep
#
# A relayout leaves the release folders it emptied standing at the library root:
# 1675 of them after the live run, each holding nothing but empty subfolders. The
# sweep pass removes only folders THIS RUN emptied — proven from the journal,
# never from "it looks empty" — so a folder someone made by hand is untouchable,
# and every removal rides the same rollback as the moves.
@test "sweep removes only the folders this run emptied" {
plan
apply_plan
mkdir -p "$LIB/A Empty Placeholder" # someone's folder, never a move source
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept"* ]]
[ ! -e "$LIB/AMERICAN POP" ]
[ ! -e "$LIB/VHS 2 (2013)" ]
[ ! -e "$LIB/Alien (1979)/Featurettes" ]
[ ! -e "$LIB/Alien (1979)/subs" ]
[ ! -e "$LIB/Marnie (1964)/Exrtas" ]
# a folder the run never moved a file out of survives, empty or not
[ -d "$LIB/A Empty Placeholder" ]
}
@test "a swept folder carries the empty junk folders inside it" {
mkdir -p "$LIB/AMERICAN POP/Screens" # screenshots, never a move source
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"0 left in place"* ]]
[ ! -e "$LIB/AMERICAN POP/Screens" ]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "a sweep whose rmdir failed tries those folders again" {
plan
apply_plan
mkdir -p "$LIB/AMERICAN POP/Screens"
chmod 500 "$LIB/AMERICAN POP" # readable, but nothing can be removed
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"left in place"* ]]
[ -e "$LIB/AMERICAN POP/Screens" ]
chmod 700 "$LIB/AMERICAN POP" # the operator cleared the blocker
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" != *"left in place: "* ]]
[ ! -e "$LIB/AMERICAN POP" ]
}
@test "sweep without --apply reports the husks and changes nothing" {
plan
apply_plan
local before jlines
before=$(find "$LIB" -type d | wc -l)
jlines=$(wc -l < "$JOURNAL")
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL"
[ "$status" -eq 0 ]
[[ "$output" == *"--apply"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
[ "$(wc -l < "$JOURNAL")" -eq "$jlines" ]
[ -e "$LIB/AMERICAN POP" ]
}
@test "a folder that still holds content is never swept" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# the film folder, the folder the run created, and the piles all stay
[ -d "$LIB/Marnie (1964)" ]
[ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
[ -d "$LIB/V-H-S-2 (2013)" ]
[ -d "$LIB/_unsorted bonus/extras" ]
}
@test "rollback after a sweep brings the folders and the files back" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ ! -e "$LIB/AMERICAN POP" ]
run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[ -e "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" ]
[ -e "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" ]
[ -e "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" ]
[ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
}
@test "a second sweep has nothing left to do and says so" {
plan
apply_plan
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
local before
before=$(find "$LIB" -type d | wc -l)
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
[[ "$output" == *"swept: 0"* ]]
[ "$(find "$LIB" -type d | wc -l)" -eq "$before" ]
}
@test "sweep never follows a symlink and never removes the folder it guards" {
plan
apply_plan
mkdir -p "$LIB/Elsewhere"
at "$LIB/Elsewhere/keep.txt" 10
ln -s "$LIB/Elsewhere" "$LIB/AMERICAN POP/pointer"
run python3 "$APPLY" sweep --root "$LIB" --journal "$JOURNAL" --apply
[ "$status" -eq 0 ]
# a link is content, not nothing: the husk it sits in survives untouched
[ -d "$LIB/AMERICAN POP" ]
[ -L "$LIB/AMERICAN POP/pointer" ]
[ -e "$LIB/Elsewhere/keep.txt" ]
}
@test "sweep refuses a journal that belongs to another library" {
plan
apply_plan
mkdir -p "$BATS_TEST_TMPDIR/Other"
run python3 "$APPLY" sweep --root "$BATS_TEST_TMPDIR/Other" \
--journal "$JOURNAL" --apply
[ "$status" -ne 0 ]
[[ "$output" == *"outside"* ]]
[ -d "$LIB/AMERICAN POP" ]
}
+459
View File
@@ -0,0 +1,459 @@
#!/usr/bin/env bats
# Seam: the TV-library relayout — scripts/media-census.py jellyfin (the metadata
# authority export) and scripts/media-shows.py (the plan).
#
# The TV library is 52k files written in five different release layouts, and the
# one thing that makes a rename of that size survivable is that the plan is
# derived from what a metadata provider already matched, not from a guess at the
# file name. So every test here states which authority a file had — name only,
# provider only, both, neither — and asserts where the PLAN files it, never how
# the planner decided. A fixture Jellyfin database stands in for the live one and
# the name caches are input files, so no test touches the network.
#
# The target contract is tvdb-grade naming:
# <Series> (<year>)/Season NN/<Series> (<year>) - SxxEyy - <Episode Title>.<ext>
# tvdb is the IDENTITY authority: a provider name a server derived from a folder
# is not a series name, so a polluted name is undone through the series' tvdb id.
# The review pile parks OUTSIDE the library — a parked folder inside it indexes
# as one junk show, and that show's rows are then mistaken for evidence.
#
# The two outcomes that cost a re-run of a 50k-file plan are a file filed as an
# episode that is not one (it steals an episode number from its real owner) and a
# move that lands on an existing file. Everything below is written to catch
# exactly those, plus the gate that matters most: the plan must be replayable by
# scripts/media-apply.py, whose preflight is the fleet's only overwrite guard.
load helpers
PLAN="$REPO_ROOT/scripts/media-shows.py"
CENSUS="$REPO_ROOT/scripts/media-census.py"
APPLY="$REPO_ROOT/scripts/media-apply.py"
# at PATH SIZE — a file with deterministic content.
at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
# title_row FILE SERIES YEAR SEASON EPISODE [EXTRA] [TMDB] [TITLE] [TVDB]
# — one row of the authority export scripts/media-census.py jellyfin writes.
title_row() {
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" \
"${5:-}" "${6:-}" "${7:-}" "${8:-}" "${9:-}" >> "$TITLES"
}
# dst SRC — the destination the plan gives SRC, or nothing when it does not move.
dst() {
awk -F '\t' -v s="$1" '$1 == "move" && $2 == s {print $3}' "$OUT"
}
setup() {
LIB="$BATS_TEST_TMPDIR/TVShows"
PARK="$LIB/_unsorted"
OUTSIDE="$BATS_TEST_TMPDIR/park"
MOVIES="$BATS_TEST_TMPDIR/Movies"
ENUM="$BATS_TEST_TMPDIR/enum.tsv"
PROBED="$BATS_TEST_TMPDIR/probed.tsv"
TITLES="$BATS_TEST_TMPDIR/titles.tsv"
CACHE="$BATS_TEST_TMPDIR/tmdb.json"
OUT="$BATS_TEST_TMPDIR/plan.tsv"
EXC="$BATS_TEST_TMPDIR/exceptions.txt"
: > "$PROBED"
SEASON="$LIB/The Examples (2010) Season 1 S01 (1080p BluRay x265 Silence)"
# An episode its own name states, with a subtitle, a season image and a
# trailer: four things that have to end up in four different places.
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" 9000
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt" 200
at "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt" 250
# A scene pack buries its subtitles two levels down, in Subs/<release-name>/.
at "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt" 220
at "$SEASON/folder.jpg" 300
at "$SEASON/Trailers/teaser.mkv" 1000
at "$SEASON/tvshow.nfo" 400
# An anime pack: no episode marker anywhere in its name, numbered by the
# provider only. Its year is the one thing no authority states.
ANIME="$LIB/Anime Pack [grp] (1080p HEVC)"
at "$ANIME/[grp] Anime Pack - 05.mkv" 5000
# Two releases claiming one episode, and a file holding two episodes.
at "$LIB/Dupes/Show Two S01E01.mkv" 4000
at "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" 7000
at "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" 7000
# A file no authority numbers, and a long file the movie provider knows.
at "$LIB/Orphan Tapes/clip001.mp4" 3000
at "$LIB/Longform/Some Documentary Special (2015).mkv" 8000
# A bonus folder inside a release pack whose own name is not the show's. The
# pack's rows name the show; the bonus folder may never invent a series.
at "$ANIME/01. Extras/NCOP 01.mkv" 1200
# A season folder whose rows are stale: the database numbers every file of this
# pack season 16 episode 20, four seasons' worth of them.
at "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" 6000
at "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" 4000
# A box-set folder whose Series item the server named AFTER THE FOLDER: the
# row's own name is junk, and only a provider id can undo it.
BOX="$LIB/Show Box (2016) Season 1 S01 (1080p WEB-DL)"
at "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" 6000
# The park inside the library: the server indexed the whole pile as ONE show
# named after the folder, so its rows state a series that is not a show.
at "$PARK/Park Show S01E01.mkv" 5000
at "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" 5200
# The enum census lists every file; the planner reads that instead of walking
# the tree, so the fixture writes the same three columns.
python3 - "$LIB" "$ENUM" <<'PY'
import os, sys
root, out = sys.argv[1], sys.argv[2]
with open(out, "w") as fh:
for d, _, fs in os.walk(root):
for f in fs:
p = os.path.join(d, f)
fh.write(f"{os.stat(p).st_size}\t{os.path.splitext(f)[1]}\t{p}\n")
PY
printf 'file\tseries\tyear\tseason\tepisode\textra\ttmdb\ttitle\ttvdb\n' > "$TITLES"
printf '{}\n' > "$CACHE"
title_row "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv" \
"The Examples" 2010 1 1 "" "" "Pilot"
title_row "$ANIME/[grp] Anime Pack - 05.mkv" "Anime Pack" "" 1 5
title_row "$LIB/Dupes/Show Two S01E01.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
title_row "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv" "Show Two" 2016 1 1 "" "" "Two Pilots"
title_row "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv" "Show Two" 2016 1 2 "" "" "Double Trouble"
title_row "$LIB/Loose.Show.S02E03.720p.WEB-DL.mkv" "Loose Show" 2019 2 3 "" "" "Loose Ends"
title_row "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi" \
"SNL Pack" "" 16 20
title_row "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv" \
"Show Box (2016) Season 1" 2016 1 1 "" "2334" "Pilot" "78850"
# The row the server wrote for a parked file: the series is the PARK FOLDER.
title_row "$PARK/Park Show S01E01.mkv" "_unsorted" "" 1 1
}
plan() {
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
--out "$OUT" --exceptions "$EXC"
}
plan_park() { # the review pile outside the library
run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --titles "$TITLES" \
--probed "$PROBED" --tmdb-cache "$CACHE" --movies "$MOVIES" \
--park "$OUTSIDE" --out "$OUT" --exceptions "$EXC"
}
@test "--tmdb with no key is reported as an unfinished plan, not a finished one" {
# A dead provider is invisible where it happens: every 401 is one line on
# stderr, and a terminal that scrolled past them shows a plan that reads as
# complete while every folder still carries the metadata provider's junk. The
# plan is still written — the answers that DID arrive are worth their rename —
# so the state has to ride in the summary line and in the review file.
# env -u keeps this offline no matter what the developer's shell exports.
run env -u TMDB_API_KEY python3 "$PLAN" --root "$LIB" --enum "$ENUM" \
--titles "$TITLES" --probed "$PROBED" --tmdb-cache "$CACHE" \
--movies "$MOVIES" --tmdb --out "$OUT" --exceptions "$EXC"
[ "$status" -eq 0 ]
grep -q 'tmdb NOT ASKED (TMDB_API_KEY unset): NOT provider-complete' <<<"$output"
grep -q 'WARNING: tmdb NOT ASKED (TMDB_API_KEY unset) — NOT provider-complete' "$EXC"
grep -qE '^plan: [1-9][0-9]* moves' <<<"$output"
}
@test "the census exports the provider's episode numbering as a TSV" {
DB="$BATS_TEST_TMPDIR/jellyfin.db"
python3 - "$DB" "$SEASON" <<'PY'
import os, sqlite3, sys, uuid
db, season = sys.argv[1], sys.argv[2]
c = sqlite3.connect(db)
pre = "/data/TVShows/" + os.path.basename(season)
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, ExtraType,
IsVirtualItem, Data)""")
sid = str(uuid.uuid4())
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
pre, None, None, 2010, None, 0,
'{"ProviderIds":{"Tmdb":"1234","Tvdb":"78850"}}'))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(str(uuid.uuid4()), sid, "MediaBrowser.Controller.Entities.TV.Episode",
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
None, None, 0, '{}'))
c.commit()
PY
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp.tsv"
[ "$status" -eq 0 ]
run head -1 "$BATS_TEST_TMPDIR/exp.tsv"
[ "${lines[0]}" = "file series year season episode extra tmdb title tvdb" ]
tail -1 "$BATS_TEST_TMPDIR/exp.tsv" \
| grep -q "^$SEASON/The\.Examples\.S01E01\.Pilot\.1080p\.x265\.mkv The Examples 2010 1 1 1234 Pilot 78850$"
}
@test "the census reads provider ids from the table newer Jellyfin keeps" {
DB="$BATS_TEST_TMPDIR/jellyfin12.db"
python3 - "$DB" "$SEASON" <<'PY'
import os, sqlite3, sys, uuid
db, season = sys.argv[1], sys.argv[2]
c = sqlite3.connect(db)
pre = "/data/TVShows/" + os.path.basename(season)
c.execute("""create table BaseItems (Id, ParentId, Type, Name, Path, IndexNumber,
ParentIndexNumber, ProductionYear, ExtraType,
IsVirtualItem, Data)""")
# Jellyfin 12 moved the ids out of the Data blob and into their own table.
c.execute("create table BaseItemProviders (ItemId, ProviderId, ProviderValue)")
sid = str(uuid.uuid4())
eid = str(uuid.uuid4())
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(sid, None, "MediaBrowser.Controller.Entities.TV.Series", "The Examples",
pre, None, None, 2010, None, 0, '{}'))
c.execute("insert into BaseItems values (?,?,?,?,?,?,?,?,?,?,?)",
(eid, sid, "MediaBrowser.Controller.Entities.TV.Episode",
"Pilot", pre + "/The.Examples.S01E01.Pilot.1080p.x265.mkv", 1, 1,
None, None, 0, '{}'))
c.execute("insert into BaseItemProviders values (?,?,?)", (sid, "Tvdb", "9999"))
c.commit()
PY
run python3 "$CENSUS" jellyfin --root "$LIB" --db "$DB" \
--out "$BATS_TEST_TMPDIR/exp12.tsv"
[ "$status" -eq 0 ]
tail -1 "$BATS_TEST_TMPDIR/exp12.tsv" \
| grep -q " Pilot 9999$"
}
@test "an episode its own name states files as the titled standard name" {
plan
[ "$status" -eq 0 ]
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv")" \
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.mkv" ]
}
@test "the tvdb identity names the series folder, not the name a folder left behind" {
printf '{"tvdb:series:78850": {"name": "Show Box", "year": "2016"}}\n' > "$CACHE"
plan
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" \
= "$LIB/Show Box (2016)/Season 01/Show Box (2016) - S01E01 - Pilot.mkv" ]
}
@test "a provider name that already carries the year is not given it twice" {
# TVDb answers the reboot as `Acapulco (2021)`. The folder contract is ONE year,
# and `Acapulco (2021) (2021)` is a folder neither matcher nor human reuses.
printf '{"tvdb:series:78850": {"name": "Show Box (2016)", "year": "2016"}}\n' > "$CACHE"
plan
[ "$(dst "$BOX/Show.Box.S01E01.1080p.WEB-DL.mkv")" \
= "$LIB/Show Box (2016)/Season 01/Show Box (2016) - S01E01 - Pilot.mkv" ]
run grep -F '(2016) (2016)' "$OUT"
[ -z "$output" ]
}
@test "a series no authority resolves keeps the provider's name and is reported" {
plan
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
grep -A9 -i "no tvdb" "$EXC" | grep -q "SNL Pack"
}
@test "a row whose series is the park folder is not evidence of a show" {
plan_park
run awk -F '\t' -v p="$PARK/" '$1 == "move" && index($3, p) == 1 {print $3}' "$OUT"
[ -z "$output" ]
run awk -F '\t' '$1 == "move" && $2 ~ /Park Show S01E01/ {print $3}' "$OUT"
[[ "$output" == "$OUTSIDE/"* ]]
}
@test "the pile's sanitized display name is still the pile, not a series" {
# Jellyfin drops the leading underscore when it shows the hidden folder, so the
# row says the series is "unsorted" while the file sits in `_unsorted`. Match
# only the underscored spelling and the plan renames real files into a visible
# folder named after the review pile — which is how 617 of them got there.
title_row "$PARK/Twisted.Metal.S01E01.1080p.WEB-DL.mkv" "unsorted" "" 1 1 \
"" "" "Twisted Metal"
plan_park
run awk -F '\t' -v l="$LIB" \
'$1 == "move" && index($3, l "/unsorted/") == 1 {print $3}' "$OUT"
[ -z "$output" ]
run awk -F '\t' '$1 == "move" && $2 ~ /Twisted\.Metal\.S01E01/ {print $3}' "$OUT"
[[ "$output" == "$OUTSIDE/"* ]]
}
@test "a subtitle follows its episode and keeps its language flag" {
plan
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.en.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.en.srt" ]
}
@test "a subtitle two folders down inside Subs/<release-name>/ still follows its episode" {
plan
[ "$(dst "$SEASON/Subs/The.Examples.S01E01.Pilot.1080p.x265/The.Examples.S01E01.Pilot.1080p.x265.jpn.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.jpn.srt" ]
}
@test "a subtitle that carries the container's word in its own name still follows the episode" {
plan
[ "$(dst "$SEASON/The.Examples.S01E01.Pilot.1080p.x265.mkv.forced.srt")" \
= "$LIB/The Examples (2010)/Season 01/The Examples (2010) - S01E01 - Pilot.forced.srt" ]
}
@test "a season image keeps its own name; a series nfo keeps the series folder" {
plan
[ "$(dst "$SEASON/folder.jpg")" = "$LIB/The Examples (2010)/Season 01/folder.jpg" ]
[ "$(dst "$SEASON/tvshow.nfo")" = "$LIB/The Examples (2010)/tvshow.nfo" ]
}
@test "bonus material files as an extra, never as an episode" {
plan
[ "$(dst "$SEASON/Trailers/teaser.mkv")" = "$LIB/The Examples (2010)/trailers/teaser.mkv" ]
[ -z "$(awk -F '\t' '$3 ~ /Season .*teaser/ {print $3}' "$OUT")" ]
}
@test "a file only the provider numbers files at the episode it is matched to, title or no title" {
plan
[ "$(dst "$ANIME/[grp] Anime Pack - 05.mkv")" \
= "$LIB/Anime Pack/Season 01/Anime Pack - S01E05.mkv" ]
}
@test "a bonus folder inside a release pack belongs to the pack's show, not to its own name" {
plan
[ "$(dst "$ANIME/01. Extras/NCOP 01.mkv")" = "$LIB/Anime Pack/extras/NCOP 01.mkv" ]
[ -z "$(awk -F '\t' '$3 ~ /\/01\. Extras\// {print $3}' "$OUT")" ]
}
@test "a Season folder outvotes the stale row its files carry" {
plan
[ "$(dst "$LIB/SNL Pack/Season 19/01 Charles Barkley, Nirvana.avi")" \
= "$LIB/SNL Pack/Season 19/SNL Pack - S19E01.avi" ]
}
@test "one file holding two episodes keeps both numbers in its name" {
plan
[ "$(dst "$LIB/Dupes/Show.Two.S01E02-E03.720p.mkv")" \
= "$LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E02-E03 - Double Trouble.mkv" ]
}
@test "two files claiming one episode: one keeps the name, the other parks" {
plan
n=$(grep -c " $LIB/Show Two (2016)/Season 01/Show Two (2016) - S01E01 - Two Pilots.mkv " "$OUT")
[ "$n" -eq 1 ]
run dst "$LIB/Dupes/Show Two S01E01.mkv"
[[ "$output" == "$PARK/"* ]]
run dst "$LIB/Dupes/Show.Two.S01E01.720p.other.mkv"
[[ "$output" == "$LIB/Show Two (2016)/Season 01/"* ]]
}
@test "a file no authority numbers is parked, never renamed into the library" {
plan
run dst "$LIB/Orphan Tapes/clip001.mp4"
[[ "$output" == "$PARK/Orphan Tapes/clip001.mp4" ]]
grep -q "clip001.mp4" "$EXC"
}
@test "a long file the movie provider knows leaves the TV library for the movie one" {
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
python3 -c 'import json, sys
json.dump({"movie:some documentary":
{"name": "Some Documentary Special", "year": 2015, "id": 55}},
open(sys.argv[1], "w"))' "$CACHE"
plan
[ "$(dst "$LIB/Longform/Some Documentary Special (2015).mkv")" \
= "$MOVIES/Some Documentary Special (2015)/Some Documentary Special (2015).mkv" ]
}
@test "a long file no movie provider knows is parked, not moved into Movies" {
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
plan
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
[[ "$output" == "$PARK/"* ]]
}
@test "a movie answer the file does not itself name stays in the review pile" {
# TMDb's top hit for a truncated query is a NEAR miss, not an answer: "mobile
# suit gundam 00" comes back as Special Edition I while the file is Special
# Edition III, and a one-word query "russia" comes back as a 1963 Bond film. A
# file leaves the TV library only when it states the title the provider names.
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
python3 -c 'import json, sys
json.dump({"movie:some documentary":
{"name": "Some Documentary Special Part Two", "year": "2015", "id": 55}},
open(sys.argv[1], "w"))' "$CACHE"
plan
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
[[ "$output" == "$PARK/"* ]]
run awk -F '\t' -v m="$MOVIES/" '$1 == "move" && index($3, m) == 1 {print $3}' "$OUT"
[ -z "$output" ]
}
@test "a movie answer from another year is not that film" {
# The year the file states outranks a provider's eagerness: the 2015 special is
# not the 1999 one, however alike their names are.
printf '8000\t%s\t7200\thevc\t1920x1080\t23.976\t4000000\tyuv420p10le\taac:eng\t0\t0\tmatroska,webm\n' \
"$LIB/Longform/Some Documentary Special (2015).mkv" > "$PROBED"
python3 -c 'import json, sys
json.dump({"movie:some documentary":
{"name": "Some Documentary Special", "year": "1999", "id": 55}},
open(sys.argv[1], "w"))' "$CACHE"
plan
run dst "$LIB/Longform/Some Documentary Special (2015).mkv"
[[ "$output" == "$PARK/"* ]]
}
@test "the plan creates every folder before the move that needs it" {
plan
python3 - "$OUT" "$LIB" "$MOVIES" "$PARK" <<'PY'
import os, sys
plan, known = sys.argv[1], set(sys.argv[2:])
made, bad = set(), []
for line in open(plan).read().splitlines()[1:]:
action, _src, dest, _kind = line.split("\t")
if action == "mkdir":
made.add(dest)
elif action == "move":
d = os.path.dirname(dest)
while d not in known and d != os.path.dirname(d):
if d not in made:
bad.append(f"{dest}: {d} was never made")
d = os.path.dirname(d)
print("\n".join(bad[:5]))
sys.exit(1 if bad else 0)
PY
}
@test "no move lands outside the library or the movie library" {
plan
run awk -F '\t' -v l="$LIB" -v m="$MOVIES" \
'$1 == "move" && $3 !~ "^" l "/" && $3 !~ "^" m "/" {print $3}' "$OUT"
[ -z "$output" ]
}
@test "the executor preflights the plan and no target is claimed twice" {
plan
run python3 "$APPLY" apply --plan "$OUT" --root "$BATS_TEST_TMPDIR" \
--journal "$BATS_TEST_TMPDIR/journal.jsonl"
[ "$status" -eq 0 ]
dup=$(awk -F '\t' '$1 == "move" {print $3}' "$OUT" | sort | uniq -d)
[ -z "$dup" ]
}
@test "re-planning a library the plan already satisfies moves nothing" {
plan
# Applying the plan and re-censusing is exactly what the operator does next:
# every authority file is keyed by path, so all three are replayed through the
# plan the way a fresh census would see them. (The enum has no header; the
# titles export does, and its header row is not a path so it survives.)
python3 - "$OUT" "$ENUM" "$TITLES" "$PROBED" <<'PY'
import os, sys
plan = sys.argv[1]
moves = {}
for line in open(plan).read().splitlines()[1:]:
action, src, dest, _kind = line.split("\t")
if action == "move":
moves[src] = dest
os.makedirs(os.path.dirname(dest), exist_ok=True)
os.rename(src, dest)
for path, col in ((sys.argv[2], 2), (sys.argv[3], 0), (sys.argv[4], 1)):
if not os.path.getsize(path):
continue
rows = [line.rstrip("\n").split("\t") for line in open(path)]
with open(path, "w") as fh:
for f in rows:
f[col] = moves.get(f[col], f[col])
fh.write("\t".join(f) + "\n")
PY
plan
[ -z "$(awk -F '\t' '$1 == "move" {print $2}' "$OUT")" ]
}
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env bats
# Seam: scripts/media-verify.py — the replay of a relayout journal against a
# media server's OWN database, which is the documented proof that a rename
# landed (a scan answers 204 whether or not it finds anything).
#
# Every test states what the database holds and asserts what the command prints
# and returns; nothing here reaches into the script's internals. The fixtures are
# minimal sqlite files with the columns the two servers really use, so the seam
# fails if the tool ever reads the wrong table, forgets Plex's `deleted_at`
# filter, or forgets that Jellyfin stores the container-side path (/data/…) while
# Plex stores the host path verbatim.
load helpers
VERIFY="$REPO_ROOT/scripts/media-verify.py"
# jf_mk <db> — the Jellyfin shape: one BaseItems table, Path is the column.
jf_mk() {
python3 - "$1" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("create table BaseItems (Id INTEGER PRIMARY KEY, Path TEXT, Type TEXT,"
" ParentId INTEGER, Name TEXT)")
con.commit()
PY
}
# jf_row <db> <path> — one item the server believes it has.
jf_row() {
python3 - "$1" "$2" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("insert into BaseItems (Path, Type) values (?, 'x')", (sys.argv[2],))
con.commit()
PY
}
# px_mk <db> — the Plex shape: media_parts joined up to metadata_items.
px_mk() {
python3 - "$1" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1])
con.execute("create table metadata_items (id INTEGER PRIMARY KEY, title TEXT,"
" deleted_at TEXT)")
con.execute("create table media_items (id INTEGER PRIMARY KEY, metadata_item_id"
" INTEGER, container TEXT)")
con.execute("create table media_parts (id INTEGER PRIMARY KEY, media_item_id"
" INTEGER, file TEXT, deleted_at TEXT)")
con.commit()
PY
}
# px_row <db> <path> [DELETED_AT] — one part; a non-empty DELETED_AT is a row a
# deleted scan left behind, which must never count as proof.
px_row() {
python3 - "$1" "$2" "${3:-}" <<'PY'
import sqlite3, sys
db, path, gone = sys.argv[1], sys.argv[2], sys.argv[3] or None
con = sqlite3.connect(db)
cur = con.execute("insert into metadata_items (title, deleted_at) values ('t', ?)",
(gone,))
mi = cur.lastrowid
cur = con.execute("insert into media_items (metadata_item_id, container)"
" values (?, 'mkv')", (mi,))
con.execute("insert into media_parts (media_item_id, file, deleted_at)"
" values (?, ?, NULL)", (cur.lastrowid, path))
con.commit()
PY
}
# move_row <src> <dst> — one move in the journal.
move_row() {
printf '{"op": "move", "src": "%s", "dst": "%s"}\n' "$1" "$2" >> "$JOURNAL"
}
setup() {
SHARE="$BATS_TEST_TMPDIR/mnt/entropy"
JFDB="$BATS_TEST_TMPDIR/jellyfin.db"
PXDB="$BATS_TEST_TMPDIR/plex.db"
JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
: > "$JOURNAL"
jf_mk "$JFDB"
px_mk "$PXDB"
}
@test "every move that the database holds at its destination is reported landed" {
move_row "$SHARE/Movies/Old Name (1999).mkv" "$SHARE/Movies/Old Name (1999)/Old Name (1999).mkv"
move_row "$SHARE/Movies/Two.mkv" "$SHARE/Movies/Two/Two.mkv"
# Jellyfin stores the container-side path: /data stands for the share.
jf_row "$JFDB" "/data/Movies/Old Name (1999)/Old Name (1999).mkv"
jf_row "$JFDB" "/data/Movies/Two/Two.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[0]}" = "jellyfin: 2 moves checked" ]
[ "${lines[1]}" = "landed: 2" ]
[ "${lines[2]}" = "still at src: 0" ]
[ "${lines[3]}" = "missing from the DB: 0" ]
}
@test "a database that has not rescanned yet is refused, naming the stale rows" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
jf_row "$JFDB" "/data/Movies/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[1]}" = "landed: 0" ]
[ "${lines[2]}" = "still at src: 1" ]
[[ "$output" == *"$SHARE/Movies/Old.mkv"* ]]
}
@test "a moved file no row of the database names is reported missing" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 1" ]
[[ "$output" == *"$SHARE/Movies/Old/Old.mkv"* ]]
}
@test "Plex stores the host path verbatim, so a row at the destination lands" {
move_row "$SHARE/TVShows/Show/Season 01/Show S01E01.mkv" \
"$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
px_row "$PXDB" "$SHARE/TVShows/Show (2002)/Season 01/Show (2002) S01E01.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
--share "$SHARE"
[ "$status" -eq 0 ]
[ "${lines[0]}" = "plex: 1 moves checked" ]
[ "${lines[1]}" = "landed: 1" ]
}
@test "a Plex row a deleted scan left behind is not proof a move landed" {
move_row "$SHARE/TVShows/Old S01E01.mkv" "$SHARE/TVShows/Show/Old S01E01.mkv"
px_row "$PXDB" "$SHARE/TVShows/Show/Old S01E01.mkv" "2026-10-30 00:00:00"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind plex --db "$PXDB" \
--share "$SHARE"
[ "$status" -eq 1 ]
[ "${lines[1]}" = "landed: 0" ]
[ "${lines[3]}" = "missing from the DB: 1" ]
}
@test "a database without the columns that kind needs is an error, not a zero" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$PXDB" \
--share "$SHARE" --container /data
[ "$status" -eq 2 ]
[[ "$output" == *"$PXDB"* ]]
[[ "$output" == *"BaseItems"* ]]
}
@test "only move rows are counted, so folder operations cannot inflate the tally" {
move_row "$SHARE/Movies/Old.mkv" "$SHARE/Movies/Old/Old.mkv"
jf_row "$JFDB" "/data/Movies/Old/Old.mkv"
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
printf '{"op": "rmdir", "path": "%s/Movies/Empty"}\n' "$SHARE" >> "$JOURNAL"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[0]}" = "jellyfin: 1 moves checked" ]
}
@test "a journal with no moves is clean, not a failure" {
printf '{"op": "mkdir", "dst": "%s/Movies/Old"}\n' "$SHARE" >> "$JOURNAL"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 0 ]
[ "${lines[1]}" = "landed: 0" ]
}
@test "the offender list stops at --show so a stale library stays readable" {
for n in 1 2 3 4; do
move_row "$SHARE/Movies/F$n.mkv" "$SHARE/Movies/F$n/F$n.mkv"
done
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data --show 2
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 4" ]
[[ "$output" == *"$SHARE/Movies/F1/F1.mkv"* ]]
[[ "$output" != *"$SHARE/Movies/F4/F4.mkv"* ]]
}
@test "the missing rows are counted by extension, so a sidecar is not mistaken for a lost film" {
move_row "$SHARE/Music/Album/Mandatory Fun.log" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.log"
move_row "$SHARE/Music/Album/Film.mkv" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Film.mkv"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data
[ "$status" -eq 1 ]
[ "${lines[3]}" = "missing from the DB: 2" ]
[ "${lines[4]}" = "missing by extension: .log 1, .mkv 1" ]
}
@test "--ignore-ext keeps a file the server never indexes out of the verdict" {
move_row "$SHARE/Music/Album/Song.flac" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
jf_row "$JFDB" "/data/Music/Weird Al/Mandatory Fun (2014)/Song.flac"
move_row "$SHARE/Music/Album/Mandatory Fun.cue" "$SHARE/Music/Weird Al/Mandatory Fun (2014)/Mandatory Fun.cue"
run python3 "$VERIFY" --journal "$JOURNAL" --db-kind jellyfin --db "$JFDB" \
--share "$SHARE" --container /data --ignore-ext log,cue
[ "$status" -eq 0 ]
[ "${lines[1]}" = "landed: 1" ]
[ "${lines[3]}" = "missing from the DB: 0" ]
[ "${lines[4]}" = "not indexed (extension ignored): 1" ]
}
+411
View File
@@ -200,6 +200,9 @@ DERIVE="$REPO_ROOT/scripts/derive-protected-experts.py"
APPLY="$REPO_ROOT/scripts/apply-protected-cast.sh"
CONVERT="$REPO_ROOT/scripts/convert-qwen38-w5nl64.sh"
GATE="$REPO_ROOT/scripts/kld-gate-qwen38.sh"
RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
PUSHER="$REPO_ROOT/scripts/calib-push.py"
WINDOW="$REPO_ROOT/scripts/qwen38-calib-window.sh"
TEMPLATE="$REPO_ROOT/scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe"
# make_gram <dir> <layer> <expert> <energy> <rows> — a RADGRAM2 down-Gram whose
@@ -525,3 +528,411 @@ gate_case() { # gate_case <kl fp8> <kl iq4r> <kl cand> <top1 cand> [extra flags.
[[ "$output" == *"has no slice 'code'"* ]]
[[ "$output" == *"it has: ops"* ]]
}
# --- the n-gram tier: the pin that starved the calibration boot ---------------
@test "the calibration boot reads the n-gram table from the file tier" {
# 2026-10-29: 95.37 GiB pinned beside a 65.59 GiB/rank pool left 8.06 GiB of
# RAM and the boot died "startup stopped: out of memory". calib only needs the
# routed weights prefilled, so it reads the table off the container's file tier.
run bash "$RUNNER" calib --dry-run
[ "$status" -eq 0 ]
grep -A1 -x -- '--ngram-placement' <<<"$output" | grep -qx 'disk'
}
@test "the recording and every score pass keep production's n-gram placement" {
# Those numbers are only comparable when the shape is identical; the tier is
# part of the shape.
run bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
grep -A1 -x -- '--ngram-placement' <<<"$output" | grep -qx 'ram'
}
@test "an unknown n-gram placement is refused by name before the boot" {
run env FRANK_QWEN38_NGRAM_PLACEMENT=ssd bash "$RUNNER" calib --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"auto|vram|ram|disk"* ]]
}
# --- the pusher: a tap without a live rig is not a tap ------------------------
# stub_podman <dir> <running|absent> — a podman that answers the one question the
# pusher asks: is the rig container running?
stub_podman() {
mkdir -p "$1/bin"
if [ "$2" = running ]; then
printf '#!/usr/bin/env bash\necho true\n' > "$1/bin/podman"
elif [ "$2" = stopped ]; then
printf '#!/usr/bin/env bash\necho false\n' > "$1/bin/podman"
else
printf '#!/usr/bin/env bash\nexit 125\n' > "$1/bin/podman"
fi
chmod +x "$1/bin/podman"
}
pusher_fixture() { # pusher_fixture <marker-container or empty>
D="$BATS_TEST_TMPDIR/push"
mkdir -p "$D/tap" "$D/bin"
printf '{"prompt":"hello"}\n' > "$D/corpus.jsonl"
if [ -n "$1" ]; then
printf 'container=%s\nimage=radiance@sha256:test\nmode=calib\npid=1\n' "$1" \
> "$D/tap/.rig-boot"
fi
echo "$D"
}
@test "the pusher refuses a tap that carries no rig marker: production is not a rig" {
# With no marker, the old driver fed four rounds of 10M tokens into restored
# production after the rig's boot had died, and recorded nothing.
D="$(pusher_fixture '')"
run python3 "$PUSHER" --corpus "$D/corpus.jsonl" --tap-dir "$D/tap" --port 1
[ "$status" -eq 2 ]
[[ "$output" == *"no rig marker"* ]]
}
@test "the pusher refuses a marker that names another container" {
D="$(pusher_fixture arbench)"
run python3 "$PUSHER" --corpus "$D/corpus.jsonl" --tap-dir "$D/tap" \
--container qwen38-bf16 --port 1
[ "$status" -eq 2 ]
[[ "$output" == *"names container 'arbench'"* ]]
[[ "$output" == *"targets 'qwen38-bf16'"* ]]
}
@test "the pusher refuses when the rig container is not running" {
D="$(pusher_fixture qwen38-bf16)"
stub_podman "$D" stopped
run env PATH="$D/bin:$PATH" python3 "$PUSHER" --corpus "$D/corpus.jsonl" \
--tap-dir "$D/tap" --container qwen38-bf16 --port 1
[ "$status" -eq 2 ]
[[ "$output" == *"is not running"* ]]
}
# --- the acceptance gate without an FP8 anchor -------------------------------
# kld_report_slices <path> <model> <kl all> <kl code> <top1 all> <top1 code>
kld_report_slices() {
python3 - "$@" <<'PY'
import json, sys
path, model, kl_all, kl_code, t_all, t_code = sys.argv[1:7]
def g(kl, t, n=25000):
return {"kld": {"mean": float(kl), "p99_9": float(kl) * 4},
"top1_agreement": float(t), "positions": n}
json.dump({"model": model, "all": g(kl_all, t_all),
"by_source": {"code": g(kl_code, t_code), "ops": g(kl_all, t_all)}},
open(path, "w"))
PY
}
gate2() { # gate2 <kl iq4r> <kl cand> <top1 iq4r> <top1 cand> [extra]
D="$BATS_TEST_TMPDIR/kl2"
mkdir -p "$D"
kld_report_slices "$D/iq4r.json" iq4r-moe "$1" "$1" "$3" "$3"
kld_report_slices "$D/cand.json" w5nl64 "$2" "$2" "$4" "$4"
run bash "$GATE" "$D/iq4r.json" "$D/cand.json" "${@:5}"
}
@test "a candidate that beats the served payload on mean, tail and top-1 passes without an anchor" {
gate2 0.0190 0.0090 0.962 0.971
[ "$status" -eq 0 ]
[[ "$output" == *"candidate vs served (no FP8 anchor)"* ]]
[[ "$output" == *"KLD GATE PASS"* ]]
}
@test "a better mean with a worse tail fails: the tail is where a mean hides its damage" {
D="$BATS_TEST_TMPDIR/kl3"
mkdir -p "$D"
kld_report_slices "$D/iq4r.json" iq4r-moe 0.0190 0.0190 0.962 0.962
python3 - "$D/cand.json" <<'PY'
import json, sys
# mean better than the served payload, p99.9 worse: the shape a mean-only gate waves through
g = {"kld": {"mean": 0.0090, "p99_9": 0.4000}, "top1_agreement": 0.971, "positions": 25000}
json.dump({"model": "w5nl64", "all": g, "by_source": {"code": dict(g), "ops": dict(g)}},
open(sys.argv[1], "w"))
PY
run bash "$GATE" "$D/iq4r.json" "$D/cand.json"
[ "$status" -eq 1 ]
[[ "$output" == *"[FAIL] 2."* ]]
[[ "$output" == *"KL p99.9"* ]]
}
@test "a candidate that wins overall but regresses one source slice fails" {
D="$BATS_TEST_TMPDIR/kl4"
mkdir -p "$D"
kld_report_slices "$D/iq4r.json" iq4r-moe 0.0190 0.0190 0.962 0.962
# 'all' is better, 'code' is worse: exactly the trade a mean-only gate cannot see
kld_report_slices "$D/cand.json" w5nl64 0.0090 0.0300 0.971 0.950
run bash "$GATE" "$D/iq4r.json" "$D/cand.json"
[ "$status" -eq 1 ]
[[ "$output" == *"[PASS] 1."* ]]
[[ "$output" == *"[FAIL] 4."* ]]
[[ "$output" == *"regressed: code"* ]]
}
# --- the calibration window: gated, and it says so ---------------------------
window_fixture() {
D="$BATS_TEST_TMPDIR/win"
mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora" "$D/bin"
echo '{}' > "$D/campaign/kld-ref/kld.json"
: > "$D/rad/qwen38-bf16.rad"
echo "{\"p\":1}" > "$D/corpora/calib.jsonl"
# the window's preflight resolves the image digest through podman and asks about the
# port/device by host pid; these stubs answer for a quiet host
cat > "$D/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "${1:-}" in
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
esac
EOS
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/ss"
printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/fuser"
chmod +x "$D/bin/podman" "$D/bin/ss" "$D/bin/fuser"
echo "$D"
}
@test "the calibration window refuses before touching a card without a reference recording" {
D="$(window_fixture)"
mv "$D/campaign/kld-ref/kld.json" "$D/campaign/kld-ref/kld.json.absent"
run env PATH="$D/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW"
[ "$status" -eq 2 ]
[[ "$output" == *"FAIL reference"* ]]
[[ "$output" == *"must not take the cards"* ]]
}
@test "the calibration window's dry run prints the gated plan, including what it would bank" {
D="$(window_fixture)"
run env PATH="$D/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$D/campaign" \
FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW" --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
[[ "$output" == *"else kill the rig"* ]]
[[ "$output" == *"calib-push.py"* ]]
[[ "$output" == *"AGENTTRASH"* ]]
}
@test "a flat calibration round is not a failure: the tap's floor is pooled" {
# round 1 of a 10.03M-token corpus against a 12M floor seals nothing, and the
# first version of this gate aborted the run there — one round too early.
run python3 - "$PUSHER" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("cp", sys.argv[1])
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
print(m.round_verdict(8192, 8192, False), # flat: continue
m.round_verdict(8192, 4096, False), # shrink: fail
m.round_verdict(8192, 20000, False), # grew: continue
m.round_verdict(0, 0, True)) # done: stop
PY
[ "$status" -eq 0 ]
[ "$output" = "flat shrink grew done" ]
}
@test "the calibration boot's step is its own knob; the KL modes keep production's" {
# Calib is the only phase whose cost is read volume per document: four 2,048-token steps
# re-stream each layer's routed experts four times. Measured 2026-10-29: 71 tok/s, 47 h tap.
run env Q38_CALIB_MNBT=8192 bash "$RUNNER" calib --dry-run
[ "$status" -eq 0 ]
grep -A1 -x -- '--max-num-batched-tokens' <<<"$output" | grep -qx '8192'
# and the same knob must NOT reach the measuring shape those numbers are compared in
run env Q38_CALIB_MNBT=8192 bash "$RUNNER" kldref --dry-run
[ "$status" -eq 0 ]
! grep -qx '8192' <<<"$output"
run env Q38_CALIB_MNBT=wide bash "$RUNNER" calib --dry-run
[ "$status" -eq 2 ]
[[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]]
}
# ---- the window must not mistake a dying production socket for the rig -------
# 2026-10-30: the runner stopped frank-archon, the rig container was still loading,
# and /health answered 200 from the production process draining its listener. The
# window logged `health ok waited=0s`, the pusher refused on the container name, and
# a 12M window died at second zero. The gate must prove the LISTENER is in the rig.
tap_window_fixture() {
export W="$BATS_TEST_TMPDIR/window"
mkdir -p "$W/scripts" "$W/campaign/calib" "$W/campaign/kld-ref" "$W/models/rad" "$W/corpora" "$W/bin"
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-calib-window.sh" "$W/scripts/"
cp "$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh" "$W/scripts/"
echo "{}" > "$W/campaign/kld-ref/kld.json"
: > "$W/models/rad/qwen38-bf16.rad"
echo "{\"p\":1}" > "$W/corpora/calib.jsonl"
# a gram so the empty-tap check passes on the positive path
: > "$W/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
cat > "$W/scripts/run-qwen38-bf16.sh" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_RIG_MARKER:-}" ] && : > "$STUB_RIG_MARKER"
sleep "${STUB_RIG_SLEEP:-3}"
exit 0
EOS
cat > "$W/bin/py" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_PUSH_SENTINEL:-}" ] && : > "$STUB_PUSH_SENTINEL"
exit 0
EOS
cat > "$W/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "${1:-}" in
inspect) echo true ;;
# the PREFLIGHT asks who owns :7770 (production, per LISTEN_PID); the WINDOW asks which
# pids belong to the rig (RIG_PID). Both stubs are the same binary, so keep them apart.
image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
top) echo PID
# the WINDOW asks "which pids are the rig's", the PREFLIGHT asks "which are
# production's". One stub, two answers: that separation IS the identity lesson.
case "${2:-}" in
qwen38-bf16) echo "${RIG_PID:-424242}" ;;
*) echo "${LISTEN_PID:-424242}" ;;
esac ;;
*) : ;;
esac
EOS
printf '#!/usr/bin/env bash\nexit 0\n' > "$W/bin/fuser"
cat > "$W/bin/ss" <<'EOS'
#!/usr/bin/env bash
echo "LISTEN 0 4096 *:7770 *:* users:((\"radiance\",pid=${LISTEN_PID:-424242},fd=9))"
EOS
cat > "$W/bin/curl" <<'EOS'
#!/usr/bin/env bash
exit 0
EOS
chmod +x "$W/scripts/run-qwen38-bf16.sh" "$W/bin/py" "$W/bin/podman" "$W/bin/ss" "$W/bin/curl" "$W/bin/fuser"
}
tap_window_env() {
env PATH="$W/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$W/campaign" \
FRANK_QWEN38_RAD="$W/models/rad/qwen38-bf16.rad" \
Q38_WINDOW_CALIB_CORPUS="$W/corpora/calib.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 \
Q38_WINDOW_HEALTH_TIMEOUT=1 \
PY="$W/bin/py" \
STUB_RIG_MARKER="$W/rig.marker" \
STUB_RIG_SLEEP=0 \
STUB_PUSH_SENTINEL="$W/push.ran" \
bash "$W/scripts/qwen38-calib-window.sh"
}
@test "a health answer from a listener outside the rig is not the rig: no push, window fails" {
tap_window_fixture
# health 200, container "running", but the listener pid belongs to production
export LISTEN_PID=999999 RIG_PID=424242
run tap_window_env
[ "$status" -ne 0 ]
[ -z "$(ls "$W/push.ran" 2>/dev/null)" ]
[[ "$output" == *"before it ever answered /health"* ]]
}
@test "when the listener is inside the rig the window proceeds and banks the tap" {
tap_window_fixture
export LISTEN_PID=424242 RIG_PID=424242
run tap_window_env
[ "$status" -eq 0 ]
[ -e "$W/push.ran" ]
[[ "$output" == *"TAP BANKED"* ]]
}
# ---- the preflight: every fact by IDENTITY, and it must fail loudly -------------------------
# Each case here is a lie that cost a real window on 2026-10-30.
preflight_fixture() {
export PF="$BATS_TEST_TMPDIR/pf"
mkdir -p "$PF/campaign/calib" "$PF/campaign/kld-ref" "$PF/bin"
echo '{}' > "$PF/campaign/kld-ref/kld.json"
echo '{"p":1}' > "$PF/corpus.jsonl"
cat > "$PF/bin/podman" <<'EOS'
#!/usr/bin/env bash
case "$1" in
image) # an image carries every digest it was pulled under; the pin is one of them
printf '%s\n' "docker.io/stilldeadcode/radiance@sha256:4beb81eea874e3e308fbffbe1a9402a2f845c19e68ea612ba949b43bd5cb7cb0"
printf '%s\n' "docker.io/stilldeadcode/radiance@${STUB_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}" ;;
top) echo HPID; echo "${STUB_PROD_PIDS:-4242}" ;;
esac
EOS
cat > "$PF/bin/ss" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_LISTENER_PID:-}" ] && echo "LISTEN 0 4096 *:7770 *:* users:((\"x\",pid=${STUB_LISTENER_PID},fd=9))"
exit 0
EOS
cat > "$PF/bin/fuser" <<'EOS'
#!/usr/bin/env bash
[ -n "${STUB_KFD_PID:-}" ] && echo "${STUB_KFD_PID}"
exit 0
EOS
chmod +x "$PF/bin/podman" "$PF/bin/ss" "$PF/bin/fuser"
}
run_preflight() {
( cd "$BATS_TEST_DIRNAME/.." && env PATH="$PF/bin:$PATH" \
FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" \
Q38_CALIB_MIN_FREE_MIB=1 \
bash scripts/qwen38-preflight.sh "$@" )
}
@test "preflight passes when every fact is true, and says so" {
preflight_fixture
run run_preflight
[ "$status" -eq 0 ]
[[ "$output" == *"PREFLIGHT PASSED"* ]]
[[ "$output" == *"carries sha256:69add58f"* ]]
}
@test "preflight accepts the service's own process on the port, not just any answer" {
preflight_fixture
export STUB_PROD_PIDS=4242 STUB_LISTENER_PID=4242 STUB_KFD_PID=4242
run run_preflight
[ "$status" -eq 0 ]
[[ "$output" == *"held by frank-archon (host pid 4242)"* ]]
}
@test "preflight fails when the tag resolves to a different digest" {
preflight_fixture
export STUB_IMAGE_DIGEST="sha256:deadbeefdeadbeef"
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"does not carry sha256:69add58f"* ]]
}
@test "preflight fails when :7770 belongs to a host pid the service does not own" {
preflight_fixture
export STUB_LISTENER_PID=999999
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *":7770 is held by host pid 999999"* ]]
}
@test "preflight fails when another process holds the GPU" {
preflight_fixture
export STUB_KFD_PID=999998
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"foreign holders:"* ]]
}
@test "preflight fails when the tap dir already holds grams from another run" {
preflight_fixture
: > "$PF/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
run run_preflight
[ "$status" -eq 2 ]
[[ "$output" == *"rotate them to .AGENTTRASH"* ]]
run run_preflight --allow-existing-tap
[ "$status" -eq 0 ]
}
@test "preflight fails when the launcher is not in the repo root" {
preflight_fixture
run env PATH="$PF/bin:$PATH" FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" Q38_CALIB_MIN_FREE_MIB=1 \
bash -c "cd /tmp && bash '$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh'"
[ "$status" -eq 2 ]
[[ "$output" == *"not the repo root"* ]]
}
+5 -2
View File
@@ -29,7 +29,7 @@ RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
# The reference shape, spelled exactly as run-qwen38-bf16.sh must pass it: this
# is the contract the recorded reference is measured at, so a silent edit here
# is a silent edit to every KLD number in the campaign.
SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 66560 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja"
SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja"
# oneline — both dry runs print one shell-quoted argument a line; joining them
# lets an option and its value be asserted as the pair they are.
@@ -157,12 +157,15 @@ flag_value() {
[[ "$output" != *"--vram-kv-mib"* ]]
}
@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
@test "the calib dry run opens the tap at the 12M floor, on the latest pin, and serves on 7770" {
run bash "$RUNNER" calib --dry-run
[ "$status" -eq 0 ]
flat="$(oneline)"
[[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
[[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
# the tap producer is the campaign's own pin (latest radiance), not production's
[[ "$flat" == *"radiance@sha256:69add58f"* ]]
[[ "$flat" != *"radiance@sha256:d590e300"* ]]
[[ "$flat" == *"--port 7770"* ]]
[[ "$flat" == *":/data/calib "* ]]
# calib serves; it does not record