Compare commits

...
Author SHA1 Message Date
Frank 7aa32d1b64 feat(media): the operator's map gives a folder the id its name cannot reach
Tvdb titles a record in the show's own language and keeps the English name in
aliases the name search never reads, so the rule that accepts only tvdb's own
spelling of a query can never resolve them: ブラッククローバー (331753) holds no
alias at all, and the bare alias "One Piece" belongs to 392276, the 2023 live
action, not to the 1999 record (81797) that 1,072-file pack is. Seventeen pack
folders on the array — 3,507 files — are in that class.

media-shows.py now reads scripts/media-shows-tvdb-ids.tsv by default: one line
per pack folder, folder <TAB> tvdb id <TAB> note, every id read from tvdb's own
record. The map outranks a provider id, because a box set's provider id is the
server's own guess at a folder name, and it merges the release folders that are
one show. Everything it touched goes to the exceptions: every folder it named,
every row that matched no folder, every disagreement with a provider id, every id
tvdb never answered. The file is read fail-closed — an id that is not a number,
or one folder named twice, stops the plan, because a typo there renames a pack of
1,268 files to the wrong show.

scripts/media-shows-tvdb.py is the sheet the map was built from, promoted out of
/tmp so the next run is reproducible: it reads a plan's mkdir rows, asks tvdb
about every destination folder carrying no [tvdbid-], and grades the answer
EXACT / PUNCT / NEAR / NONE by how much was guessed to reach it. Acceptance is
unchanged from the passes it replaces — bare-form equality plus the year-span
guard — and that gate is what rejected three confident picks: Fairy Tail's 2024
sequel (410031), the 2026 Hokuto no Ken remake (465216) and Mugyutto! Black
Clover (378762) offered for Black Clover. The sheet is for the operator; the map
is what the planner reads, and only after he approves the rows.

Tests: seven for the map's behaviour — rename to the record it names, two release
folders becoming one series, precedence over a provider id and the line stating
the disagreement, an id tvdb never answered, a row whose folder is gone, a
malformed id refusing the run, a folder named twice refusing the run — plus one
that the shipped file parses and never stops a plan.
2026-10-31 13:10:32 -07:00
dank 576c3421b8 Merge pull request 'feat(media): the folder carries its tvdb id; specials and year-seasons resolve' (#105) from feat/shows-ids-and-specials into main 2026-10-31 12:56:35 -07:00
Frank e4656cfd7a fix(media): a tvdb episode page is not the episode list
Tvdb's default episode query answers at most 500 rows and states no total.
One Piece (1999), Crayon Shin-chan (1992) and Detective Conan (1996) each
answer exactly 500 while holding over a thousand, so the contradiction rule
added in the previous commit read every long-running show as contradicted:
the pack books more episodes than "the authority holds", and the rule that
exists to stop a wrong anime title would have stripped the right one from
every file of the season.

A capped list now records that it is capped (`tvdb:capped:<id>`, so a warm
cache answers the same question a cold one did) and contradicts nothing;
titles for the episodes the page does hold still ride.

Test: a 500-row page with a 501-episode pack keeps ` - Episode 1` and lists
no contradiction. Red before the change, proved in a copy of the tree with
the cap raised past the fixture.
2026-10-31 12:40:49 -07:00
Frank acab4cef22 feat(media): the folder carries its tvdb id; specials and year-seasons resolve
Three defects in the TVShows planner, all three visible in the applied plan:

- A series folder named a show by words alone. Jellyfin's Tvdb agent and
  Plex's scanner both read `[tvdbid-NNNN]`, so the resolved id now rides the
  series folder — and only the folder: a file name gets 180 characters and
  the episode title needs the room.
- 1,412 files parked because their provider row said `season 0`. Season 0 is
  a legal season; Jellyfin's contract is `Season 00` for specials.
- 1,033 files parked because Jellyfin's date matcher writes the AIRDATE YEAR
  into the season column. Such a row is now renumbered by the title tvdb
  states for the series' id, read backwards off its own episode list. A
  title tvdb gives to two episodes names no episode, and that file parks
  with the reason it always carried.

A season tvdb's own episode list CONTRADICTS — a release pack books more
episodes there than the authority holds, the anime numbering hazard — keeps
its rename and loses the tvdb title, because the slot its number points at
is a different episode. Those series are listed in the exceptions file,
which is where the operator reads them.

Tests: six through the fixture (five red before the change; the ambiguous
title is a guard that the recovery must never guess). Full bats suite green.
2026-10-31 12:27:15 -07:00
dank 04802b3e48 Merge pull request 'fix(media): verify read the library the way the servers write it' (#103) from fix/media-verify-wal-read into main 2026-10-31 12:12:23 -07:00
dank 44e55ad172 Merge pull request 'feat(media): tvdb names the show — titled episode files, id-backed series folders, park outside' (#102) from feat/tv-tvdb-naming into main 2026-10-31 12:12:15 -07:00
dank 132805a25a Merge pull request 'fix(dankserver): mount /mnt/entropy from its LUKS mapper at boot' (#104) from fix/dankserver-entropy-automount into main 2026-10-31 12:04:08 -07:00
Frank 2406641bc6 fix(dankserver): mount /mnt/entropy from its LUKS mapper at boot
The crypttab entry named the md RAID array UUID (61bd6eaf-...) where a luks
entry takes the LUKS container UUID (cd86c728-...), so
systemd-cryptsetup@entropy.service could never produce /dev/mapper/entropy and
the fstab entry could not resolve either; the volume only appeared when a
desktop session let udisks2 unlock it, which kept the broken boot path hidden
for a year. The keyfile was wrong as well: 27 bytes, i.e. the 26-character
passphrase plus a trailing newline, and cryptsetup hashes the file whole.

Records the host config as dankserver/host/{crypttab,fstab} (the dankclaw/host
precedent), rewrites the Entropy Volume page to own the chain, the UUID law and
the keyfile contract, and comments out the now-unmountable /dev/md0 line.

Proven on the host: /etc/{crypttab,fstab} byte-identical to the repo copies,
reboot into 7.0.0-42-generic mounts /mnt/entropy from /dev/mapper/entropy, all
12 containers returned, frank-archon back to 141 t/s (was 6.7 t/s on a wedged
amdgpu queue). ./tests/run.sh: ALL TESTS PASSED (749).
2026-10-31 11:56:39 -07:00
Frank dbcf8a2c35 docs(media): the verifier prints the four verdict classes itself
The taxonomy was written as something an operator assembled by hand; the tool now
emits `on disk at dst` / `REAL` / `gone` lines, which is the difference between
reading `missing from the DB: 1851` as 1851 lost albums and reading it correctly.

cwd: /home/blank/.omp/wt/wt-20261030-163847-6d1fc1f · session: wt-20261030-163847-6d1fc1f
2026-10-31 10:29:33 -07:00
Frank a5a9c04da2 feat(media): the verdict splits what the database alone cannot
"No row at the destination" was one number, and it conflated three different
things: a sidecar the server never indexes, a file a later pass renamed again,
and a file that never left its source — the last one being actual damage. The
runbook already named the four classes; the tool never computed them, so an
operator reading `missing from the DB: 1851` had no way to tell a booklet.pdf
from a lost album and might have run a rollback over a false alarm.

`missing` rows are now stat'd on both paths and reported as
`on disk at dst, no row names it` / `REAL — file still at src` /
`gone — at neither path`, with offenders listed in that order. A row still
naming `src` stays a separate line: that is the server's own stale pointer, a
phantom, and it costs a re-scan rather than a restore.

Two new tests (REAL, and gone-vs-sidecar) plus two existing layout assertions
loosened from line positions to content. 14/14 in tests/media_verify.bats.
2026-10-31 10:28:44 -07:00
Frank c95ae20bfb Merge main into feat/tv-tvdb-naming: the arms moved to their own pages
main split the music and audiobook arms out of Media Relayout.md and put a
locator in their place, so the branch's trims inside those sections now live on
the two child pages, and the parent keeps main's locator. The two Jellyfin laws
the branch moved to Jellyfin.md stay moved; the sweep law keeps the branch's
wording ("until the next sweep"), which is the one the next line depends on.

Full suite: 758 tests, all green.
2026-10-31 10:22:34 -07:00
Frank a0dfa61623 docs(media): the TVShows id-map proposal — where it lives and what may be applied unreviewed 2026-10-31 10:18:16 -07:00
Frank 4b67c3c4d8 fix(media): the census read Plex the same wrong way
The two Plex arms of media-census.py — `plex` (the movie planner's title
authority) and `audiobooks` (the chapter rows the audiobook plan files by) —
opened the library `mode=ro&immutable=1` while the Jellyfin arms already used
plain `mode=ro`. Same defect as media-verify.py: an uncheckpointed row of a
running scan is invisible, so a just-scanned item is simply absent from the
authority file a plan is built from, and the pass cannot say why.

Both now go through open_plex_ro(), which tries plain ro first and keeps
immutable only as the fallback for a database whose -shm cannot be read. The
new audiobooks test leaves a chapter row committed-but-uncheckpointed and
asserts the export still carries it; red before the change, green after.
2026-10-31 10:13:17 -07:00
Frank 7ee831adee docs(media): what a verification verdict actually means, and what a rollback can undo
Three things the 2026-10-31 five-library relayout taught, written where the
next operator reads them:

- A verify run must open the servers' databases plain `mode=ro`. Both keep
  their libraries in WAL mode; `immutable=1` skips the write-ahead log, so a
  scanner's own committed rows read as missing and the read can fail as
  "database disk image is malformed". The census page claimed immutable — now
  it says what the code does.
- "No row at the destination" splits into four classes, and only REAL (no row
  at dst AND the file still at src) is the relayout's own damage. Phantom rows
  naming a pre-move path are normal behaviour for both servers. And the owning
  server decides whether a pass can convict: Jellyfin holds no audiobook
  library, so that journal verifies against Plex or it proves nothing.
- A chain of passes rolls back newest journal first, or the tool refuses; the
  refusals measured that day are the tool working, not failing.

The music and audiobook arms moved to their own pages (progressive disclosure;
the parent drops 175 lines and stays the owner of the shared gates, the
executor, the journal law and the rollback). Stale per-run counts destroyed:
episode counts, byte sizes, move tallies for a pass that already happened.
2026-10-31 10:13:11 -07:00
Frank afad21dda5 fix(media): verify read the library the way the servers write it
A verify run beside a running scan reported rows the scanner had just
written as MISSING, and could fail outright as "database disk image is
malformed". Both libraries live in WAL mode, and the verifier opened them
with immutable=1, which tells sqlite the file never changes: it skips the
write-ahead log, so every committed-but-uncheckpointed row is invisible.
134 MB of jellyfin.db-wal was riding past the verifier's eyes during the
2026-10-31 rescan.

Open plain mode=ro first — sqlite then reads the WAL through its -shm —
and keep immutable=1 only as the fallback for a database whose -shm
cannot be read. The bats fixture leaves a journaled move uncheckpointed
and proves the verifier still counts it landed.
2026-10-31 10:00:11 -07:00
Frank f430e2264d docs(plex): the scan recipe kept its own promise about the token
The block said the token rides stdin and never argv, then handed it to curl as
`-H "X-Plex-Token: $T"` — world-readable in /proc/<pid>/cmdline. It now writes
a mode-600 curl config inside the container, which is also how the overnight
rescan was actually issued (five sections, all accepted). Two truths the old
block got wrong are stated: a section answers 200 with a Directory body when
Accept: application/json is sent, and a backslash-newline inside a quoted
printf format is literal, so the config builder writes two printfs.
2026-10-31 09:50:22 -07:00
Frank 82a2183d40 docs(media): a film handoff needs the file's own words and year
The provider's first hit for a salvaged query is a near miss, and the page said
only that a provider must name the file. The bullet now states the two checks and
the two real misfiles they exist to stop.
2026-10-31 07:39:45 -07:00
Frank c70f9b3f6d fix(media): a film leaves the TV library only for a film it actually names
The first tvdb-grade plan handed 12 files to the movie library, and two of them
were wrong in a way the exceptions file could not show: an episode of the
documentary series `Russia 1985-1999: TraumaZone` was renamed to the 1963 Bond
film `From Russia with Love`, and `Gundam 00 Special Edition III - Return The
World` became `Special Edition I - Celestial Being`.

Two guards, each alone insufficient:

- `title_of` salvages a short query from a long release name ("russia", "mobile
  suit gundam 00"), and tmdb() took the FIRST search result. The call site then
  compared the provider's words against that salvaged query, so any title merely
  CONTAINING the query passed. The provider's title must now be a run of words the
  FILE itself states: `Special Edition I: Celestial Being` is not in a filename
  that says `Special Edition III - Return The World`.
- tmdb() read the year from `year`/`first_air_date`, which the MOVIE search does
  not return — it returns `release_date`. Every film answer therefore arrived
  year-less and the year guard at the call site could never fire. That is how a
  1963 answer survived a filename stating 1991.

Tests: a provider answer the file does not name parks, and nothing lands in the
movie library; an answer from another year is refused. The year-from-the-wire half
is proven by the live re-plan, which now dates every movie answer.
2026-10-31 07:38:54 -07:00
Frank 32b8901bff docs(media): the pile is matched with its underscore stripped
The server drops the leading underscore from the hidden folder's display name, so
the park-echo rule that compares spelled names lets the review pile survive as a
visible series. The law now says which spelling is compared; the mechanism stays
with the guard's docstring.
2026-10-31 07:28:22 -07:00
Frank 6d03bbc8e7 fix(media): the review pile is not a series, and a folder carries the year once
Two destination defects the first tvdb-grade plan exposed in its own output —
both visible by reading the plan's folder names, neither caught by any gate.

1. Jellyfin sanitizes a hidden folder's display name: `_unsorted` comes back as
   `unsorted`. The park-echo guard compared the row's series against the folder
   with the underscore, so 1,091 pile rows survived as a real series and the plan
   renamed 617 files out of the pile into a VISIBLE folder named `unsorted` —
   inventing the junk show the guard exists to prevent. The comparison now strips
   leading underscores from both sides.

2. TVDb hands back some series with the year already on the name (`Acapulco
   (2021)`). label_of appended the year anyway, producing `Acapulco (2021)
   (2021)` — a folder no matcher and no human reaches for twice. A name that
   already ends in `(YYYY)` is left alone.

Both are covered by fixtures: a pile row whose series is the sanitized spelling
must park, never file under the pile's name; a provider name carrying its year
produces one year in the path and the plan contains no `(YYYY) (YYYY)` anywhere.
2026-10-31 07:27:47 -07:00
Frank 32b58dd787 fix(media): TMDb's v3 key travels as a query parameter; Bearer is a JWT only
The first tvdb-grade TV plan built 49,771 moves with not ONE TMDb answer: every
search and every external_ids call came back 401. The credential on this box is
the v3 key (~/.config/secrets/tmdb, `api_key=`), and TMDb takes a v3 key only as
the `api_key` query parameter — a v4 request token is the only thing the Bearer
header accepts. Measured on the same credential in one pass: the query form
resolves `New Girl (2011)`, the Bearer form raises HTTP 401.

tmdb_request() now picks the transport from the shape it was handed (three
dot-separated segments = a JWT → Bearer, anything else → query parameter), and
the old header comment's reasoning is corrected: a token in argv is
world-readable in /proc/<pid>/cmdline, but this URL is built inside the process
and never reaches argv, so the query form leaks nothing.

The failure mode that let it look like a finished plan is fixed with it. The
401s were stderr-only, so the summary printed a confident move count while every
series folder still carried the metadata provider's folder-echo name — the exact
junk this contract exists to undo. Provider health is now part of the summary
line and of the exceptions file, and a test covers it offline with
`env -u TMDB_API_KEY`: a plan that asked and could not ask says NOT
provider-complete in both places, and still writes the moves it can justify,
because the answers that did arrive are worth their rename.
2026-10-31 07:16:46 -07:00
Frank 0a365274b6 docs(agent): a PR comment posts to the issues endpoint, and an empty jq body is silent
Measured while commenting on PR 102: POST /pulls/102/comments answers 405 to a
plain body (it is the review-comment store); the conversation thread is
POST /issues/<PR-number>/comments, which answers 201 and returns an html_url on
the PR. Gitea maintains one issue index, so the PR number is the same number.

The trap that hid it: building the body with jq --rawfile b /dev/stdin inside a
command substitution feeds jq nothing (the substitution does not inherit the
heredoc), Gitea answers 422, and jq -r .id on the error prints nothing — a
silent no-op that looked like a posted comment. The body goes in a file.
2026-10-31 07:11:13 -07:00
Frank e531e90278 docs(media): a scan request belongs to the server's own page
Media Relayout.md hit the vault's 500-line gate the moment the tvdb contract
landed, and the fat in it was not the new text: the Plex and Jellyfin scan
blocks (35 lines) sat inside the relayout procedure even though they are the
way you talk to those servers after ANY change, and two Laws bullets repeated
the same facts a third time. Both blocks now live on [[Plex]] and
[[Jellyfin]]; the relayout page keeps the two laws an operator needs at that
moment — 204 is acceptance and anything else queued nothing, and one accepted
refresh is the whole procedure, because a second moves `Scan Media Library` to
`Cancelling` and every later 204 is discarded.

Same pass, the dead numbers the vault's own law says to destroy: an applied
music pass's date, an applied audiobook pass's move/park counts, and the 146
sweep stragglers of one specific first sweep. The laws they taught stay.

490 lines, `bats tests/docs_completeness.bats` 6/6 (length violation gone,
link integrity clean).
2026-10-31 07:10:28 -07:00
Frank c3ab565d9b feat(media): tvdb names the show — episode titles, id-backed folders, park outside
The TV relayout that landed copied Jellyfin's Series names verbatim, and the
naming census showed what that launders in: one Series item per release folder
(1,107 items for 1,107 top-level dirs), the same show spelled six ways, and
box sets living in 346 separate folders for 821 identities. Jellyfin names a
Series after the folder it scanned, so `24 (2001) Season 1` is a folder echo,
not a show name.

The name authority is now tvdb, reached through the id the server matched:

  * media-census.py jellyfin emits two more columns — `title` (the episode row's
    own Name) and `tvdb` (the series id, read from BaseItemProviders where
    Jellyfin 12 keeps provider ids and from the row's Data blob where older
    builds do, so the census does not care which shape the live DB has);
  * media-shows.py resolves each series to tvdb — the row's tvdb id, else its
    TMDb id through TMDb's external ids, else an exact tvdb name search — and
    writes tvdb's name and year. That also collapses the provider's spellings
    into ONE identity, which is what turns a box set split over five release
    folders into one folder and what makes two files claiming one episode MEET
    in the duplicate rule;
  * the episode file becomes `<Series> (<year>) - SxxEyy - <Title>`: the title
    comes from tvdb's episode list, else the row's own title, else nowhere — no
    title means no separator, never an invented one.

Two refusals come with it. A row whose series is the hidden top-level folder the
file itself sits in is NOT evidence: the review pile parked inside the library
was indexed as one show named `_unsorted`, and its rows would have renamed real
shows to it. And inside such a folder a file's own words may MATCH a registered
show but never INVENT one — invention is what filled the pile. `--park`
therefore belongs outside the library, which is also what makes the next plan
re-file the pile instead of skipping it.

Both providers answer offline-reproducibly: every answer is cached in one JSON
file, so a plan can be re-planned without a key. Lookups stay behind --tmdb and
--tvdb, and the tvdb token is read from a mode-600 file, never argv.

tests/media_shows.bats is 23 tests, none of which touch the network — the
fixture Jellyfin DB carries the providers table and the name cache is an input
file. ./tests/run.sh: 753 green.
2026-10-31 07:01:50 -07:00
Frank 5dbbfcb8a0 docs(media): an accepted refresh can still be dropped by a wedged task runner
Tonight's sequence, measured: the post-sweep refresh answered 204 while
`Scan Media Library` sat in `Cancelling`, and Jellyfin's stale-path counts froze
for an hour at 0 % CPU — the request was accepted and discarded. Restarting the
user unit cleared the runner; one POST after that reported State `Running` at
130 % CPU and the counts began to move. So the status code is not the evidence,
and the restart is `systemctl --user restart jellyfin.service`, never
`deploy.sh`, which would run the never-tested metadata-view cutover.
2026-10-31 04:13:12 -07:00
Frank 02a568fb97 docs(media): the Jellyfin scan 401 was curl-config quoting, not the server
Corrects the lesson written an hour ago, which said a non-204 was transient and
should simply be re-issued. It is reproducible and mechanical: `printf` with
`\"` inside the single-quoted format string drops the backslash, curl then ends
the quoted value at the first bare quote, and Jellyfin receives
`Authorization: MediaBrowser Token=` with an empty key. Both forms were run
against the host in one pass — `\\"` answers 204, `\"` answers 401 — so the
block's double backslashes are load-bearing, and the handoff script that deviated
left a 58k-move library unscanned for 40 minutes.

Also the trap that diagnosis walked into: a second accepted refresh while a scan
is running moves `Scan Media Library` to `Cancelling` and the pass starts over.
One accepted request is the procedure; wait it out.
2026-10-31 03:41:22 -07:00
Frank 1e1527d101 docs(media): the SSD metadata view has never been deployed on Dankserver
The vault described the overlay as the live shape. It is not: `media-view.sh
status` answers NOT MOUNTED, /var/lib/media-meta does not exist, and the
installed quadlets under ~/.config/containers/systemd still carry the pre-view
binds (/mnt/entropy:/data:ro for Jellyfin, /mnt/entropy:/mnt/entropy:ro for
Plex) while git binds the view with ConditionPathIsMountPoint. The deploy gate
(deploy-remote.sh:ensure_media_view) has therefore never run: the first
`deploy.sh dankserver --service jellyfin|plex` is an untested cutover of both
media servers, and Jellyfin's read-only sidecar error is today's expected state,
not a fault.

Same commit, the scan-acceptance law from tonight's TV handoff: it logged
`jellyfin scan: 401`, so Jellyfin never rescanned a 58k-move library until the
request was re-issued with the identical key and header form (204). Anything
that is not 204 means no scan was queued — re-issue, do not diagnose.
2026-10-31 03:13:53 -07:00
Frank 9eb85ea1f9 fix(media): order the sidecar reap only where the view is mounted
Every apply and rollback printed "next: scripts/media-view.sh reap", but on
Dankserver there is nothing to reap: the metadata view has never been installed
(`media-view.sh status` -> NOT MOUNTED, `/var/lib/media-meta` absent) and the
live quadlets still bind the array read-only. All three relayout arms tonight
issued that order — 242,084 moves — and it pointed at a directory that does not
exist. Where no overlay is mounted the servers bind the array itself, so the
sidecars moved with the media.

`--metadata-view` (default /var/lib/media-meta/view) names the overlay and the
note now rides only when that path is a mountpoint. Red first: two new cases in
tests/media_relayout.bats (apply and rollback); the shared helper names a real
mountpoint so the pre-existing presence assertion keeps its meaning. 37 ok.
2026-10-31 03:13:53 -07:00
dank 96944f0b17 feat(media): media-verify.py replays a journal against the servers' own tables
A scan answers 204 whether or not it found anything, so the fleet's documented
proof — every move row has a row at its dst and zero rows still name its src —
was a hand-written sqlite query every time anyone needed it. That is the step a
half-finished relayout is judged on, so it is now a command with an exit code.

It encodes the two doors that bite: Jellyfin stores the container-side path
(/data stands for the share) while Plex stores the host path verbatim and hides
deleted scans behind deleted_at on two tables. The report breaks the missing rows
down by extension, because a .cue or .log has no row to land in and must not read
as a lost album; --ignore-ext keeps those out of the verdict.
2026-10-31 01:02:48 -07:00
dank 33f97cd3b3 docs(media): the TV library arm gets its own journal, and its plan lives on the array
Two relayouts writing one journal cannot be rolled back apart: the mover
(Movies -> TVShows) and the library re-filer both pointed at
media-relayout-tv.jsonl. The re-filer now writes media-relayout-shows.jsonl,
and its plan/exceptions move off /tmp, which a reboot eats and which is not a
record.
2026-10-31 00:55:32 -07:00
dank 254fb41fd0 docs(media): the music relayout is mid-flight, and the array takes one writer
The page said the music plan "was applied" while 62k of its 131k journal rows
were still being written; a runbook line that gets believed is worse than none.
Also lands the single-writer law the audiobook handoff exists to honour, and the
durable plan path (/tmp is not a record).
2026-10-31 00:40:30 -07:00
dank 3613f69593 Merge pull request 'feat(media): the audiobook relayout, filed by the book Plex matched' (#101) from feat/media-audiobook-relayout into main 2026-10-31 00:38:02 -07:00
dank 6bf70c6a64 fix(media): a shelf needs a name, so a mistagged chapter number parks
Plex's artist row for 272 chapters is a chapter number the ripper put in the
artist tag (0.4, 31, 05), which would have filed them under a shelf called 0.4.
A narrator is a person, so it must contain a letter; the same test is deliberately
not applied to a book title, which may honestly be 1984.
2026-10-31 00:34:40 -07:00
dank f1b8b60817 fix(media): a tag-scan placeholder is not a book identity
The first real plan filed 1,106 chapters from 14 unrelated top-level folders
onto one shelf, because all 1,106 carry the album tag [Unknown Album] and the
same artist tag. Plex's album title is the authority for the shelf only when it
names a book; a placeholder names nothing, and unrelated books share it. Those
chapters now park with the path they have today, timestamp-glued variants
included; Various Artists stays a legitimate multi-cast narrator.
2026-10-31 00:31:30 -07:00
dank 4ae2bde9ab docs(media): fit the audiobook section inside the OKF caps, refresh the index
The linter caps a page at 500 lines and a description at 120 chars: the section
lost four lines of prose (no law went with them) and the description lost its
restatement of the format names. gen-vault-index.py regenerated the locator row.
2026-10-31 00:25:58 -07:00
dank ceb5617a42 docs(media): the audiobook arm, and the music plan's applied state
One page owns the relayout facts: the new section carries the Plex-only reader,
the shape, the four commands, and the two laws a future operator would otherwise
re-litigate (never renumber a chapter; the shelf is the narrator). The music
section said its plan was NOT applied — it was applied on 2026-10-31, and a stale
line in a runbook gets believed, so it now names the journal and the rollback.
2026-10-31 00:24:50 -07:00
dank a7ceda95fd feat(media): the audiobook relayout, filed by the book Plex matched
The audiobook tree is the one library Jellyfin does not read at all and Plex
reads completely: 58,975 chapters matched into Artist/Album/Track, where the
album is the BOOK and the book's single artist row is the NARRATOR. That is the
authority the new planner files by, never the collection folder a pack made.

  Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>

Two laws are the opposite of the music tool's, both forced by the rows: files
are never renumbered (72% of the books carry chapters with no usable index, and
playback order comes from the embedded track tag either way, so an invented
number can only lie), and the shelf is the narrator, not the author (every book
has exactly one artist row; the author survives in 10% of album titles and in no
column Plex stores, so one rule beats a shelf half authors and half narrators).

media-census.py gains the audiobooks export: the chapters Plex matched under a
root, host paths straight through, nothing outside the root and nothing Plex
deleted. 22 tests in tests/media_audiobooks.bats pin the export's join and the
plan's output.
2026-10-31 00:24:45 -07:00
dank a32361ee5e Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-31 00:10:58 -07:00
dank ce6b3b1d3b docs(media): how to prove a relayout landed, read from both servers' tables
A scan answers 204 whether or not it found anything, so proof is the journal
replayed against Jellyfin's BaseItems.Path and Plex's media_parts.file: every
move has a row at its dst, no row names its src. Records the four wrong doors:
no sqlite3 binary on the hosts or in either container, Jellyfin's single
BaseItems table (no MediaItems), its container-side /data path prefix, and
Plex's HTTPD closing on ?path= / on /search matching titles not paths.
2026-10-31 00:10:50 -07:00
dank 382e3eb0b8 Merge pull request 'feat(media): the music library's own relayout, filed by its tags' (#100) from feat/jellyfin-provider-keys into main 2026-10-31 00:04:41 -07:00
Frank 54a393edab Merge remote-tracking branch 'origin/main' into feat/jellyfin-provider-keys 2026-10-30 23:58:34 -07:00
Frank a5459b0b1e docs(media): the music library's own relayout, with the laws it earned 2026-10-30 23:58:34 -07:00
Frank b03931f7ce feat(media): a music relayout that files by the tag row Jellyfin matched
The music library is rips and discography packs whose filenames say nothing the
matcher needs. The authority is Jellyfin's own audio rows: one per track, with
the album and album-artist ids it carries in BaseItemProviders (the item JSON
holds an empty ProviderIds for albums in this build).

Grouping is what the tags assert, not where files sit: (album-artist tag, album,
year), case-folded. A compilation is one album whose tracks name many performers,
so it files once under Various Artists; two album-artist tags on one album name
are two albums. The folder spelling is the dominant one, because a name taken
from whichever row sorted first depends on row order and re-names albums that are
already filed.

An occupied path is never a destination: the executor refuses occupied targets,
so a plan that aims at one fails after the renames meant to clear it. A file
standing on its own target, or already carrying a collision suffix, is a keep -
otherwise the plan shuffles instead of being a fixed point.
2026-10-30 23:58:34 -07:00
dank 82f6ec55c2 Merge pull request 'feat(media): plan a TV relayout from the numbering Jellyfin already has' (#99) from feat/media-tv-relayout into main 2026-10-30 23:38:36 -07:00
Frank 536949c18d docs(media): the TV library's own relayout, with the evidence chain it believes
One page owns relayout: the census commands, the order the planner believes
things in, the two refusals a dry run on the real library proved necessary, the
apply and rollback path, and the standing wart that a provider name can itself
be scene junk ("Black Clover 001", 92 episodes) — renaming that is Jellyfin's
job, not the mover's.
2026-10-30 23:35:14 -07:00
Frank 337afac91e fix(media): a TV plan that follows the authorities it was given
A dry run of the TV planner against the real 58k-file library produced a root
series called "01 NCOP" out of a bonus subfolder, copied Jellyfin's stale
numbering (every file under SNL 16-20/Season 16..19 carries season 16 episode
20; one Gundam row says season 2004), and lost subtitles a scene pack had
buried in Subs/<release-name>/.

- Folders now RESOLVE a series and never invent one: a folder's words match a
  registered name as a token run ("(CBB) Tengen Toppa Gurren Lagann - The
  Complete Series (BD 1080p)" names Tengen Toppa Gurren Lagann), and folder
  words alone name nothing, so an unknown pack's bonus folder parks instead of
  opening a new show at the library root.
- Numbering priority: the marker the file's own name states, then the Season NN
  folder plus the number the name leads with inside it, then the provider row.
  A row that still insists on season 1989 parks its file rather than creating
  Season 1989.
- The provider row keeps identity: the registered name and its year survive a
  file whose own marker numbers it, and the pack index a release leads with
  ("4a. Seed DESTINY") is folded out of the folder name.
- A sidecar searches every ancestor folder for the video it names, not just the
  parent, so Subs/<release-name>/<release>.jpn.srt follows its episode.
- The shared bonus-folder test no longer reads an index as a title: name_words
  drops "extras" as release junk, so "01. Extras" reduced to "01" and matched
  the "01" of "NCOP 01", cancelling the bonus evidence. Only words can name a
  film.
2026-10-30 23:29:34 -07:00
Frank c0c77590b1 fix(media): a sidecar named "episode.mkv.srt" anchors, and loses the container word
The first run against the real library died in the anchor lookup: a subtitle
whose own name carries the media extension matches BOTH prefix forms of its
video, and the "keep the longest" comparison read the destination slot of the
tuple instead of the length slot. The fixture only ever matched one prefix, so
the seam was green over a crash.

The tail the sidecar keeps is now stripped of that container word: Jellyfin and
Plex both read "Show S01E01.forced.srt", not "Show S01E01.mkv.forced.srt", and
an nfo still carries the media extension because that one is constructed, not
parsed.
2026-10-30 23:17:55 -07:00
Frank 441b908937 feat(media): plan a TV relayout from the numbering Jellyfin already has
The Movies relayout planned from Plex; the TV library has no Plex rows and
58k files whose names lie about which episode they hold. Jellyfin has already
matched and numbered 47,676 of them, so the planner treats that numbering as
the authority and the file name as the fallback: an episode number the name
states, or a provider row, or nothing - and "nothing" parks the file under
_unsorted with a reason instead of inventing a number.

  - media-census.py jellyfin: one TSV row per matched episode, its container
    path mapped back to the host, series name/year/TMDb id from the Series row
    (the episode row carries none of them). Both TSV writers now end lines with
    \n so a shell consumer sees the last field clean.
  - media-shows.py: the plan. Series folders take the provider's own spelling;
    one file claiming two episodes keeps both numbers; two files claiming one
    episode - the bigger keeps the standard name, the other parks; bonus
    material files under the series' extras folder, never as an episode; a long
    file inside the TV library leaves for Movies only when TMDb answers with a
    matching title, otherwise it parks. Re-running the plan on a library it
    already filed moves nothing.
  - tests/media_shows.bats: 15 behaviors, including the executor preflight and
    the re-plan stability law.
2026-10-30 23:15:32 -07:00
dank 1a1dbac268 Merge pull request 'Jellyfin metadata providers: keys ride a host file the deployer installs' (#98) from feat/jellyfin-provider-keys into main 2026-10-30 22:48:21 -07:00
Frank 949d3cb6f4 docs(jellyfin): one page owns the provider keys and where they live
The page carries the deploy-time shape: which four providers are core plugins
in Jellyfin 12, which one is not, the exact XML each key lives in, the host
file the keys come from and its 600 rule, the three-way pin on the TVDB
artifact with its rollback, and what the deployer does when the keys file is
absent. OMDb is recorded as deliberately unconfigured -- keyed by email,
1000 req/day, and a ToS that forbids storing what it returns. index.md's row
carries the same locator.
2026-10-30 22:47:49 -07:00
Frank 3ce8d62e70 feat(jellyfin): the provider keys ride a host file the deployer installs
Jellyfin 12 ships TMDB, TVDB, StudioImages and MusicBrainz as core plugins:
no DLL, only a configuration file under the data root, so the keys have to
reach that XML from somewhere that is not git. They live in the fleet's
per-service env-file convention -- /home/dank/Containers/jellyfin/provider-keys
(TMDB_API_KEY=, TVDB_API_KEY=, optional TVDB_PIN=, mode 600) -- and
scripts/jellyfin-providers.sh plan|install|verify reads only that file, prints
no key byte, and refuses a keys file that is not mode 600 or that has no line
for the provider about to be written. install is idempotent: it never touches
a key the UI already saved, backs up every XML it edits into
~/.AGENTTRASH/<date>-jellyfin-providers/ first, and only installs the TVDB
plugin when the pinned official zip carries the plugin guid.

The deployer runs install BEFORE it restarts jellyfin, because the plugin and
its keys are read at start-up and a restart-first deploy boots the old state.
A missing keys file warns instead of failing: a host without the keys still
deploys, degraded rather than broken. No plugin configuration XML is tracked.
2026-10-30 22:47:39 -07:00
dank 38737ab84e Merge pull request 'Media metadata view: Jellyfin writes, Plex reads, the array stays read-only' (#97) from feat/media-metadata-view into main 2026-10-30 22:18:27 -07:00
Frank 4437519430 chore(media): retire the legacy Jellyfin block from dankserver/compose.yaml
The container has had one owner since the Quadlet cutover
(dankserver/jellyfin/quadlet/jellyfin.container). The block was a second owner,
and wrong on every path: jellyfin/jellyfin:latest instead of the pinned ghcr
image, /opt/jellyfin/* data roots that no longer exist, and a read-only
/mnt/entropy:/data bind that the metadata view has since replaced.

The planner no longer counts jellyfin as a compose service, so editing the
inventory cannot restart a media server; tests/deployment_scoping.bats pins that
contract for both jellyfin and plex.
2026-10-30 22:17:04 -07:00
Frank 3b29e173c2 refactor(media): mount the view inside /var/lib/media-meta instead of under /mnt
The merged door was /mnt/media, which read like a third volume and split the host
state across two trees. It is now /var/lib/media-meta/view, so the array layer,
the SSD upper layer, the kernel workdir, the reap trash and the merged tree are
one directory — and the mount table gains no new /mnt entry beside the array.

systemd names a mount unit after its ESCAPED path, so the repo copy keeps the
readable name media-meta-view.mount while `install` writes the host file as
var-lib-media\x2dmeta-view.mount; `plan` prints both and the page records them, so
nobody hunts for a unit that "does not exist". install also creates the mount
point itself, because systemd will not.

The seam now additionally pins that the unit's Where= is exactly the path both
containers bind — guard and bind cannot drift apart. The Quadlet Lifecycle
phantom-path example is corrected too: it named an invented /mnt/media/*, while
the real phantom is the legacy /opt/jellyfin/* that dankserver/compose.yaml still
carries for a service whose roots moved to /home/dank/Containers/jellyfin.
2026-10-30 22:05:31 -07:00
Frank 9706f5d2de docs(media): the view moves sidecars into Plex's reach, it does not enable its NFO agent 2026-10-30 21:42:30 -07:00
Frank 206dc3baf1 docs(media): one page for the metadata view, and the three pages it re-points
New owner page Dankserver/Media Metadata View.md: the shape, the two binds and
why Plex keeps the /mnt/entropy container path, the commands, the laws (SSD-only
upper layer, never mount over an unmounted array, never delete through the view
because a whiteout only hides the array's copy, reap after any relayout, inotify
does not cross the merge), the failure signatures, and the rollback.

Jellyfin.md: /data is the view, so Local metadata works; the
"Read-only file system : '/data/…/movie.nfo'" line that used to be expected noise
is now the signature of a missing view. Plex.md: same bind, one fact changed.
Media Relayout.md: two new laws for the metadata that does not follow a rename.
AGENTS.md: runbook-index row plus a summary row. Root index regenerated.
2026-10-30 21:41:56 -07:00
Frank f5a3badb56 feat(media): the relayout executor names the reap its moves leave behind
A relayout renames on the raw array; the sidecars the media servers wrote live in
the NVMe upper layer of the view and stay at the old coordinates, which is how a
ghost folder (artwork and an .nfo, no video) is born. apply and rollback now
print the follow-up — scripts/media-view.sh reap — at the moment the operator
needs it, and the existing apply seam asserts the hint is in the output.
2026-10-30 21:41:56 -07:00
Frank 3e4b4f24f7 feat(media): share one metadata set between Jellyfin and Plex over an SSD overlay
Both media servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo /
episode.nfo) only when it sits next to the media file, and the library lives on
the RAID6 array, deliberately mounted read-only for every container and slow to
seek for a thousand tiny XML reads. So neither server binds the array any more:
both bind /mnt/media, an OverlayFS merge of the array (lowerdir, never written)
with /var/lib/media-meta on the NVMe (upperdir, where every write lands).

Jellyfin binds the view rw at /data and is the fleet's only metadata writer;
Plex binds it ro at /mnt/entropy, the container path its own database already
recorded, so only the host side of that bind moved and its library paths keep
resolving.

scripts/media-view.sh plans, installs, verifies, reports and reaps the view. It
refuses to mount over an unmounted array, refuses an upper layer on rotational
storage, and refuses a non-root install. deploy-remote.sh raises the view before
it restarts either media server, both units carry
ConditionPathIsMountPoint=/mnt/media so a missing view skips the start instead of
letting a scan clear the library database, and the planner routes the installer
and dankserver/host/** to the jellyfin+plex pair.

tests/media_overlay.bats (11 seams) pins the resolved shape, both refusals, the
two binds and their guard, the deployer's ordering, the planner mapping, and the
reap behaviour against a fixture (ghosts listed, then moved to the SSD trash --
files that exist only above are sidecars, not ghosts, so only directories are
candidates).
2026-10-30 21:41:51 -07:00
Frank 86f5903e74 docs(media): how to file a below-threshold episode by hand
Six single-episode files stayed in the movie library when the threshold refused
them; the hand path is the same executor on a hand-written plan and the same
journal. The trap recorded here is the series year -- never inherit it from the
title whose bonus folder an episode happens to be parked in.
2026-10-30 20:45:24 -07:00
Frank bd4ee78811 docs(media): clear the two OKF violations the last push shipped
The seam ran through `grep -c`, which returned success on a non-zero count and
let a red suite through: description over the 120-char cap, and a timestamp
that check-okf reads against the local date, so a UTC stamp a day ahead of
Seattle is 'in the future'. index.md regenerated for the new description.
docs_completeness 6/6, exit status checked this time.
2026-10-30 20:40:27 -07:00
Frank b7b39cdc92 docs(media): Jellyfin 12.4 needs the MediaBrowser header; the scan key is on the box
X-Emby-Token is refused for every key on 12.4 -- which is why this page
carried the false claim that no credential on the box could trigger a scan and
that the two Seerr keys were dead. The documented snippet now authenticates
with Authorization: MediaBrowser Token=..., reads 204 from /Library/Refresh,
and points at ~/.config/secrets/jellyfin (mirrored as JELLYFIN_API_KEY in the
ignored dankserver/.env). Law: prove a 401 against the ApiKeys table before
blaming the key.
2026-10-30 20:39:51 -07:00
Frank 09532e5216 docs(media): a deletion is staged on the item's own filesystem
Trashing 31 GiB of duplicate Monk to ~/.AGENTTRASH would have been a
cross-device copy of the thing being deleted, into 187 GiB of free space, on a
volume at 100% -- and rename(2) refuses it regardless. The stage belongs beside
the item.
2026-10-30 20:37:27 -07:00
Frank e1786e7cf2 docs(media): the Monk duplicate is verified; compare libraries with the series engine
The destination-hold refusal's evidence, stated exactly: 124 episodes both
sides, same episode list, every byte count identical, 31.5 GiB each way,
head+tail blake2b identical on a 6-episode sample.

Plus the law this came out of: a path-substring filter for a series name also
returns other shows (monk matches Monkey Dust), and whichever file a walk
reaches first standing in for an episode turns two libraries into a fake
duplicate — the same sloppiness aimed the other way deletes a unique file.
ep_of on both sides, and a hash before anything is trashed.
2026-10-30 20:33:41 -07:00
dank 409a8d8b0c Merge pull request 'feat(media): a TV mover that will not file a show the library already has' (#96) from feat/media-tv-mover into main 2026-10-30 20:31:38 -07:00
Frank e83162a2ac feat(media): a TV mover that will not file a show the library already has
Two fixes found by one live sweep, shipped together because the second is
what makes the first finishable.

1. scripts/media-tv.py — the other half of media-plan.py's refusal to file
   TV. Groups episode-shaped files by the series name before the marker
   (media-titles.ep_of, imported not restated), writes a plan in the
   executor's own format: <library>/<Series> (<year>)/Season <NN>/<own
   name>.<ext>. Refuses a group below --min, an episode naming no season,
   and an episode the destination library already holds at the same byte
   count — the movie share carries a second copy of all 124 Monk episodes
   the TV library has had for years; a duplicate is Frank's call, never a
   tool's copy. Applied by media-apply.py with its own journal and
   --root /mnt/entropy.

2. scripts/media-apply.py — gone meant "rmdir intended", so a folder whose
   rmdir failed was blacklisted forever and no later sweep could finish it
   (live movie sweep: 4517 targeted, 146 permanently stranded). gone now
   means "actually gone": a failed intent is retried by the next sweep.

tests: 9 new tests in media_relayout.bats, each red against the previous
revision — 8 TV + 1 sweep retry behind a chmod 500 parent. ./tests/run.sh
633 green. No live move run yet.
2026-10-30 20:30:24 -07:00
dank 5cd9a0c4c2 Merge pull request 'fix(media): the series is the name before the marker; a sweep carries its own empty subfolders' (#95) from fix/media-series-key into main 2026-10-30 20:20:24 -07:00
Frank a803aa046a fix(media): a sweep carries the emptied folder's own empty subfolders
The live sweep removed 2271 husks and left 146 standing, every one a release
folder blocked by a junk subfolder (Screens/, Sample/, a BDMV leaf) that no
file ever moved out of. Eligibility was per-folder, so the blocker was never a
target and its parent could never be removed.

Proof now carries downward: inside a folder the journal shows emptied, every
empty directory qualifies. Nothing under such a folder can hold data, and every
removal is still journalled as rmdir and undone by rollback.

The new test is red against the previous revision, green now. The live sweep
touched no file: 8193 before, 8193 after.
2026-10-30 20:18:54 -07:00
Frank 73f77a13c3 fix(media): a series is the name before the episode marker
Keyed on the whole episode title, every episode of a box set became a group of
one and never reached the five-episode size that calls something a series, so
the live run filed 124 Monk episodes as 124 movie folders, each named after its
own episode. ep_of now returns the series — the name before SxxEyy / 1x04 /
Part N — and the planner groups and names the quarantine pile by that series,
so one show is one pile: 'Monk - S05E03 - Mr. Monk and the Big Game' is series
monk. The threshold keeps its job: a lone film named Part 2 stays a film.

Re-planning the live census with this moves nothing differently except the 124
Monk files, which now join the TV pile: 168 episode files across 3 series.
2026-10-30 20:12:05 -07:00
dank dd76fcf5d4 Merge pull request 'feat(media): sweep removes the folders a completed run emptied' (#94) from feat/media-sweep into main 2026-10-30 20:04:24 -07:00
Frank 1cb28095a7 feat(media): sweep removes the folders a completed run emptied
An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
2026-10-30 20:03:16 -07:00
dank e1760e3316 Merge pull request 'feat(media): the movie-library relayout toolchain — census, plan, executor' (#93) from feat/media-relayout into main 2026-10-30 19:38:03 -07:00
dank 272e06daf6 feat(media): the movie-library relayout toolchain — census, plan, executor
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
2026-10-30 19:33:44 -07:00
dank 4db985b90d Merge pull request 'fix(qwen38): the preflight asks whether the pin is ONE of an image's digests' (#92) from fix/quant-pipeline-hardening into main 2026-10-30 16:30:07 -07:00
Frank 44d72b9e78 fix(qwen38): the preflight asks whether the pin is ONE of an image's digests
A by-digest pull leaves the image carrying every digest it was pulled under, so
reading RepoDigests[0] reported a healthy pinned image as wrong and refused a
window that was ready to run. Membership, not position. Also records the measured
step optimum: 106 tok/s at an 8,192-token step against 76 at 16,384 on a 32,768
calibration window, so the wide step has a ceiling, not a plateau.
2026-10-30 16:30:03 -07:00
dank 8073d0aa9d Merge pull request 'fix(qwen38): one identity preflight owns every fact a calibration window needs' (#91) from fix/quant-pipeline-hardening into main
Merge pull request #91
2026-10-30 16:26:41 -07:00
Frank 5c678b804c fix(qwen38): one identity preflight owns every fact a calibration window needs
Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid

scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
2026-10-30 16:26:07 -07:00
dank 81e0995f7d Merge pull request 'fix(qwen38): the calibration window must prove the rig owns the socket' (#90) from fix/quant-pipeline-hardening into main 2026-10-30 16:08:29 -07:00
Frank 167e6d6de3 fix(qwen38): the calibration window must prove the rig owns the socket
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.

The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.

Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
2026-10-30 15:54:57 -07:00
dank 608b0058e3 Merge pull request 'campaign(qwen38): the calibration floor is 12M tokens' (#89) from fix/quant-pipeline-hardening into main 2026-10-30 15:47:25 -07:00
Frank 4043798053 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:47:15 -07:00
Frank 521537a34c Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:11 -07:00
Frank 19fdffe8bd docs(agents): a start-only quadlet handler no-ops an image bump
The law today's authentik bump paid for: start on an active unit is a no-op, so
the new digest installs and the old container keeps serving behind three green
gates. Restart applies a pin.
2026-10-30 15:34:10 -07:00
Frank 54a4a4180a Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:34:03 -07:00
Frank f7aec7f154 fix(authentik): restart the units so an image bump actually applies
The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
2026-10-30 15:34:02 -07:00
Frank 299078eaba campaign(qwen38): the calibration floor is 12M tokens
Frank's rule: the served iq4r this arm competes with was calibrated on a
10M-token set, so the arm is calibrated at least as deeply. The floor was
lowered to 1.5M on the vendor's '10M beats 1M by under 1%' note; that
measures the vendor's own plate, not ours — the arm is judged against a
10M-calibrated reference, so cutting the calibration 6x to save wall clock
buys a worse position on the only comparison that matters.

One round of the re-chunked corpus is ~9.31M tokens, so a 12M floor seals
in round 1 plus part of round 2. A round that seals nothing is normal (the
floor is pooled) and round_verdict treats it as continue, never failure.

Also fixes tests/qwen38_campaign.bats' window dry run: its fixture lands on
/tmp (a 16 GiB tmpfs) where the 60 GiB seal preflight can never pass, so the
test now uses the documented Q38_CALIB_MIN_FREE_MIB override instead of
asserting environment luck.
2026-10-30 15:33:52 -07:00
dank 7b2b5b66fe Merge pull request 'fix(qwen38): the calibration tap runs on latest radiance, not the production pin' (#87) from fix/quant-pipeline-hardening into main 2026-10-30 15:30:21 -07:00
Frank e1d4d59773 docs(dragoon,zealot): record both runtime pins as they now run
frank-dragoon: engine 0.18.20 (image e35133b2…), the 18-model NPU catalog, the
re-cut qwen3-embedding-0.6b device programs, and the wire surface the new engine
actually answers (GET /tokenize is 405, /props is 200, /completion and /ping
stay 404). Eras gains the 2026-10-30 row.

frank-zealot: fork build 12118 (0.7.0-dev, 024474448) with the payload-load
evidence (202 pq2_0 tensors, 202 Hadamard-folded weights), plus the law a
rebuild pays for: cmake --install strips the rpath and installs bin/+lib/, so
the configure needs CMAKE_INSTALL_RPATH and LLAMA_BIN names …/bin/llama-server.
2026-10-30 15:27:21 -07:00
Frank 5787c86954 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:27:21 -07:00
Frank acbb539b97 feat(frank-zealot): PrismML fork b10777 -> build 12118 (0.7.0-dev, 024474448)
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.

LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.

The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.

Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
2026-10-30 15:25:24 -07:00
Frank 49a83ef1d5 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:25:24 -07:00
Frank d012654fc9 Merge remote-tracking branch 'origin/main' into fix/quant-pipeline-hardening 2026-10-30 15:23:06 -07:00
dank b45771e87e Merge pull request 'docs(radiance): reference the engine — options, plugins, conversion, tools, operations' (#88) from docs/radiance-engine-reference into main 2026-10-30 15:17:58 -07:00
Frank 011ceeb344 fix(npu fetch): never resume into a file from another revision
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
2026-10-30 15:13:42 -07:00
Frank 88a9bc02be Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:42 -07:00
Frank 6ac544a7b6 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:13:23 -07:00
Frank 33396d2daa feat(frank-dragoon): halogen-flash-server 0.16.2 -> 0.18.20, NPU lane re-pinned
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.

0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
2026-10-30 15:13:19 -07:00
Frank bdc0613739 docs(radiance): correct the tokeniser corpus name and statsdump auth note 2026-10-30 15:03:31 -07:00
Frank 3e32cb9cd1 campaign(qwen38): the tap producer is latest radiance, at a vendor-backed floor
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.

The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).

Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.

The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
2026-10-30 15:02:58 -07:00
Frank 6e80cd44f0 Merge remote-tracking branch 'origin/main' into wt/20261030-143311 2026-10-30 15:01:40 -07:00
Frank 3a04e34414 docs(radiance): reference the engine's options, plugins and conversion
Nine pages under LLM/ document the upstream radiance engine at 8da043c:
- engine shape, plugin model, RAD2 container, ABI and environment
- every serve flag grouped with defaults and the law behind each knob
- the HTTP surface, request fields and refusal classes
- rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring
- the (arch id, quant) family table and the recipe of record per model
- kernel/quantiser plugin authoring, encodings, build and checks
- architecture plugin authoring: declare, step, probe, lane and media rules
- rad-info/radbench/radtune and the shell harnesses
- the container contract, host prerequisites, readiness and restart triage

Also completes the dangling Related line on the frank-archon endpoint page.
2026-10-30 15:01:37 -07:00
Frank 9085a80bd1 Merge branch 'chore/fleet-image-refresh' into main 2026-10-30 15:01:07 -07:00
Frank 58184008dd chore(images): refresh every registry pin to current upstream digest
Container images re-resolved against their registries today; each pin is the
manifest-list digest the tag answers now, so no container is served from a
build that upstream has already replaced.

Version-tag moves: authentik 2026.8.3 -> 2026.8.5 (server/worker/outpost and
the ldap image, which now carries the tag instead of a bare digest so the pair
can never drift apart), seerr v3.4.1 -> v3.5.1, calibre 9.15.0 -> 9.17.0,
calibre-web 0.6.27-ls402 -> ls407, plex ls329 -> ls330, tautulli
v2.18.1-ls245 -> v2.18.2-ls250.

Digest-only moves inside the same stream: postgres 16-alpine, mariadb 11,
redis 7-alpine, valkey 9-alpine, grafana, alertmanager, gluetun, searxng,
open-webui, jellyfin, romm, chromedp/headless-shell, lightpanda/browser.

Rollback is the previous digest on the same Image= line.
2026-10-30 15:00:52 -07:00
Frank b4a1e6069e fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.

Four gates, each with the measurement that motivated it:

- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
  @sha256:665775ed…), overridable only by naming --image deliberately. Proven by
  the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
  One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
  filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
  starts the service. `podman rm -f` does not kill the engine: an orphan made
  production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
  restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
  target configurable, so the tap lives on /mnt/entropy instead of the root fs.
2026-10-30 14:18:03 -07:00
dank 9f997f0841 Merge pull request 'perf(quant): the calibration boot's step and slots are its own knobs — probe at the measured 71 tok/s' (#86) from fix/qwen38-calib-shape into main 2026-10-29 16:50:50 -07:00
Frank f6298e3501 perf(quant): the calibration boot's step and slots are its own knobs (probe: 71 tok/s, 47 h tap)
Measured from the engine's own counters on the 2026-10-29 drain: prefill 71 tok/s steady state, a
180 s window showing 7 prefill steps of the capped 2,048 tokens and 3 preemptions. One 7,168-token
document is therefore ~100 s, round 1 (2,367 docs, 10.03M tokens) ~39 h, and the 12M-position
floor ~47 h of drained production — not the ~18 h the brief assumed.

The tap is the only phase whose cost is read volume per document: at the inherited 2,048-token step
a document is four steps, and each step re-streams every layer's routed expert union from the
container (43.4 GiB a rank live on disk). One wide step reads each layer's union once.

The 'one step, one slot count' audit law binds kldref and score because their KL numbers are only
comparable at an identical shape. Calib produces layer Hessians and no number any report compares,
so Q38_CALIB_MNBT / Q38_CALIB_SEQS are its own, defaulting to the inherited value — nothing moves
unless a probe asks. The KL modes ignore both.

Tests: 589/0 — the knob reaches calib's argv, does not reach kldref's, and a non-numeric value is
refused by name.
2026-10-29 16:50:50 -07:00
dank f14efdce1a Merge pull request 'fix(quant): a flat calibration round is not a failure — the tap's floor is pooled' (#85) from fix/qwen38-flat-round into main 2026-10-29 16:17:52 -07:00
Frank a6fb0209e8 fix(quant): a flat calibration round is not a failure — the tap's floor is pooled
RADIANCE_CALIB_TOKENS=12000000 is a pooled floor and calib-7168.jsonl carries 10.03M tokens, so
round 1 of a corpus push seals nothing by construction. The gate PR #84 added failed exactly
there, one round too early, while the drain was already running.

Now round_verdict() separates the cases: 'shrink' (the tap lost bytes) fails the run, 'flat' says
so and continues, 'grew' continues, 'done' stops. A run that never sees the engine's completion
line still exits non-zero on max rounds, and the rig marker remains the lock that stops a push
into anything that is not the rig.

Test: round_verdict's four verdicts, behaviorally.
2026-10-29 16:17:52 -07:00
dank 92a412e99d Merge pull request 'fix(quant): a tap is only a tap when its rig is running — marker gate, gated calib window, tail-and-slices gate' (#84) from fix/qwen38-tap-marker into main 2026-10-29 16:14:34 -07:00
Frank c238e32d6f fix(quant): a tap is only a tap when its rig is running
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.

- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
  GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
  kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
  non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
  and that container is running. With the driver's existing tap-progress gate, a push can no
  longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
  instead of falling through the health wait, bank the 336 GiB reference container into trash
  only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
  172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
  read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
  section that names the live campaign instead of the retired swift15 era.

Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
2026-10-29 16:14:16 -07:00
dank da5eb968f9 Merge pull request 'fix(quant): harden the campaign pipeline, then refactor its corpus tools onto one module' (#83) from fix/quant-pipeline-hardening into main 2026-10-29 15:29:33 -07:00
dank b3052e9243 Merge pull request 'fix(frank-archon): fold the pinned host pool back to 49,152; keep the 2,048 step' (#82) from feat/qwen38-w5nl64 into main 2026-10-29 15:29:29 -07:00
Frank 3e5a37a21f refactor(quant): one corpus plumbing module, and the builders out of main()
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.

Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
2026-10-29 15:26:08 -07:00
Frank 1e4fa14772 fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.
2026-10-29 15:14:43 -07:00
Frank e6fba5d97e fix(frank-archon): fold the pinned host pool back to 49,152; keep the 2,048 step
PR #81's pool half measured inert the day it deployed: the 66,560 boot printed the
same elastic 26.73 GiB / experts 20.72 / kv cache 5.64 / vram_weights 24.30 as the
49,152 boot and opened the pool with 0 B used, and the qwen38-w5nl64 rig moved
vram_kv 269.99 -> 271.25 MiB (window still 7,436 tokens: its 95.37 GiB n-gram table
claim decides, not the pool). The step half stays. Tests: 576 ok / 0 fail.
2026-10-29 15:10:33 -07:00
Frank ca1475543d docs(frank-archon): the 2026-10-29 boot ledger - the 66,560 pool gives the same cache as 49,152
The container opened the raised pool with 0 B used and printed the same elastic 26.73 GiB / experts 20.72 GiB / kv cache 5.64 GiB / vram_weights 24.30 GiB as the 49,152 boot, so the era's superseded ledger line is destroyed and the evidence table gains the ceiling-case row beside the 45,568 one it already carried. Tests: ./tests/run.sh 576 ok / 0 fail.
2026-10-29 13:48:34 -07:00
dank f83c241668 Merge pull request 'feat(frank-archon): raise the pinned host pool to 66,560; the campaign inherits MNBT 2,048' (#81) from feat/frank-archon-pool-step into main 2026-10-29 13:14:27 -07:00
Frank 255796b3dc feat(frank-archon): raise the pinned host pool; the campaign inherits the step
frank-archon.env: HOST_POOL_MIB 49,152 -> 66,560 a rank (tp x 65 GiB against the
246 GiB the box has), and the TTM pair 162,000/204,800 -> 197,000/224,000 — the
floor the bats ledger formula computes at the new pool. The pool is the KV lever
under the budget resolver: every MiB pinned in host RAM lowers the expert floor on
the card one-for-one and the cache takes it.

scripts/run-qwen38-bf16.sh: the campaign's private FRANK_QWEN38_MNBT (1,024) is
gone — MNBT now comes from FRANK_ARCHON_MAX_BATCHED_TOKENS, production's 2,048 and
the widest step the quantised expert file tiers stream. That 1,024 was PR #80's
answer to a starved cache; the pool is the fix, a narrow step was the symptom's
disguise, and a per-mode literal measures a shape nothing serves.

tests: frank_archon_unit greps the new pool value; qwen38_reference's SHAPE fixture
carries it, and the two MNBT-override tests become the inheritance assertion (every
mode's argv quotes the value the env file owns) plus the calib assertions.

vault: the shape block moves to the new numbers and the Eras table gains the row for
the era this replaces. The 2026-10-20 boot-ledger line stays as the old era's record
until the new boot's ledger replaces it.
2026-10-29 13:10:32 -07:00
dank b4d7f6e7fe Merge pull request 'fix(qwen38): calib rides the shared step; the wide one cost the container its cache' (#80) from feat/qwen38-w5nl64 into main 2026-10-29 11:18:36 -07:00
Frank a3ff55efd8 fix(qwen38): calib rides the shared step; the wide one cost the container its cache
The calibration mode kept a private step (8 seats at MNBT 65,536) that it
called "free throughput". The step's activation arena is billed to the card,
so the wide step cost the container its own expert slab and KV: at 65,536
the arena is 10.65 GiB and the claim falls to 16.85 GiB (weights 14.24 GiB
in VRAM, 56.24 GiB stranded on disk, KV 271 MiB for 8 seats), and the boot
dies with "startup stopped: out of memory" (measured 2026-10-29, while the
kldref boot at MNBT 1,024 — arena 504 MiB, claimable 30.87 GiB — had just
recorded for 16 hours without fault).

Every mode now takes its seat count from FRANK_ARCHON_MAX_SEQS, the file that
owns it, exactly as PR #77 established for the card split, and its step from
FRANK_QWEN38_MNBT. The per-mode literals in kldref and score follow, so one
number defines the geometry for all three. The test that pinned the old
exemption is re-pinned to the invariant that replaced it.

tests/run.sh: 576/576 green.
2026-10-29 05:32:14 -07:00
dank fad87aa522 Merge pull request 'feat(scripts): deterministic .AGENTTRASH purger with bats seam' (#79) from wt/20261028-122814 into main 2026-10-28 19:22:20 -07:00
Frank a3c2465638 feat(scripts): deterministic .AGENTTRASH purger with bats seam
Portable POSIX-sh purge of exactly the fleet's deletion-policy trash: a
root's own .AGENTTRASH is emptied in place (the convention survives),
deeper .AGENTTRASH trees are removed with their nested entries accounted
once, symlinked/foreign-owned candidates are refused, / is refused.
Dry-run by default; --execute performs the plan and appends one dated log
line; every printed line is fixed at plan time in the on-disk plan table,
so dry-run and execute output can never drift.

Documented in Operations/Payload and Disk Reclaim.md section 6 (one fact
one owner - no new vault page). Tests: 14 green, ./tests/run.sh 576 green.
2026-10-28 15:54:23 -07:00
dank 58dc7a64c1 Merge pull request 'feat(qwen38): score the inherited corpus at the window the container really has' (#78) from feat/qwen38-w5nl64 into main 2026-10-28 11:04:59 -07:00
Frank 9c386d4f89 feat(qwen38): score the inherited corpus at the window the container really has
The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.

Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.

The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
2026-10-28 00:26:28 -07:00
dank fca47d25d8 Merge pull request 'fix(qwen38): the rig's card split is inherited from frank-archon.env, not invented' (#77) from feat/qwen38-w5nl64 into main 2026-10-27 23:53:12 -07:00
Frank 1c56dea67a fix(qwen38): record at MNBT 1,024 — the step size the cache can actually pay for
The rerun after 96c5a06 refused every corpus document again ("document 196 is
8507 tokens and the KV pool holds 7436"), and the boot table shows why the
host-pool change could not have helped: vram_budget.cpp put the container in its
`no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed
share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host
pool. The only supplier left there is the activation arena, and the engine says
so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales
with it)". The log prices both rungs directly (a 2,048-token step costs 1.08
GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache
grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents
(longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT.

This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys
KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as
single-source-of-truth, but on this payload the pool is not the cache's supplier.
2026-10-27 22:19:51 -07:00
Frank 96c5a0625c fix(qwen38): the rig's card split is inherited, not invented
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768,
copied from the w4-era swift15 rig. The engine derives the whole per-card split from
those two numbers, and the KV cache's funding is `elastic - expert_floor` where the
expert floor is a function of the pinned host pool — so the pool buys cache as well
as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the
recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV
pool holds 7436"), exiting 1.

The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from
scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and
the header records the law instead of the claim that the pool never touches logits.
tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with
it; the suite is green (558 tests).
2026-10-27 21:54:19 -07:00
dank 8270524edd Merge pull request 'feat(qwen38): the w5nl64 heavier-quant campaign toolchain (fetch, BF16 reference, arms, FP8-anchored gate)' (#76) from feat/qwen38-w5nl64 into main 2026-10-27 19:59:37 -07:00
Frank a7329d190f feat(qwen38): the BF16 reference instrument - build it, drain once, record logits
Two tools the KLD gate depends on.

build-qwen38-bf16-rad.sh converts the checkpoint to a BF16 .rad with NO recipe
line, so every tensor stays bf16 - the reference must not carry the quantization
it is there to measure. It gates before writing 335 GiB: the checkpoint present
and fully verified by the fetcher's own verifier, the peak-disk bill (350,000
MiB: the checkpoint's 343,323 plus margin, because the .rad is built beside it),
an existing output retired to ~/.AGENTTRASH by --force instead of deleted, and
--plan-only before the real convert. --max-ctx 32768 is a declare-time check, not
a weight change: dropped, the resolver is handed the checkpoint's 262,144 window,
which at fp8 cache is 258.8 GiB on a 192 GiB machine.

run-qwen38-bf16.sh serves that container in three modes - kldref (record the
reference logits), score (record an arm's logits against that reference), calib
(open the Gram tap for the imatrix). It drains frank-archon and restores it on
every exit path, and the two logits modes ride ONE step size
(FRANK_QWEN38_MNBT, default 2048) and ONE window (FRANK_QWEN38_MAX_MODEL_LEN,
default 32768) so a reference and an arm cannot be measured at different shapes:
that is the whole validity of the KL number. Four slots over 2,048 matches
frank-archon.env's operating point (MAX_SEQS=4, MAX_BATCHED_TOKENS=2048); the
smaller host pool (12,288 vs 49,152) buys residency, which moves speed, not
logits. calib keeps 65,536 - decode-bound serving, never the gate.

kldref refuses before the drain when the corpus's longest record cannot fit the
declared window (limit = window x 3 chars, the pessimistic tokenizer bound; the
measured corpus is 38,263 chars, 2.6x inside 32,768). It runs on --dry-run too,
so the check is visible before a 20-minute drain is spent. 12 seam tests, each
red-proofed by neutralising the gate it pins.
2026-10-27 19:40:48 -07:00
Frank 5fe04e569f test(qwen38): campaign seam for fetch, protected plane, arms and gate
32 black-box tests over the four tools the campaign runs, driven through their
own argv with fixture corpora - no GPU, no container, no rad-convert. Each test
is an output an operator reads: the verifier's MISSING line and exit, the byte
bill the derivation lands, the refusal text, the plan and the convert the driver
prints per arm, the interval verdict the gate returns.

Includes the planner seam the swift15 campaign lacked: the campaign's paths plan
NO deployable service, so merging this toolchain can never drain frank-archon.
The driver fixture carries four grams so --min-grams has a thin-pool control,
and the arm-driver invocation passes env assignments positionally after the
driver's own flags, so the seam exercises the real call shape.
2026-10-27 19:36:58 -07:00
Frank bb027265db feat(qwen38): the FP8-anchored KLD gate - the served payload is the floor, not BF16
The swift15 gate compared an arm against the BF16 reference and passed on
KL <= 0.03. That bar is now the wrong instrument: the fleet already ships the
publisher's FP8-native iq4r-moe payload, so an arm that beats BF16 but loses to
what is running is a downgrade. The gate takes three recordings - fp8 anchor,
iq4r anchor, candidate - and requires the candidate's KL strictly inside the
interval the two anchors define, at every slice, plus top-1 agreement no worse
than the served payload.

The anchors must be ordered (fp8 better than iq4r) or the run stops: an
unordered pair means the instrument or the recording is wrong, not that the arm
lost, and one number out of the report is named rather than guessed at.
--recover prints the tightened numbers the same way it prints the failing ones,
so the operator sees the bar they would have had to clear.
2026-10-27 19:36:53 -07:00
Frank 4e551fafd7 feat(qwen38): the w5nl64 recipe template and the three-arm convert driver
One template (every routed expert at five bits, E4M3 scales in 64-groups, the
top level rotated through the 128-wide Hadamard chosen by GPTQ against CALIB)
plus a driver that generates the per-arm expert row, so the other 60 lines can
never drift between arms: control (cd=3), cd=6, and act_order=1 with NO cd= -
libquant refuses the pair (lq_registry.cpp:105), so the AoA arm is its own
sweep, not a fourth column.

The header carries the campaign's order (fetch -> build reference -> one drained
window for the calibration tap and the KLD reference -> trash the bf16 container
-> imatrix -> arms -> fetch the FP8 anchor last -> gate -> trio edit), because
that sequence is point-of-use for whoever runs it next, and the disk sequencing
is forced: 754 GiB free does not hold checkpoint + reference + three arms.

Gates that bite before 5h13m of convert time is spent: the BF16 reference still
occupying the volume, a recording too thin to be a calibration (converting
uncalibrated is silently worse and only whispers on stderr), an imatrix outside
the mounted campaign dir, an arm that already exists, a protected derivation
that cannot land its target, and a destination that cannot hold the output. Each
arm runs --plan-only first so the engine's own fit check bills at plan time; the
served container's argv shape is what the arms declare.
2026-10-27 19:36:53 -07:00
Frank 3d8d18a45c feat(qwen38): fold the calibration tap's Grams into libquant's imatrix GGUF
rad-convert's GPTQ path reads importance from an imatrix GGUF; the reference
container's --kld-record pass emits per-expert and per-hidden Grams as raw
recordings. This folds them into one v3 file: a single expert-major f32 entry
per MoE layer plus the hidden-width entries, every data offset a multiple of
general.alignment as the loader requires.

Refusals, each one a silent-worse outcome the swift15 campaign hit: an expert
with no gram of its own carries the layer's pooled diagonal and the count is
reported (a silent zero is a down-quantized expert nobody priced); expert grams
with no pooled gram, a gram short by one float (names the file), the same global
expert recorded twice with different numbers, an empty recording directory, and
an existing output. --min-rows sets the noise floor below which a recording is
dropped. rows is u64 so a corpus that outgrows u32 survives the header instead
of wrapping.
2026-10-27 19:36:38 -07:00
Frank c13a2568d4 feat(qwen38): size the protected bf16 expert plane by byte bill, not by ratio
The swift15 rule (top 7% by imatrix energy, bar-only) left the plane size to the
distribution. The w5nl64 arm trades a *pinned host pool* against KL, so the
operator needs to say how many GiB of bf16 experts the pool can afford and have
the set land there: --target-count walks the ranked candidates until the byte
bill matches, keeps the bar-only pass for experts that are disproportionate for
their size, and reports the pool's ceiling instead of padding when the count
cannot be reached.

TP=2 cuts the expert plane into two 256-expert rank halves, and an odd count in
either half unbalances the placement, so the parity audit is a refusal (exit 2,
'PARITY LAW BROKEN'), not a silent fix-up: the driver stops rather than
converting a payload whose expert placement no one priced.
2026-10-27 19:36:32 -07:00
Frank e7a835e41f feat(qwen38): fetch and verify the vendor Qwen3.8-Flash-Next BF16 checkpoint
The w5nl64 campaign measures against the vendor checkpoint, not the Swift-1.5
finetune: the served payload (qwen3.8-next-flash-fp8-iq4r-moe) is quantized from
this base, so fidelity must be scored against the same reference. 335.3 GiB
across 10 shards.

Verifier gates on the index's own shard list, each shard's declared byte count,
and the Hub's declared total - a curl without -f writes the 401 body as the
'shard' and reports success. --fetch refuses when the destination cannot hold
the checkpoint, because a 335 GiB download that fills the disk mid-shard costs
the whole campaign window.
2026-10-27 19:36:26 -07:00
dank 0ecfe830f7 Merge pull request 'fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs' (#75) from fix/signal-relay-fw-order into main 2026-10-21 16:59:16 -07:00
Frank ce370a5b28 fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs
iptables -I prepends, so the install block's final -I of the blanket DROP
landed it at position 1, above all three ACCEPTs: every packet to
18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each
ACCEPT). The host's own health probe timed out beside a LISTENing socket
and the whole dankclaw drain aborted at the alertmanager gate twice.

Install now rebuilds the tail: delete any blanket DROP wherever it sits,
keep the ACCEPTs at the head, append the DROP last - converges on hosts
poisoned by the old order. New bats case runs install_signal_relay for
real against an iptables sandbox seeded with the poisoned order (red on
the old code, green on this one).
2026-10-21 16:52:40 -07:00
Frank bfe4d185bd docs(vault): OpenWebUI picker opened to all three models for every account (2026-10-21)
Per Frank, frank-archon is public alongside dragoon/zealot: three-leg DB
trio (conn enable, model row ctx 200000, access_grant principal='*'),
verified as role=user. Token-mint traps recorded (cwd, PID-1 secret).
2026-10-21 16:48:32 -07:00
dank 94a9f3aa31 Merge pull request 'feat(security): digest-pin every registry image; drop batch-killing HostAddr' (#74) from security/quadlet-digest-pins into main 2026-10-21 16:32:44 -07:00
Frank d2810ab610 feat(security): digest-pin every registry image on all three hosts (fleet law)
Closing the command-center security queue #70/#71/#72/#73/#74: rolling tags
let a rebuild move under a running service with nobody approving it, and
make a CVE audit ungradeable. 36 quadlet units across dankclaw, dankserver
and frankbook-pro now carry tag@sha256; localhost/* builds stay exempt
(their build is the pin). The pihole and copyparty pins pull the fixed
builds; valkey 9.1.2 (already running) and redis 7.4.11 are pinned in
place, vendor fixes TBD.

New gate tests/quadlet_image_pins.bats: no unpinned registry image, no
generator-rejected key anywhere. Three seams that pinned the old exact-tag
law (frankbook readarr, open_webui, seerr) now require tag AND digest -
strictly stronger. Quadlet Lifecycle docs: batch-abort law, entrypoint
flags-only law, pin law.
2026-10-21 16:32:36 -07:00
Frank 7a8cc83385 fix(quadlet): drop HostAddr from prometheus.unit - podman 5.7.0 rejects the key and the quadlet generator then aborts the ENTIRE user batch, silently orphaning every other unit (prometheus.service vanished while the container ran; deploy died 'Unit prometheus.service not found', live 2026-10-21). host.containers.internal resolves natively; the key was never needed. 2026-10-21 16:32:36 -07:00
dank c4623f1629 Merge pull request 'fix(blackbox): flags-only Exec; bind loopback+LAN; exec-form HealthCmd' (#73) from fix/blackbox-entrypoint-bind into main 2026-10-21 16:25:32 -07:00
Frank 065a6911f9 fix(blackbox): Exec carries flags only (image has ENTRYPOINT) and the exporter binds loopback+LAN
Three stacked lessons, each proved live 2026-10-21:
- the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the
  binary made it a positional arg: 'unexpected /bin/blackbox_exporter'.
- the podman gateway 169.254.1.2 is NOT bindable from the host netns - it
  exists only inside each rootless netns: 'bind: cannot assign requested
  address'. The scrape path does not need it bound: a bridge container
  dialing the gateway lands on the host LOOPBACK (proved 200 from the
  prometheus container), so the exporter binds 127.0.0.1 + LAN.
- HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token
  and quadlet folds the following --health-interval/--health-retries (and
  the image) into the command string; the plain exec form the sibling units
  use is correct.

Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a
bind address); deploy-remote gate stays loopback, which is now honest.
2026-10-21 16:04:53 -07:00
dank 745d66d60d Merge pull request 'fix(blackbox): --web.listen-address (repeatable), not --listen-address' (#72) from fix/blackbox-web-listen-flag into main 2026-10-21 15:58:36 -07:00
Frank 4009f118c1 fix(blackbox): the flag is --web.listen-address, repeatable, not --listen-address
Third boot failure: the exporter exited 100 ms in with 'unknown long
flag --listen-address' (proved against the pinned image, plus --help).
The bind semantics were right; the flag name was invented. --web.listen-address
is repeatable - one per address - so the dual gateway+LAN bind stands.
Proven again pre-deploy with the real argv: the run starts and dies
only on the deliberate /dev/null placeholder config.
2026-10-21 15:55:21 -07:00
dank 7de343c171 Merge pull request 'fix(blackbox): complete the unit - Image, Network=host, honest bind topology' (#71) from fix/blackbox-unit-generation into main 2026-10-21 15:52:30 -07:00
Frank 139a3cd4fa fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it
The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:

- Prometheus is bridge-networked and the netavark hairpin gap bars it
  from its own host's LAN IP, so a LAN-only prober bind is unscrapable
  and a loopback-only bind was claimed to ride the gateway on faith.
  The prober now binds 9115 on the podman gateway (scrape path) AND
  the LAN (healthcheck/diagnostics); the listen_address line leaves
  blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
  bridge container names do not resolve in the host netns; the
  published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
  shipped file by name); copier test pinned module presence instead
  of the removed listen line. Docs updated to the real topology.
2026-10-21 15:41:37 -07:00
dank 5ffbf19fb2 Merge pull request 'fix(deploy): blackbox-exporter was gated out of its own deploy path' (#70) from fix/blackbox-service-gates into main 2026-10-21 15:34:49 -07:00
Frank 1fd98cab3f fix(deploy): blackbox-exporter was gated out of its own deploy path
PR #69 wired the copier, planner entry and health gate but missed the
three dispatch gates: the client valid_service, the remote
service_is_allowed, and deploy_service's case list. The client gate
rejected --service blackbox-exporter outright; even past it the
dispatcher's catch-all hard-fails on dankclaw (no compose since
2026-09-16). New seam pins the class: every dankclaw service the
remote gate allows must carry a deploy_service case - proven red by
planting a gated, undispatched name.
2026-10-21 14:59:47 -07:00
dank d61c19d41f Merge pull request 'Observability: kill every structural error (blackbox, labels, gates)' (#69) from fix/observability-errors-gone into main 2026-10-21 14:53:16 -07:00
Frank e6fcf217b4 docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon
Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB,
iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not
fit that budget, and the page's 122 GB /system_stats figure is the
aperture, not headroom. Adds the fit table and the
stop-mask-render-verify procedure (the 3090 bench law, applied to the
APU); ComfyUI.md gotcha links it.
2026-10-21 13:35:50 -07:00
Frank cb86ad8fd7 docs(observability): post-blackbox state, copier law, label-promotion why
Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
2026-10-21 13:33:03 -07:00
Frank 078b550bd8 test(observability): copier sandboxes, probe-target seam exception, cop gates
- caddyfile seam learns the documented exception: URL-shaped targets
  are legal only inside a prober job (state is per-job; the first cut
  leaked it and passed a planted URL).
- observability seams: blackbox unit + planner entries, both copiers
  executed for real in sandboxes, alloy engine-validation refusal
  (image-gated), dashboard panel invariants.
- AGENTS.md: structural target downs retired.
2026-10-21 13:33:03 -07:00
Frank 765a8cb7de feat(deploy): blackbox copier, alloy engine-validation gate, planner wiring
copy_alloy_config now validates the rendered config with the very
alloy image that will run it, before installing: a misplaced attribute
crash-loops the unit behind a green deploy, and no textual seam can
see it. copy_blackbox_config ships the prober config; the planner maps
the new unit and shared/blackbox/** to the blackbox-exporter service,
ordered before prometheus.
2026-10-21 13:33:03 -07:00
Frank 999c4fe440 fix(alloy): promote CONTAINER_NAME to an indexed Loki stream label
Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
2026-10-21 13:33:03 -07:00
Frank 2af2e6ac56 fix(grafana): observability and logs-overview boards read real data
observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
2026-10-21 13:32:58 -07:00
Frank fbe54ca1ad feat(gitea): enable /metrics for Prometheus scraping
Gitea shipped metrics disabled, so its scrape target was a permanent
structural down beside a healthy service.
2026-10-21 13:32:58 -07:00
Frank f073b91033 feat(observability): blackbox exporter replaces the HTML-probe jobs
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
2026-10-21 13:32:58 -07:00
dank 425994515a Merge pull request 'fix(deploy): grafana copier provisions EVERY dashboard json' (#68) from fix/grafana-copier-glob into main 2026-10-21 09:49:15 -07:00
Frank c41384e03d fix(deploy): grafana copier provisions EVERY dashboard json, not a hand-listed subset
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.

Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
2026-10-21 09:40:57 -07:00
dank 127621bdd7 Merge pull request 'docs(vault)+fix(grafana): audit tranche 1 and dashboard data-flow fixes' (#67) from fix/dragoon-context-200k into main 2026-10-21 09:28:43 -07:00
Frank c96d0228a4 fix(grafana): dashboard census fixes — every panel now receives data
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).

- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
  non-matching pairs and the right side collapses to {job} while the left
  carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
  live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
  container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
  so the '> 0' filter can never match. Rewritten to share-of-host-total via
  label_replace join of the node exporter's MemTotal by host (the parens
  form is a PromQL parse error: nothing may follow group_left); 62 series
  live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
  (ALERTS carries only firing/pending series); description now says so,
  per the Stale Signal rule that 0-series-on-ALERTS is not a defect.

logs-overview and observability: all targets already returning data.
2026-10-21 02:45:57 -07:00
Frank a61308d72e docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):

- Observability Provisioning: destroyed the 63-line 'Current delivered
  state' block — chain topology/relay/delivery restates [[Observability]]
  and [[Alertmanager]] (those are the owners). Its three sole-custody laws
  MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
  down-beside-healthy -> read lastError; Prometheus alert state in process
  vs Alertmanager's --storage.path persistence; candidate-arm metrics are
  never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
  never reuse a B/token ratio across geometries; pin pool bytes
  floor-sized), era-labelled the rates, destroyed the measured ladder
  table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.

check-okf 109/0, vault links 0 unresolved, docs seams green.
2026-10-21 02:35:44 -07:00
dank ccb805cf0b Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66) from fix/dragoon-context-200k into main 2026-10-21 02:25:42 -07:00
Frank c6e2887f94 fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/
as well; our mounts filled only /opt/xilinx/xrt/lib and the install path.
Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming
the missing prefix ('No such library
/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine
supervisor takes the WHOLE container down on a dead lane — measured
2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the
GPU lines looked healthy.

Each XRT library now mounts THREE times from the same real file (never a
symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename
so the bats fixture proves the derivation. Proven end-to-end before commit:
production-shaped scratch boot + the three extra mounts answers the full
boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling.

A dead lane is not survivable on this engine — arm it only with all three
prefixes (page law updated).
2026-10-21 02:21:53 -07:00
dank 9eb53ef937 Merge pull request 'feat(frank-dragoon): 200,000-token harness window via the native 262,144 ceiling' (#65) from fix/dragoon-context-200k into main 2026-10-21 01:40:40 -07:00
Frank 688928d489 docs(dragoon): the 262144 ceiling, the 200,000 window, and the 95 % trigger law
Shape + harness contract on the endpoint page, AGENTS.md, and the OMP
Harness worked example recomputed to the live pair; the 400-zone law now
names the 85 %-comments vs 95 %-config disagreement as the recurring
cause and points at the seam that enforces the arithmetic.
2026-10-21 01:34:28 -07:00
Frank 56503dbad1 feat(frank-dragoon): raise HALOGEN_CTX to the native 262144 ceiling
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.

Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.

The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
2026-10-21 01:34:28 -07:00
dank fbeb9701e4 Merge pull request 'fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway' (#64) from fix/gpu-serve-and-cadvisor-target into main 2026-10-21 00:39:54 -07:00
Frank 650b6e0281 fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
2026-10-21 00:34:17 -07:00
dank b62a05837f Merge pull request 'fix(deploy): client allowlist missing cadvisor/gpu-exporter/prowlarr' (#63) from fix/deploy-sh-allowlist-drift into main 2026-10-21 00:15:45 -07:00
Frank 0367244d2a fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.

Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
2026-10-21 00:15:21 -07:00
dank 656d9567d5 Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62) from chore/plex-1435 into main 2026-10-21 00:07:28 -07:00
Frank 0b4bc60a38 chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.

Rollback: revert this commit (one line).
2026-10-21 00:06:53 -07:00
dank a2011f4eb6 Merge pull request 'feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet' (#61) from feat/gpu-stats into main 2026-10-20 23:50:11 -07:00
Frank b02647b992 feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.

scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.

It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
2026-10-20 23:42:26 -07:00
dank b35269bfa5 Merge pull request 'docs(archon): four-slot ledger, resolver law from source, probe self-calibration' (#60) from wt/20261020-213526 into main 2026-10-20 23:30:27 -07:00
Frank 1aeb7af6df docs(probe): price the corpus chars/token ratio at run time
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
2026-10-20 23:26:53 -07:00
Frank cd0be97b39 docs(vault): the four-slot ledger, and the resolver's funding law from the source
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.

The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
2026-10-20 23:26:53 -07:00
dank 7993a336bf Merge pull request 'feat(frank-archon): four decode slots over a >=700k-token KV pool' (#59) from wt/20261020-213526 into main 2026-10-20 23:14:35 -07:00
Frank 1bb6399524 feat(frank-archon): four decode slots over a >=700k-token KV pool
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.

Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.

TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
2026-10-20 23:10:05 -07:00
dank 31b71a16d6 Merge pull request 'feat(observability): per-container metrics via cadvisor + Containers dashboard' (#58) from feat/containers-observability into main 2026-10-20 23:08:23 -07:00
dank a4edaa7d9d Merge pull request 'feat(frank-archon): arm --prefill-interleave at 1' (#57) from wt/20261020-213526 into main 2026-10-20 22:55:58 -07:00
Frank aa75b45543 feat(frank-archon): arm --prefill-interleave at 1
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
2026-10-20 22:52:31 -07:00
Frank 31908c623f feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
2026-10-20 22:51:42 -07:00
Frank b1fc1113c6 docs(vault): replace an invented --checkpoint-policy name with the three that exist
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly
geometric-backoff (default), tip-only, keep-all and refuses anything else at
boot, so the recorded arm could never be deployed. Replaced with the real
semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric
ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live
ledger line -- so a turn diverging d behind the tip can resume at tip-3d:
diverge at 120k, replay 21.7k); retention is not the write path, so keep-all
buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4
GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room);
and shortening --checkpoint-interval cannot fix the deep case -- the coarsest
ladder scale is interval x 2^k under the depth, so the deepest gap stays about
half the prompt at every interval. §9 keeps only the three-name locator.
2026-10-20 22:34:13 -07:00
Frank 9f7457e39e docs(vault): answer "what are we doing wrong" on decode-vs-prefill
frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.

Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.

§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.
2026-10-20 22:08:00 -07:00
Frank 687dbfbabe docs(scripts): correct frank-archon's step-sharing and slot-count comments
The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1
source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding
sequence and 1/8 of the budget per waiting request, and refuses to shrink the
chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment
carried a 512 that is not this build's auto value (0 until the held-step
counter asks; 64 in help mode). The serve header still described three decode
slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.
2026-10-20 22:07:55 -07:00
dank 961713e92d Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56) from feat/e2e-audit-coverage into main 2026-10-20 21:21:44 -07:00
dank 01db29e754 Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55) from feat/grafana-starter-dashboards into main 2026-10-20 21:20:53 -07:00
Frank ab06e4c221 test(e2e): cover the two guard-surface gaps found in the full-suite audit
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:

- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
  vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
  upstream would stay invisible - the exact 17-hermes class. Added to the 14-
  guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
  new 21- that proves authenticated reachability to open-notebook's own login
  through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
  wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
  there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
  answers {status:success}.

Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
2026-10-20 21:16:23 -07:00
Frank 1006e96312 feat(grafana): starter dashboards for host/system, Loki logs, and observability self-monitoring
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
  Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
  container is parsed from the JSON line at query time (journal streams have
  no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
  interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.

Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
2026-10-20 21:10:06 -07:00
dank 7d85fed1c9 Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54) from fix/grafana-authentik-sso into main 2026-10-20 21:04:02 -07:00
Frank d8196ce499 docs: Grafana reads claims from userinfo, not USE_ID_TOKEN
The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.
2026-10-20 20:41:57 -07:00
Frank 6f442a14a3 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:41:36 -07:00
Frank afe72781bc docs: correct Grafana Authentik SSO facts
The vault claimed the fork property-mapping evaluator was broken and userinfo
returned only sub, so email could never reach Grafana (outcome unverified). That
was a misdiagnosis: the real causes were the NULL authorization_flow (the
NoneType.slug crash) and the email scope mapping returning a bare string rather
than a dict. Record the four verified causes and fixes, point to the repair
script and the new landing spec, and drop the unverified caveat.
2026-10-20 20:40:15 -07:00
Frank 200039d589 test(e2e): cover Grafana relying-party SSO landing (spec 20)
13-sso proves an authenticated session reaches Grafana login page but never
follows the OAuth button, so the relying-party login could be dead (500, or an
email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives
/login/generic_oauth through the real Anubis+authentik round-trip and asserts a
live session (/api/user 200 with a non-empty email). Wire 20- into the standard
project (qa-e2e session).
2026-10-20 20:40:15 -07:00
Frank 91d9864bb0 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.

scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:40:15 -07:00
dank 980491668c Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53) from feat/alloy-root-journal into main 2026-10-20 20:05:00 -07:00
Frank e494617599 docs(alloy): document the root journal-read cutover
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
2026-10-20 19:50:15 -07:00
Frank 9809ed9c74 test(observability): pin the root alloy shape
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
2026-10-20 19:50:15 -07:00
Frank 634fef242b feat(alloy): run it as a root host-network service so it reads the journal
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.

As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.

deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
2026-10-20 19:50:15 -07:00
dank 2fe68e69af Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52) from fix/observability-vllm-loki into main 2026-10-20 19:39:38 -07:00
Frank 70dd7fbe6d docs(observability): correct current state and log-pipeline root cause
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
2026-10-20 19:28:20 -07:00
Frank 2626332828 feat(grafana): provision the Loki datasource
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
2026-10-20 19:28:20 -07:00
Frank eb4d9910cb feat(observability): add vllm-fleet rule group for frank-archon
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.

Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
2026-10-20 19:28:20 -07:00
dank 5cc2b3c25a Merge pull request 'docs(openwebui): TTS reflects live frank-tts wiring (wilwheaton, LAN IP)' (#51) from docs/openwebui-tts-voice into main 2026-10-20 18:20:04 -07:00
Frank 18a29c40d2 docs(openwebui): TTS config reflects the live frank-tts wiring
The Text-to-speech section recorded audio.tts base_url host.containers.internal
and voice narrator1. Re-keyed the live config DB 2026-10-20: voice wilwheaton
(narrator1 retired), api_base_url the canonical host LAN IP 192.168.0.143:7772.
7772 is published so host.containers.internal also reaches it, but the LAN IP
matches this page's own on-host law. Browser-kokoro example voice updated to the
current pack; the dated 2026-10-18 verification is left as history.
2026-10-20 18:18:01 -07:00
dank 73a7ec3fc1 Merge pull request 'chore(frank-tts): repoint narrator slots to live voice packs' (#50) from chore/narrator-voice-repoint into main 2026-10-20 18:12:52 -07:00
Frank 130e9d739f chore(frank-tts): repoint narrator slots to the live voice packs
Frank replaced the narrator1/2/3 packs with cloned character voices
(wilwheaton, vaati, dr.kern, garrickhagon, jeffhays). The three narrator
slots are now carried by real packs: narrator1=wilwheaton, narrator2=vaati,
narrator3=dr.kern.

- frank-tts.env DEFAULT_VOICE narrator1 -> wilwheaton (a voice-less request
  400'd on the dead name; wilwheaton is narrator1's successor).
- Docs: correct the /health census and the podcast-speaker references in
  Frank TTS.md and Open Notebook.md to the live packs. Dated 2026-10-18/19
  measurement logs are left as history.

Open Notebook's live dank-duo speaker profile was repointed over REST
(Frank->wilwheaton, Comrade->vaati) - app-side, not in this commit.
Bats fixtures keep narrator1 as a self-contained sample pack; untouched.
2026-10-20 18:11:40 -07:00
dank 854890a155 Merge pull request 'chore(open-notebook): admin-side positioning + single-tenant note' (#49) from chore/open-notebook-admin-tenant into main 2026-10-20 17:44:45 -07:00
Frank d9f3d1be1e chore(open-notebook): position as admin-side; document single-tenant
open-notebook has no user accounts upstream (PasswordAuthMiddleware = one
shared password; SurrealDB has no user/account/owner table — verified in the
running image). One shared password = one shared workspace; the guard chain
gates access but cannot partition data, and Authentik identity is never
consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC
declined (#607). So move the homepage card from the user grid to the admin
tier and add the single-tenant boundary to the vault page. Fix the card's
stale 'frank-zealot' -> frank-archon. Update the seam's incidental category
pin to the new intentional value. Password rotated on the host (not repo).
2026-10-20 17:38:22 -07:00
dank afa3a4521c Merge pull request 'docs(authentik): record 7-day login-session TTL + setter script' (#48) from chore/authentik-session-ttl into main 2026-10-20 16:57:29 -07:00
Frank 9cc8ca1c88 docs(authentik): record the 7-day login-session TTL and add a setter script
The login-session lifetime (raised to days=7 on 2026-10-20) is owned by the
default-authentication-login stage's session_duration, not a global setting and
not SESSION_COOKIE_AGE. Authentik exposes no SESSION_SAVE_EVERY_REQUEST, so the
window is absolute. session_duration is a STRING idiom; a timedelta stores a form
timedelta_from_string rejects, crashing login. Add scripts/authentik-set-login-session-duration.sh
that validates the form before writing (DB state, not repo-managed, re-run after
a blueprint re-import). Trim the page back under the OKF 500-line cap. tests/run.sh green.
2026-10-20 16:55:44 -07:00
dank 9c05c40aa0 Merge pull request 'docs(agent): scaffold agent-skills config in the vault' (#47) from chore/agent-skills-vault-setup into main 2026-10-20 16:11:47 -07:00
Frank 63219787af docs(agent): scaffold the agent-skills config as three vault pages
Point the mattpocock-lineage skills (triage, to-tickets, to-spec,
wayfinder) at repo facts instead of a parallel docs/ tree: Gitea is the
issue tracker (credential-safe curl shape, token-only writes), the five
canonical triage roles keep their default label strings (created on the
tracker today), and GLOSSARY.md/docs/adr/ stay lazy with this vault
owning operational facts. AGENTS.md gains the ## Agent skills pointer
block; indices regenerated.
2026-10-20 16:09:30 -07:00
Frank e914ecbca6 docs(pi-hole): the record-load trap rides a container restart, not a boot
The 2026-10-19 write-up inferred a host restart from FTL's 16:25 start line. It was wrong: uptime -s reads 2026-10-18 01:24 (two days up) and the user journal shows 'Stopping pihole.service' at 16:25:23 with StartedAt 16:25:27 — a plain container restart inside the open-notebook deploy window. That moves where the verification belongs: after every pihole restart, not just after a reboot.
2026-10-20 10:56:39 -07:00
Frank 1eebf3b129 docs(netns): the by-name path to frank-dragoon is verified from the consumer
open-notebook resolves frank-dragoon-halogen to 10.89.1.39, GET :8731/v1/models answers, and the engine logs the app's POST /v1/embeddings 200. That makes the 2026-10-19 dial census a pre-move record: pasta's forwarding rule is unchanged, the endpoint was moved out of pasta, so the 'menu is frank-tts + frank-archon' line is now past tense. Frontmatter date was still 2026-09-02 against October content; bumped.
2026-10-19 22:48:44 -07:00
Frank af5a1f69a2 docs(pi-hole): FTL can boot with the record file mounted and serve none of it
The 2026-10-19 reboot left every franksdank.cloud name NXDOMAIN for LAN clients while external names kept answering from cache; the file was mounted and readable and dns.hosts was intact, and 'pihole restartdns reload' restored every name in seconds. Mechanism not established. Instrument and remedy now on the owner page, plus a failure-mode row for the FTL rate-limit case seen beside it (10.0.2.x = a pasta container's forwarder).
2026-10-19 22:46:57 -07:00
Frank 0930510f9b docs(vault): the embedding lane has a consumer, and two shapes that bite
open-notebook's embedding default now points at qwen3-embedding-0.6b on frank-dragoon's XDNA2 lane, proven through the app's own paths (model test 1024 dims, worker 'Successfully embedded', vector search ranking two unrelated notes 0.659/0.292 and 0.906/0.318). Records the two app-side traps: defaults store the model ID and a name is accepted then fails in the worker, and POST /api/embed 500s on an upstream RecordID typing bug after the vector is already written. frank-dragoon's lane section moves from 'off until the mount fix' to armed-and-serving with the boot-ledger contract and the measured idle RSS; the page's stale 'not on dankclaw_default' reachability claims are retired.
2026-10-19 22:46:57 -07:00
dank af327bcc5e Merge pull request 'feat(frank-dragoon): open the NPU node the way rootless can actually reach it' (#46) from feat/frank-dragoon-npu-lane into main 2026-10-19 22:35:49 -07:00
Frank fcf94562d1 feat(frank-dragoon): open the NPU node the way rootless can actually reach it
The armed boot reached the engine and died in its own preflight:

  halogen npu: /dev/accel/accel0 is here, but this process cannot open it
    (crw-rw---- nobody:nogroup).

The node is root:render 0660 and a rootless container maps exactly one uid/gid,
so `render` is unmapped inside it and `--group-add keep-groups` — the vendor's
advice, and the ComfyUI pattern for /dev/kfd — cannot grant a group the
namespace does not map. Measured 2026-10-19 from a `dank` shell already in
`render`: without the flag denied, with the flag denied, with the udev mode
0666 the open succeeds (OPEN_OK). Same wall 99-comfyui-gpu-access.rules already
solved for the GPU compute nodes, so the accel node gets its counterpart:

  dankclaw/udev/98-halogen-npu-access.rules — KERNEL=="accel[0-9]*",
  SUBSYSTEM=="accel", MODE="0666"

The deployer installs the rule, reloads rules, re-triggers the node and only
then checks the mode; the serve script refuses any node that is not
other-writable, naming the rule, so this cannot return as a crash loop (189
restarts while chat was down is the failure mode being closed).

Both checks read the permission bit directly: `[ -o FILE ]` is not
"other-writable" — it means "exists" and `-o` doubles as the OR operator inside
test, which is why the first cut of the gate refused everything.

Dragoon seam: 33 -> 35 tests. The fixture node is 0666 like the installed one,
a 0660 node is refused with the rule named and nothing rendered, and the rule
file's shape plus the deployer's install/trigger are asserted.
2026-10-19 22:35:06 -07:00
dank 861920797d Merge pull request 'fix(deploy): the dragoon gate reads the deployer's own payload-dir twin' (#45) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:49:16 -07:00
Frank 3ef711f146 fix(deploy): the dragoon gate reads the deployer's own payload-dir twin
The containment gate added in #44 parsed scripts/frank-dragoon.env through a
$CHECKOUT variable that does not exist in deploy-remote.sh, so the very next
deploy died at `set -u` before it touched the unit:

  ./scripts/deploy-remote.sh: line 1445: CHECKOUT: unbound variable

deploy-remote.sh runs from the checkout root and every other path in that block
is a parity-twin default (LLM_DRAGOON_CKPT/_VISION/_TOK_DIR), so the payload dir
gets the same treatment: LLM_DRAGOON_MODEL_DIR, defaulted to the deployed path,
compared directly. The seam now asserts that twin against the env file next to
the NPU twins it already checks, and the containment comparison itself is
verified both ways (inside accepted; the old outside path and a shared-prefix
sibling both refused).
2026-10-19 21:48:47 -07:00
dank 823fd52d0e Merge pull request 'fix(frank-dragoon): stage the NPU lane inside the payload mount' (#44) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:46:25 -07:00
Frank bda011d26f docs(vault): the NPU lane's one door into the container, and the pin that landed
frank-dragoon's lane section: the staging dir must live inside the payload dir
(the image ships no /models, so a nested :ro bind has no mountpoint to land on),
the entrypoint's rejection of HALOGEN_NPU_DIR, and the fabric-clock paragraph
corrected — the vendor's root host unit is installed and reports `held (manual)`
as of today, so "nothing is held" is retired along with the hand-write it
replaced.

dankclaw-iommu-translation-off: the deferred `pp_dpm_fclk` pin is no longer
deferred; the fact is owned by [[frank-dragoon]] and this page links it.
2026-10-19 21:43:36 -07:00
Frank b19f1c49d6 fix(frank-dragoon): stage the NPU lane inside the payload mount
The lane's first boot died at container init, not at the engine:

  runc create failed: ... create mountpoint for /models/npu mount:
  mkdirat .../merged/models/npu: read-only file system

The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").

Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.

Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
2026-10-19 21:42:31 -07:00
dank b1ca4e02fa Merge pull request 'feat(frank-dragoon): run the XDNA2 NPU lane beside the flash model' (#43) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:28:41 -07:00
Frankandomp 7f65cf0252 docs(vault): the pasta endpoint has no address an app container can use
The 2026-10-19 legs table blamed the frank-dragoon refusal on a root/rootless
netavark rule-set split. It is not that: the container is rootless (the root
unit's User=dank runs podman run), it is on pasta, and pasta installs no DNAT
for rootless-bridge traffic while refusing a second network outright. The
gateway alias does reach it (200) and is link-local, which is the same dead end
the Open Notebook URL guard hit. Measured with the move onto the bridge now in
the deployer, including the /proc/net/tcp line that shows 8730 is loopback-bound
and so stays private after the move.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:27:07 -07:00
Frankandomp 346e1d22f4 feat(frank-dragoon): put the endpoint on the app bridge so app containers can name it
An endpoint on podman's rootless default (pasta) has no address another
container can use, measured on 2026-10-19 from inside open-notebook:

  - 192.168.0.143:7770            connection refused (pasta forwards the
                                  published port in the host netns; no netavark
                                  DNAT exists for traffic arriving from a
                                  rootless bridge)
  - host.containers.internal:7770 200, but the alias resolves to 169.254.1.2 —
                                  link-local, which open-notebook's URL guard
                                  refuses to store
  - frank-dragoon-halogen:8731    name unknown: different netns
  - --network pasta --network dankclaw_default
                                  "cannot set multiple networks without bridge
                                  network mode" — pasta cannot be added to

So reaching the NPU endpoints from an app container means moving the container
onto dankclaw_default (FRANK_DRAGOON_NETWORK; empty keeps today's shape). This
is not an exposure change: the engine's unauthenticated token port binds the
container's own loopback (/proc/net/tcp: 127.0.0.1:8730 against 0.0.0.0:8731),
so only the API port becomes name-resolvable — the same trust the published
7770 already carries on the LAN. The deployer refuses a network that does not
exist rather than starting a unit that restart-loops on it.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:27:07 -07:00
Frankandomp 53ef121180 feat(frank-dragoon): run the XDNA2 NPU lane beside the flash model
halogen-flash-server already ships an NPU path on the pin we run: for each
model named in HALOGEN_NPU_MODELS it loads the vendor's precompiled AIE
programs (XDNA2 executes only AIE-compiled graphs) and answers them on
/v1/embeddings inside the SAME container behind port 7770. That gives the
fleet an in-house embedding endpoint without touching the iGPU trunk.

The lane is opt-in and one line wide: HALOGEN_NPU_MODELS blank is the
pre-lane endpoint byte for byte (no device, no mounts, no engine env).

The three host preconditions the vendor names are gated in the serve script
and mirrored in the deployer (the 2026-10-01 twin law), each refused by name:
  - the staged model files. /models is read-only and HALOGEN_DOWNLOAD stays
    unset so the container opens no outbound connection at boot, so
    scripts/download-frank-dragoon-npu.sh stages them from the pinned
    revision with the catalog's own byte counts and sha256s (the engine
    re-checks both at boot);
  - the distro XRT's libxrt_coreutil/libxrt_core/libxrt_driver_xdna, each
    mounted twice — where the image looks for XRT and at its own path —
    because those libraries find each other where they were installed and
    an /opt/xilinx/xrt of symlinks dangles inside the container;
  - the GPU fabric clock HELD. On Strix Halo the iGPU and the NPU co-running
    hang the machine (and corrupt NPU results) while FCLK changes speed, and
    halogen refuses to start in that state; it cannot read host sysfs from
    inside, so the hold is proven host-side. dankclaw/host/ carries the
    vendor's halogen-fabric-clock{,.service} verbatim at tag v0.16.2 (sha256
    a3a4f7f4e0142e71… / 5f9f1cc4c85a1c02…); the deployer installs, enables
    and gates it only while the lane is named, and releases the clock plus
    drops the boot hook when it is not.

After health, the deployer's lane gate is a real embedding through
/v1/embeddings — a 200 on /health proves nothing about the NPU programs.

Planner: dankclaw/host/halogen-fabric-clock* re-dispatches frank-dragoon
(the deployer installs it); dankclaw/host/grub and the fetch script plan
nothing.

Seam: 11 new tests — the off arm really renders nothing lane-specific, the
on arm renders the device/mounts/env, every missing precondition refuses by
name, the deployer twins and the host unit shape are pinned.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:22:50 -07:00
dank fa093e617d Merge pull request 'docs(vault): halogen NPU lane on the current pin, and the greedy-schema law' (#42) from feat/open-notebook-portal into main 2026-10-19 21:14:52 -07:00
Frank 6215f1a6c8 docs(vault): halogen has an NPU lane on the pin we already run, and it is greedy-only
- frank-dragoon: measured the response_format contract. json_schema and json_object answer
  schema-valid JSON at temperature 0 and draw a 400 at any sampled temperature, and because
  the sampler default is server-side an absent temperature is sampled. The page claimed
  "response_format is unsupported", which sends the next agent past a working feature.
- frank-dragoon: new "The NPU lane" section. The pinned 0.16.2 image already ships the
  five-program AIE catalog including qwen3-embedding-0.6b, so /v1/embeddings is one env var
  plus the accel device, the distro-XRT mount and the fabric-clock hold; embeddinggemma-2
  needs 0.18.3+, the stock-kernel-driver memory-budget fix 0.18.8+. Host census recorded:
  no /opt/xilinx/xrt, distro XRT 1:2.21.75 with libxrt_driver_xdna, clocks unheld.
- Open Notebook: scriptwriter probe verdict — greedy-only, and open-notebook's model record
  is {credential, name, provider, type} with no per-row config, so it cannot ask for greedy;
  frank-archon stays the writer. The embedding gap now has a named door instead of "build a
  new llama.cpp slot".
- Frank TTS: "the NPU is not a door at all" was too broad. It is not a door for the TTS
  family; halogen's precompiled catalog is a door on the same device.
- System Specs: NPU runtime row corrected to the distro XRT actually installed, and the
  stale "frank-zealot serves 27B" line retired in favour of [[frank-zealot]] as sole owner.

Tests: ./tests/run.sh, 455 passed.
2026-10-19 16:40:48 -07:00
dank 8bac97ec87 Merge pull request 'feat(open-notebook): guarded vhost, single origin, live homepage card' (#41) from feat/open-notebook-portal into main 2026-10-19 16:24:11 -07:00
Frank 69739237c8 docs(open-notebook): record the deployed shape and what the first render proved
The page described an undeployed service whose scriptwriter was going to be
frank-zealot. Both halves are now false, and the reason is worth more than the
correction.

- Status, ports, coordinates: deployed 2026-10-19, guarded vhost, API published
  on loopback only. The first episode is the proof: 747 s of speech in 301 s of
  wall clock (2.5x real time, knobs at 1), download route 200 with the bearer
  and 401 without.
- New section for the guard chain: provider-before-route, one upstream because
  the image rewrites /api/* itself, and the API_URL law — autodetect appends
  :5055, which is seerr's port on this host, so an unset API_URL yields a UI
  that loads and an app that cannot call itself.
- Provisioned records table (credentials, model rows, dank-duo, dank-daily) and
  the two contract traps measured on the live API: generate takes profile
  NAMES, and the fleet has no /v1/embeddings, so every note save fails its
  embed_note command while renders from content work fine.
- Why frank-archon writes the scripts: frank-zealot is reachable from this
  netns only at 169.254.1.2, and open-notebook's URL guard refuses to store a
  link-local base URL with no override. Census lands on [[Container Network
  Namespaces]], which also gains the root-vs-rootless publish rule — a
  root-published port is invisible to a rootless container, so "publish the
  port" was the wrong remedy as written there.
- Frank TTS: the request's model field is ignored; only voice selects a pack.

Tests: ./tests/run.sh, 455 passed.
2026-10-19 16:22:38 -07:00
Frank e5580abb6f feat(open-notebook): put the podcast generator behind the guard chain
The service came up internal-only: UI on the LAN at 8502, API at 5056, both
published on every interface, with only its own shared password in front. That
put the episode library outside the fleet's auth chain and one port scan away
from anything on the network.

- Caddy: guarded vhost notebook.franksdank.cloud (Anubis -> Authentik; provider
  notebook-proxy, internal_host caddy:8017) plus the post-auth route to the
  published UI port. One upstream is enough, because the image's own Next.js
  layer rewrites /api/* to the FastAPI beside it — so the API needs no outward
  face at all, and 5056 moves to loopback, where only the deployer's health gate
  dials it.
- scripts/open-notebook.env: API_URL and CORS_ORIGINS become the single guarded
  origin. Left to itself the app auto-detects <host>:5055, the port seerr holds
  on this host; a LAN-origin API_URL would also take the browser around the
  guard chain, where its authenticated session does not ride.
- addn-hosts: the LAN A record. The homepage card goes live and links the vhost.
- Seam: the API origin is derived from whatever the Caddyfile routes to 8502, so
  renaming the vhost moves the test instead of leaving it blessing a stale name,
  and a new test proves the vhost hands the WHOLE request to Anubis.
2026-10-19 16:13:53 -07:00
dank ee6d8610b3 Merge pull request 'fix(open-notebook): the database probe has to be exec-form JSON' (#40) from feat/open-notebook-portal into main 2026-10-19 16:07:53 -07:00
Frank 933420fd29 fix(open-notebook): the database probe has to be exec-form JSON
podman wraps a bare HealthCmd as CMD-SHELL (measured on Dankclaw's podman 5.7:
--health-cmd "/surreal is-ready" stores {"Test":["CMD-SHELL","/surreal is-ready"]}),
and the pinned SurrealDB v2 image is distroless — no /bin/sh, not even `ls`.
So the probe could never run: every 30 s interval exited 1 in ~30 ms with empty
output while the database was listening and answering, and open-notebook's first
deploy died at the gate ("did not become healthy in 90s"). The hand-check
`podman exec … /surreal is-ready` printed OK, which is why the old comment
believed the probe worked.

The JSON array survives the Quadlet generator (--health-cmd "[\"/surreal\",\"is-ready\"]")
and stores ["CMD","/surreal","is-ready"], which execs directly: a throwaway
instance of the pinned digest went healthy on that form. The seam is re-pinned
to the shell-free contract; the old assertion forbade quotes in any HealthCmd,
which is what pinned the broken shape.
2026-10-19 16:07:28 -07:00
dank 9ffa99902d Merge pull request 'fix(frank-archon): the state-dir guard names what the directory is now' (#39) from fix/frank-archon-state-dir-comment into main 2026-10-19 15:02:01 -07:00
Frank b8f564f3bd fix(frank-archon): the state-dir guard names what the directory is now
The refusal message still named "the /data state dir"; /data left with the
prefix disk tier. The serve script mounts /models ro, the rad-tune cache and the
chat template only, and tests/frank_archon_unit.bats asserts no :/data mount
exists, so the line an operator reads at the moment a boot refuses pointed at a
mount that is gone.

Text only. The guard still refuses: the block was run standalone with the
variable unset and every neighbour set - it prints the new message and exits 1.
bash -n clean, ./tests/run.sh green (454 tests).
2026-10-19 15:02:01 -07:00
dank d42d053abe Merge pull request 'docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape' (#38) from docs/frank-archon-181-two-slot into main 2026-10-19 14:56:59 -07:00
Frank 7eed636bd1 docs(perf): destroy §9's vendor absolutes, keep the ratios
The t/s ladder and the TTFT delta triple measure a configuration neither
deployed nor compared; per the purpose test they are dead weight. What survives
is the knob pricing (halving the shared chunk costs roughly a fifth of prefill
throughput), the long-prompt-only shape of the span win, and the method. Run-on
now names the era the census was taken in.
2026-10-19 14:55:43 -07:00
Frank d9b82b372e docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape
The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the
--vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and
bats carried the new shape while the page and the AGENTS.md block still
described the 1.0.13 3-slot era, so every point-of-use number was wrong.

Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not
from the stale pre-#31 draft: runtime block (digest, version oracle, the two
CLI breaks that are boot refusals), shape table (demand-signal rows, derived
VRAM split, restored spec/batch row, host-only prefix tier, step sharing),
container contract (no /data mount), the measured 2-slot ledger, a new Eras row
for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and
the laws: pool sizing and the per-card budget re-scoped to the pair era, a new
resolver law (funds ~90% of the worst case, host RAM below a slab step is inert,
MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the
fix ordered-not-verified, the knob-availability law flipped, and the slots law
now pricing the third slot in KV (4.20 -> 2.76 GiB).
2026-10-19 14:55:43 -07:00
dank 3d10a4ca4f Merge pull request 'docs+fix(frank-zealot): the adapter's measured effect, and a unit description that stopped lying' (#37) from feat/zealot-uncensored-lora into main 2026-10-19 14:47:59 -07:00
Frank e5b731216c fix(frank-zealot): the unit description still described the 27B era
`systemctl status` said `2x262144 slots over one 524288 pool, unquantized f16
K+V, MTP off` — three payload eras and two reslices out of date, and still
wrong about the thing that decides whether the endpoint can be trusted (the KV
dtype and whether the drafter is armed). It now quotes the env: 3x200704 over
one 602112 pool, f16 K + q8_0 V, MTP armed, adapter, Q8_0 vision.

Touching the unit selects the service in the planner, so this deliberately does
NOT come with a deploy: the hub is serving live conversations. The revision
marker stays behind until the next real frank-zealot change.
2026-10-19 14:47:30 -07:00
Frank ce8d82d4a7 docs(frank-zealot): record what the adapter did, and where provenance is read
Before/after on the two probe prompts (both opened `I cannot provide…`; neither
now carries a marker — one ran to its 2048-token cap, the other finished at
1930), the residual `Important Disclaimer` heading named for what it is (style,
not the circuit), and the same-boot proof that chat/tools/vision still answer.
Provenance now points at the boot ledger: the fork prints every
`prism.abliterate.*` key at each start, so the PTQ1_0-vs-PQ2_0 mismatch is
visible to anyone who boots the service — the page links that instrument
instead of restating it.
2026-10-19 14:47:30 -07:00
dank 02d04bc1f6 Merge pull request 'feat(frank-zealot): apply the refusal-direction adapter (rank-1 LoRA) at load' (#36) from feat/zealot-uncensored-lora into main 2026-10-19 14:43:26 -07:00
Frank 428ac22736 Merge remote-tracking branch 'origin/main' into feat/zealot-uncensored-lora 2026-10-19 14:43:06 -07:00
Frank 5d5cac4da3 docs(frank-zealot): the adapter is a payload row, a shape line, and two laws
- Shape table: an Adapter row (bytes, sha, what it edits, and the PTQ1_0
  provenance asymmetry), the fetch line now names both fetchers, the flag list
  carries --lora-scaled.
- One paragraph: the adapter changes behaviour, not the budget equation, and the
  probe that proves it must budget >= 2048 tokens because the thinking channel
  eats a short budget whole and returns empty content with finish_reason=length.
- Eras: the new era's rollback is one env line on a file already loaded.
- Laws: provenance is metadata, not a gate (tensor names, not hashes; an
  embedding rank would crash, which is why the publisher left token_embd alone);
  the fork pin now has a third reason, since older builds refuse the adapter with
  `unsupported prism.hadamard.version: 2`. Base-alignment law updated: the 9B
  refusal escape hatch was a fetch, and it has now been taken.
- AGENTS.md: topology payload cell and the zealot operator block.
2026-10-19 14:42:56 -07:00
Frank 3eaba57925 feat(frank-zealot): apply the refusal-direction adapter at load
Frank ordered the adapter; this adds it to the shape as one knob and one flag.

- `LORA_FILE` / `LORA_SCALE` in the env, `--lora-scaled <file>:<scale>` as ONE
  argv token so a scale can never reach the engine without its adapter. The
  rollback follows the mmproj convention exactly: `LORA_FILE=` (explicitly
  empty) drops the flag entirely, no rebuild, and `LORA_SCALE=0` is the card's
  own "inert but present" value.
- New fetcher `scripts/download-frank-zealot-uncensored-lora.sh`: `-f`, byte
  count, sha256, `--verify-only`, idempotent. The file is NOT LFS (a plain git
  object), so the pinned sha is the content hash, not an LFS oid.
- The deployer gates on the adapter beside the pack and the projector, and its
  error names the ADAPTER's fetcher — sending an operator to the bonsai9 script
  for a 3 MB file is how a payload ships without it.
- Seam: path+scale pinned as one token, the scale knob including 0, absence on
  the empty opt-out, a refusal to boot on a missing adapter, and the env ↔
  deployer anti-drift pair extended to LORA_FILE/LLM_ZEALOT_LORA. 454/454.

Provenance, stated where it will be read: the adapter's own metadata names
`Ternary-Bonsai-2-9B-PTQ1_0.gguf` (3561df7a…) as the pack the direction was
fitted against. This endpoint serves the PQ2_0+MTP+Q8_0 packing (c79fcdfa…),
which is not that file and is not on this host. The loader matches tensor names,
not hashes, so it applies — and build 10777 is the build the card verified on —
but the strength is calibrated on a sibling quantization. The proof is therefore
the refusal probe against the live endpoint, captured before and after, and not
the card's table.
2026-10-19 14:42:56 -07:00
dank 9b9c32005f Merge pull request 'docs(frank-tts): the engine computes on the gfx1151 now; the NPU question is closed' (#35) from docs/frank-tts-igpu-followup into main 2026-10-19 14:38:33 -07:00
Frank ea2fc6440b docs(frank-tts): record the iGPU cutover, close the NPU question
The cutover PR merged the runtime; this carries the surface that has to be
true afterwards.

- Engine page: the shape is now `--device Vulkan0` on the gfx1151 as a
  co-tenant beside frank-dragoon; the registry/thread/streaming laws gain a
  fourth — the device law — whose numbers stay in the verdict report, and the
  deploy record gets the cutover steps including the `glslc` configure gate.
  The CPU-era arm/rollback prose keeps the qualitative argument and drops the
  dead numbers per the purpose test.
- Endpoint page: "Where the engine computes" now says iGPU + co-tenancy, and
  states the NPU verdict in one paragraph instead of a research tour.
- Quadlet comment: it claimed supplementary groups are what grant render
  access. On this host the udev rule world-writes the render nodes and the
  container's `id` shows no groups at all, so keep-groups is belt-and-braces;
  the RSS note now matches the device shape instead of the CPU trial.
- Verdict report: gate 2 (contention) recorded as measured after the cutover,
  and the build-stage prediction corrected — the package is `glslc`, not
  `libvulkan-dev glslang-dev`.
- index.md regenerated (one description changed).
2026-10-19 14:37:29 -07:00
dank 6248f02ca2 Merge pull request 'feat(frank-tts-engine): synthesize on the gfx1151 iGPU (Vulkan), CPU stays as the rollback' (#34) from feat/frank-tts-vulkan into main 2026-10-19 14:27:11 -07:00
Frank 25df9d05a2 feat(frank-tts-engine): synthesize on the gfx1151 iGPU via Vulkan
The NPU path is a dead end for this payload (verdict:
report/tts-npu-spike-verdict-2026-10-19.md); the iGPU is the accelerator that
exists on this box today, and it is already shared with frank-dragoon.

- Containerfile.engine: -DGGML_VULKAN=ON, plus the three packages the door
  needs to configure — libvulkan-dev, spirv-headers, and glslc, which Ubuntu's
  glslang-tools does NOT ship (find_package(Vulkan COMPONENTS glslc REQUIRED)
  fails without it) — and libvulkan1 + mesa-vulkan-drivers (the RADV ICD) at
  runtime. No NVIDIA ICD on purpose: one Vulkan device in the enumeration is
  what makes the NAME Vulkan0 safe to write down.
- tts-engine-start.sh: QWENTTS_DEVICE -> --device, default Vulkan0. A value
  that is set but EMPTY passes no flag at all, which is the CPU rollback; that
  is why the default is written as an -v test and not with :=, because := also
  fires on an empty value and would silently undo the rollback.
- quadlet: AddDevice=/dev/dri + GroupAdd=keep-groups (a rootless container that
  drops its supplementary groups enumerates no render device). The node is not
  named: this host is hybrid, renderD128 is the NVIDIA card and renderD129 the
  amdgpu iGPU.
- env: QWENTTS_DEVICE=Vulkan0 plus the device law in the header, including the
  rollback line.
- tests: the two CPU-only assertions now assert the Vulkan shape, and a new one
  pins the rollback (empty device => no --device in argv). 451/451 green.

Measured before writing any of it: the configure in the digest-pinned base
answers "Found Vulkan: ... found components: glslc" and "Including Vulkan
backend"; gate 1 (RADV GFX1151 visible to a rootless container off this base)
is recorded in the verdict report.
2026-10-19 14:26:58 -07:00
dank 7601edfe28 Merge pull request 'docs(LLM): the decode stall a prefill causes, and the knobs that turn it' (#27) from docs/radiance-prefill-decode-census into main 2026-10-19 14:21:45 -07:00
Frank e133d2412d Merge origin/main into docs/radiance-prefill-decode-census
Two conflicts, both from the 2026-10-19 vault compaction on main:

* frontmatter description — main shortened it to satisfy OKF DESC
  (max 120 chars); this branch's longer wording would have failed the
  seam. Kept main's shape and folded the new §9 in by naming
  step-shape attribution instead of profiler attribution.
* the Related bullet for [[vLLM Porting and Compile Triage]] — that
  page was destroyed on main, so the link is dropped rather than
  resurrected; this branch's [[frank-archon]] row is kept.

Index rows regenerated with scripts/gen-vault-index.py.
2026-10-19 14:20:05 -07:00
Frank e3a02e25c7 docs(LLM): census the decode stall a prefill causes, and the knobs that turn it
Live step-shape census on frank-archon (radiance 1.0.13, boot of 2026-10-15):
decode-only steps run 18.7 ms, prefill+mixed steps 317 ms — 60 % of engine
wall-clock inside 8 % of the steps — and a day of Prometheus shows decode at
13.3 t/s median while prefill runs >=1000 t/s against 106-240 t/s when it does
not. The lag test says the penalty is confined to the prefill window: no
lingering expert-residency tail, and preemptions are zero.

Adds the method as LLM Performance Diagnosis S9 (counter arithmetic, the lagged
bucket correlation that separates step contention from residency churn, and the
four radiance knobs with their published costs), and the endpoint's own numbers
plus the verdict as two frank-archon laws: --busy-prefill-chunk and
--prefill-interleave are absent from this pin (proven via the running
container's --help), so MNBT is the only crank here and an image bump is gated
by the 1.2.0 numerics dead end.

No behavior change; docs only.
2026-10-19 04:11:20 -07:00
200 changed files with 23830 additions and 1195 deletions

No files matched your search

+67 -31
View File
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -166,19 +166,22 @@ bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
dashboard are provisioned from `shared/grafana/provisioning/`, and
`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
+ recording rules, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
without `--metrics`. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.
The former structural target downs (HTML-scraped "probes", Gitea metrics off)
are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
every Prometheus job reports up. `frank-zealot` was down for a different reason
until 2026-10-18: llama.cpp will not serve `/metrics` at all without
`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing
@@ -290,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
2. Bench against the service itself. No scratch ports on an LLM host — one
process owns the card. Use the runtime's own harness (`llama-bench`; a
live llama.cpp server also reports honest `timings` per request, including
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
`ignore_eos` run on the raw `/completion` path there answers **500 "The
model produced output that does not match the expected Content-only
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -417,16 +420,17 @@ disagrees with it is the bug.
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
drafter always armed (the adaptive default disarms it and halves decode);
server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
under-counts, so a harness window equal to the ceiling makes every long
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
**per-request ceiling** (a longer prompt is a 400, never a truncation);
`HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
pool positions buy concurrency, the per-request window never moves;
`HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
default disarms it and halves decode); server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
estimate under-counts (trigger is 95 %), so a harness window at the ceiling
makes every long request a guaranteed 400.
Reasoning arrives in `reasoning_content`; the
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
@@ -435,12 +439,16 @@ disagrees with it is the bug.
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
loader (the payload carries the fork's custom PQ2_0 tensor type plus
`scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests).
tower can crash the server with concurrent image requests) + the rank-1
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
— and the fork BUILD is also the gate that lets it load: anything older than
b10777 refuses it with `unsupported prism.hadamard.version: 2`.
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -458,12 +466,15 @@ disagrees with it is the bug.
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
+ 64 GiB disk, n-gram table pinned to RAM.
`stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk
tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal,
not a no-op.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
is why this is a root unit, and `ttm-preflight.sh` raises
@@ -730,6 +741,13 @@ linked runbooks.
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
says identity mode costs nothing, so the XDNA2 NPU returned for free.
[[dankclaw-iommu-translation-off]]
- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
`systemctl --user start` on an already-active unit does nothing, so installing
a new digest under a running service keeps the OLD container serving while the
deploy, its health gate and the Playwright suite all pass — the authentik
handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
outpost, the one handler that restarted). A pin change is applied by
`restart`, never `start`.
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -789,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -796,6 +816,20 @@ dies, its page is destroyed. Rollback is a commit, not a document.
`serverAppConfigs-docs/index.md` is the locator for every page that survives;
retired scripts live in `old/` and `.AGENTTRASH/`.
## Agent skills
### Issue tracker
Issues live in Gitea (`dank/serverAppConfigs` on Dankclaw), via its REST API with the stored credential. See `serverAppConfigs-docs/Agent/Issue Tracker.md`.
### Triage labels
The five canonical roles keep their default strings (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `serverAppConfigs-docs/Agent/Triage Labels.md`.
### Domain docs
Single-context: root `GLOSSARY.md` + `docs/adr/`, created lazily; this vault stays the owner of operational facts. See `serverAppConfigs-docs/Agent/Domain Docs.md`.
## Why
Direct edits on the remote drift from version control, and future agents
@@ -816,5 +850,7 @@ history.
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. **Not yet deployed on Dankserver: the live quadlets still bind the array read-only — see [[Media Metadata View]].** |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
+29
View File
@@ -162,6 +162,25 @@ marreta.franksdank.cloud {
}
}
# --- Open Notebook (guarded: Anubis → Authentik; notebook-proxy provider, internal_host caddy:8017) ---
# The podcast generator. ONE upstream carries the whole app: the image's own
# Next.js layer rewrites /api/* to the FastAPI beside it (INTERNAL_API_URL,
# default http://localhost:5055 inside the container), so a browser on this
# vhost never needs the API port. That is why scripts/open-notebook.env sets
# API_URL to exactly this origin — left unset, the app auto-detects
# `<host-you-opened>:5055`, and 5055 on this host belongs to seerr — and why
# the published API port sits on loopback. App-level login
# (OPEN_NOTEBOOK_PASSWORD) stays on behind the chain: two owners, edge identity
# and the app's own session.
# Dialled by published LAN port, not container name: open-notebook is a
# rootless quadlet and this Caddy lives in root's storage, so root's DNS does
# not see the container (the same reason marreta and SearXNG are dialled by IP).
notebook.franksdank.cloud {
reverse_proxy anubis:8923 {
header_up X-Real-Ip {remote_host}
}
}
# --- Pi-hole (quadlet on slirp4netns; published 8083) ---
pihole.franksdank.cloud {
basicauth {
@@ -420,6 +439,16 @@ http://:8017 {
handle @marreta {
reverse_proxy 192.168.0.143:8082
}
@notebook host notebook.franksdank.cloud
handle @notebook {
# Bare proxy, like ai.franksdank.cloud: the app's SSE answers
# (/api/search/ask, chat sessions) stream for minutes through this
# same listener with no flush or timeout tuning, so nothing here
# needs it — a read/write timeout on the route would be the thing
# that cuts them.
reverse_proxy 192.168.0.143:8502
}
}
# --- Optional: Jackett ---
@@ -23,7 +23,7 @@ Network=dankclaw_default
# Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
# the tag's nested manifest list, so pin the single-platform digest).
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
Environment=AUTHENTIK_INSECURE=false
Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}
@@ -20,7 +20,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-outpost
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=authentik-postgres
Image=docker.io/library/postgres:16-alpine
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
Environment=POSTGRES_DB=authentik
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
[Container]
ContainerName=authentik-server
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=server
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -16,7 +16,7 @@ Wants=authentik-server.service
[Container]
ContainerName=authentik-worker
Image=ghcr.io/goauthentik/server:2026.8.3
Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
Exec=worker
Network=dankclaw_default
EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -15,7 +15,7 @@ After=network.target
[Container]
ContainerName=gitea-postgres
Network=gitea.network
Image=docker.io/library/postgres:16-alpine
Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
EnvironmentFile=/home/dank/Containers/gitea/env
Environment=POSTGRES_USER=gitea
Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
[Container]
ContainerName=gitea-server
Network=gitea.network
Image=docker.io/gitea/gitea:latest
Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
EnvironmentFile=/home/dank/Containers/gitea/env
Environment=DATABASE_TYPE=postgres
Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
Environment=DATABASE_PORT=5432
Environment=USER_UID=1000
Environment=USER_GID=1000
# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
# the container; the only route in is this host's published 3000, and the
# exposed surface is counters only — same posture as the LLM /metrics.
Environment=GITEA__metrics__ENABLED=true
Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
PublishPort=0.0.0.0:3000:3000
PublishPort=2222:22
Volume=/home/dank/Containers/gitea/data:/data
@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
[Container]
ContainerName=chrome-devtools-protocol
Network=host
Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
Entrypoint=/headless-shell/headless-shell
Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
ShmSize=1073741824
+65
View File
@@ -0,0 +1,65 @@
#!/bin/sh
# halogen-fabric-clock hold | release | status: the AMD GPU's fabric clock (FCLK), held at its top level.
#
# On Strix Halo the GPU and the NPU working at the same time can hang the machine, and corrupt the NPU's results,
# while the GPU's fabric clock changes speed; held at its top level, the two run together cleanly. halogen-flash-server
# with HALOGEN_NPU_MODELS refuses to start until the clock is held. This holds it: the performance level `manual` and
# only the top pp_dpm_fclk level, on every amdgpu device that has the control (the GPU and SoC clocks stay automatic).
# Cost: about 1 W at idle. Run as root; halogen-fabric-clock.service runs `hold` at every boot and after a resume.
#
# hold hold it (waits up to 60 s for the GPU driver to come up)
# release give it back to the driver (the performance level `auto`)
# status print each device's state; exit 0 when every device is held
set -u
SYS="${HALOGEN_SYSFS:-/sys}"
devices() {
for d in "$SYS"/class/drm/card*/device; do
[ -w "$d/pp_dpm_fclk" ] || [ -r "$d/pp_dpm_fclk" ] || continue
[ -e "$d/power_dpm_force_performance_level" ] || continue
echo "$d"
done | sort -u
}
held() { # held DEVICE_DIR: the performance level high, or manual with only the top fclk level selected
perf=$(cat "$1/power_dpm_force_performance_level" 2>/dev/null)
[ "$perf" = high ] && return 0
[ "$perf" = manual ] || return 1
top=$(grep . "$1/pp_dpm_fclk" | tail -n 1)
[ "$(grep -c '\*' "$1/pp_dpm_fclk")" = 1 ] && [ "${top%\*}" != "$top" ]
}
cmd="${1:-status}"
case "$cmd" in
hold)
n=0
while [ -z "$(devices)" ] && [ "$n" -lt 60 ]; do sleep 1; n=$((n + 1)); done
[ -n "$(devices)" ] || { echo "halogen-fabric-clock: no amdgpu device with a fabric clock control (pp_dpm_fclk) under $SYS/class/drm" >&2; exit 1; }
rc=0
for d in $(devices); do
lvl=$(grep . "$d/pp_dpm_fclk" | tail -n 1 | cut -d: -f1)
# the driver applies the level several seconds after the write, so the read-back waits up to 30 s
ok=0
if echo manual > "$d/power_dpm_force_performance_level" && echo "$lvl" > "$d/pp_dpm_fclk"; then
t=0; until held "$d" || [ "$t" -ge 300 ]; do sleep 0.1; t=$((t + 1)); done
held "$d" && ok=1
fi
if [ "$ok" = 1 ]; then
echo "halogen-fabric-clock: $d: held at level $lvl ($(grep . "$d/pp_dpm_fclk" | tail -n 1 | awk '{print $2}'))"
else
echo "halogen-fabric-clock: $d: could not hold it (level $lvl)" >&2; rc=1
fi
done
exit "$rc" ;;
release)
for d in $(devices); do echo auto > "$d/power_dpm_force_performance_level" && echo "halogen-fabric-clock: $d: released (auto)"; done ;;
status)
[ -n "$(devices)" ] || { echo "halogen-fabric-clock: no amdgpu device with a fabric clock control"; exit 1; }
rc=0
for d in $(devices); do
if held "$d"; then echo "$d: held ($(cat "$d/power_dpm_force_performance_level"))"
else echo "$d: NOT held ($(cat "$d/power_dpm_force_performance_level"))"; rc=1; fi
done
exit "$rc" ;;
*) echo "usage: halogen-fabric-clock hold | release | status" >&2; exit 2 ;;
esac
@@ -0,0 +1,19 @@
# Holds the AMD GPU's fabric clock at its top level at boot and after every resume, for halogen-flash-server with
# the NPU (HALOGEN_NPU_MODELS): on Strix Halo the GPU and the NPU at once are safe only with that clock held.
# Install (as root), from this directory:
# install -m 755 halogen-fabric-clock /usr/local/sbin/
# install -m 644 halogen-fabric-clock.service /etc/systemd/system/
# systemctl daemon-reload && systemctl enable --now halogen-fabric-clock.service
# Check: halogen-fabric-clock status. Undo: systemctl disable halogen-fabric-clock.service, then
# halogen-fabric-clock release (or reboot). The unit runs once at boot and once after each resume (a resume can hand
# the clock back to the driver); it holds nothing open, so it shows inactive between runs.
[Unit]
Description=Hold the AMD GPU fabric clock at its top level (GPU and NPU together)
After=systemd-modules-load.service suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/halogen-fabric-clock hold
[Install]
WantedBy=multi-user.target suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
@@ -23,7 +23,7 @@ After=gluetun.service network.target
[Container]
ContainerName=marreta_browser
Network=gluetun.container
Image=docker.io/lightpanda/browser:nightly
Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
[Service]
TimeoutStartSec=300
+1 -1
View File
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
[Container]
ContainerName=marreta
Network=gluetun.container
Image=ghcr.io/manualdousuario/marreta:latest
Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
# Relocate off SearXNG's 8080 inside the shared netns.
Environment=NGINX_HTTP_PORT=8085
Environment=APP_NAME=Marreta
@@ -33,7 +33,7 @@ Wants=network-online.target
[Container]
ContainerName=alertmanager
Image=docker.io/prom/alertmanager:latest
Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
Network=dankclaw_default
User=0:0
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
+35 -39
View File
@@ -1,55 +1,51 @@
# Quadlet unit: alloy (rootless, boot-persistent).
# Quadlet unit: alloy (ROOT system service, boot-persistent).
#
# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
# Unit name: alloy.service (container name: alloy)
# Install path on dankclaw: /etc/containers/systemd/alloy.container
# Unit name: alloy.service (container name: alloy) — SYSTEM manager, root.
#
# Log shipper. This replaces the compose service of the same name in
# dankclaw/compose.yaml ("Observability" block).
# WHY ROOT (not rootless, unlike its observability siblings). Alloy reads the
# host journal. The journal files are root:systemd-journal 0640 (other::---);
# a rootless container carries no supplementary groups and cannot open them —
# a rootless read of system.journal returns DENIED (see the Alloy page in the
# vault). Running as root, one loki.source.journal reads system.journal AND
# user-1000.journal, so it carries both host logs and the rootless Quadlet
# containers' logs. Validated 2026-10-20: a root probe read 31,825 journal
# lines and pushed 26,966 into Loki in 20 s, zero drops.
#
# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
# boot from reaching for that local tag at a registry.
# NETWORK=host. Loki publishes 127.0.0.1:3100 (loopback only) on the host, and
# no bridge container — rootless or root — can reach the host's loopback, so
# this unit runs on the host network and pushes to 127.0.0.1:3100. The side
# effect: the UI must bind loopback too, so Exec sets the listen-addr to
# 127.0.0.1 (the old rootless unit used 0.0.0.0 only because container loopback
# was not host loopback; under Network=host they are the same address, so the
# unauthenticated UI stays loopback-only — nothing off-host reaches it).
#
# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
# resolves the container name loki:3100.
# THE /etc/machine-id MOUNT IS LOAD-BEARING. sd_journal locates the journal
# directory by the local machine id; without it the container opens nothing and
# the journal source reads exactly 0 lines — silently, no error, health green.
# This mount is the entire reason the pipeline carries data. Never remove it.
#
# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
# refuses the ui block), so the compose service had no reachable health
# endpoint at all: the homepage status card for alloy reported "down"
# continuously and there was nothing for a deploy to wait on. The command
# below therefore moves the UI to the container's wildcard address and the
# port is published on host loopback only, which puts alloy in the same class
# as the other three: reachable from the host, by Caddy over the private
# network, and by nothing off-host. Alloy has no authentication, which is
# exactly why it is not published wider.
#
# SOURCES. The host journald socket and the host log/container-storage trees
# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
# lives at /opt/alloy/config/config.alloy, installed from
# shared/alloy/config.alloy on every deploy.
#
# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
# client fails and its log pipeline backs up.
# IMAGE. Built rootless by build_compose_image alloy, then mirrored into the
# root image store by the deployer (a root build runs in the build container's
# own resolver and fails on DNS here). Pull=never: boot must not reach for the
# local tag at a registry.
[Unit]
Description=Alloy log shipper (Dankclaw)
After=network-online.target loki.service
Wants=network-online.target loki.service
After=network-online.target
[Container]
ContainerName=alloy
Image=localhost/franksdank/alloy:latest
Pull=never
Network=dankclaw_default
Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
PublishPort=127.0.0.1:12345:12345
Volume=/opt/alloy/config:/etc/alloy
Network=host
Exec=run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy
Volume=/opt/alloy/config:/etc/alloy:ro
Volume=/etc/machine-id:/etc/machine-id:ro
Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
Volume=/var/log:/var/log:ro
Volume=/var/lib/containers:/var/lib/containers:ro
HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
Volume=/var/log/journal:/var/log/journal:ro
Volume=/run/log/journal:/run/log/journal:ro
HealthCmd=/usr/local/bin/busybox wget --spider -q http://127.0.0.1:12345/-/healthy
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
@@ -60,4 +56,4 @@ Restart=always
RestartSec=10
[Install]
WantedBy=default.target
WantedBy=multi-user.target
@@ -0,0 +1,56 @@
# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
#
# Install path on dankclaw:
# ~/.config/containers/systemd/blackbox-exporter.container
#
# WHY Network=host. The probes it runs are (a) the host's own LAN IP
# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
# bridge container cannot reach; netavark DNATs published ports only, see
# Container Host Reachability in the vault) and (b) ComfyUI's published port
# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
# reach neither class reliably.
#
# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
# bridge container dialing the podman gateway 169.254.1.2 lands on the host
# LOOPBACK — that is the scrape path, and it needs this exporter to be
# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
# address itself is NOT bindable from the host netns — it exists only
# inside each rootless netns; binding it dies "bind: cannot assign
# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
# No public DNS name, no guard chain: internal net only, per the exposure
# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
# flag per bind; plain --listen-address does not exist and the exporter
# exits in 100 ms with "unknown long flag" (live 2026-10-21).
[Unit]
Description=Blackbox exporter (HTTP probe prober)
After=network-online.target
[Container]
Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
ContainerName=blackbox-exporter
Network=host
# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
# exits "unexpected /bin/blackbox_exporter".
Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
# is passed as one quoted token and quadlet then folds the following
# --health-interval/--health-retries (and the image name) into the command
# string — unit died twice that way, live 2026-10-21. wget only; this
# busybox variant has no curl, and host-network wget answers on loopback.
HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
HealthInterval=30s
HealthRetries=3
[Service]
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
@@ -0,0 +1,56 @@
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
#
# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
# Unit name: cadvisor.service (container name: cadvisor)
#
# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
# now nothing reported per-container CPU/memory — Grafana could only see the
# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
# tree and exports container_cpu_* / container_memory_* per container.
#
# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
# subtree: cadvisor exported a single root series until this flag was added,
# then it walked the whole tree (996 per-container series measured 2026-10-20).
# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
# libpod/podman socket is needed to name containers.
#
# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
# The bridge PublishPort is what keeps the unauthenticated endpoint off the
# host's other interfaces: it publishes exactly loopback (host-local probes and
# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
# container, reaches a host-published port at its LAN address — the same netavark
# rule node-exporter relies on, only here the port really is published).
#
# It is rootless on purpose: dank owns the app containers' cgroups, so a
# rootless cadvisor reads every one of them; root would buy nothing here.
[Unit]
Description=cAdvisor container metrics (Dankclaw)
After=network-online.target
Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.143:8080:8080
Volume=/sys:/sys:ro
Volume=/:/rootfs:ro
Exec=--housekeeping_interval=30s
HealthCmd=wget --spider -q http://localhost:8080/healthz
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=40s
[Service]
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
[Container]
ContainerName=grafana
Image=docker.io/grafana/grafana:latest
Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
Network=dankclaw_default
User=0:0
PublishPort=127.0.0.1:3002:3000
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
# client_id query parameter — /application/o/<slug>/authorize/ does not exist.
Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
# ONLY when the token carries no `email` claim (authentik has no /emails path and
# 404s it, failing the login) — so the claim must actually be there: the fix is
# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
# <...>/userinfo/?api=1/emails, a 403.
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
# not a known scope here; requesting it makes the authorize view crash.
# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
# truncates SCOPES to its first token.
Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
# reaches this callback — the Anubis -> Authentik guard fronts every request.
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
@@ -33,7 +33,7 @@ Wants=network-online.target
[Container]
ContainerName=node-exporter
Image=docker.io/prom/node-exporter:latest
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
Network=host
PodmanArgs=--pid=host
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
@@ -43,8 +43,16 @@ Wants=network-online.target
[Container]
ContainerName=prometheus
Image=docker.io/prom/prometheus:latest
Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
Network=dankclaw_default
# Same-host scrape targets are dialed at host.containers.internal (the podman
# gateway): a bridge container cannot dial its own host's LAN IP (netavark
# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
# it unsupported and the WHOLE user quadlet batch then fails to generate,
# silently orphaning every other unit (live 2026-10-21: prometheus.service
# vanished while its container kept running as an orphan).
User=0:0
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
PublishPort=127.0.0.1:9090:9090
@@ -48,13 +48,22 @@
# without `podman unshare` — accepted for a datastore nobody edits by hand;
# episode AUDIO is a host directory for the opposite reason (see the app unit).
#
# HEALTH. The base image is distroless: no shell, no curl, no wget, so no
# CMD-SHELL probe and no wget/curl probe is possible here (same class of unit
# as loki, whose probe had to move into the image). SurrealDB ships its own
# probe instead — `surreal is-ready` connects to ws://localhost:8000 by default
# and needs no credentials — verified against this digest: it prints `OK` and
# exits 0 while the server is up. Argument-only HealthCmd, like every unit on
# this host, because the quadlet generator splits it on spaces.
# HEALTH. The base image is distroless — measured against this digest, not even
# `ls` exists (`exec: "ls": executable file not found in $PATH`), so there is no
# `/bin/sh`, no curl and no wget. SurrealDB ships its own probe instead:
# `surreal is-ready` connects to ws://localhost:8000 by default, needs no
# credentials, prints `OK` and exits 0 while the server is up.
#
# It must be handed over as a QUADLET-SIDE JSON ARRAY, not a bare command.
# podman wraps a bare `HealthCmd` string as `CMD-SHELL` (measured on this host's
# podman 5.7: `--health-cmd "/surreal is-ready"` stores
# `{"Test":["CMD-SHELL","/surreal is-ready"]}`), and with no `/bin/sh` in the
# image that probe fails in ~30 ms with empty output on every interval while the
# database answers perfectly — which is exactly how this service's first deploy
# died on 2026-10-19 (`did not become healthy in 90s`). The quoted array survives
# the generator (`--health-cmd "[\"/surreal\",\"is-ready\"]"`) and stores
# `["CMD","/surreal","is-ready"]`, which execs directly and goes healthy. Do not
# "simplify" it back to a bare command.
[Unit]
Description=SurrealDB (open-notebook datastore)
@@ -70,7 +79,7 @@ EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Exec=start --log info rocksdb:/mydata/mydatabase.db
Volume=open-notebook-db:/mydata
HealthCmd=/surreal is-ready
HealthCmd=["/surreal","is-ready"]
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
@@ -33,17 +33,21 @@
# and 5055 is exactly the port seerr already holds on this host
# (`PublishPort=127.0.0.1:5055:5055`; `ss -ltn` on Dankclaw 2026-10-18 shows
# 127.0.0.1:5055 taken). So the API leaves the container on host port 5056.
# That forces one setting: the frontend auto-detects the API as
# `<host-you-opened>:5055`, so API_URL in scripts/open-notebook.env must name
# the remapped port explicitly, or the UI loads and every call fails. Both
# ports are published on all interfaces because the consumer is a browser on
# the LAN — the fleet's browser-facing services (open-webui, comfyui, gitea)
# bind the same way, while loopback-only is reserved for services whose user
# path is the guard chain. This service has NO vhost and no Caddy route: it is
# internal-only until it is guarded, and its own password
# (OPEN_NOTEBOOK_PASSWORD) is what stands in front of it. That password is not
# decoration: with it unset the API answers everyone on the LAN, including the
# unauthenticated episode-audio download.
# Only 8502 is published on all interfaces, because it is the single upstream
# the guard chain dials: notebook.franksdank.cloud → anubis → authentik →
# http://:8017 → 192.168.0.143:8502 (dankclaw/Caddyfile), and this Caddy lives
# in root's storage, so a published port is the only way it can reach a rootless
# quadlet. The API needs no outward face at all — the image's own Next.js layer
# rewrites /api/* to the FastAPI beside it, so one origin serves UI and API —
# and the fleet's rule is that loopback-only is for services whose user path is
# the guard chain, which this became on 2026-10-19; 5056 stays published on
# loopback because the deployer's health gate dials it there. That is also why
# API_URL in scripts/open-notebook.env names the vhost rather than a port: left
# to itself the frontend auto-detects `<host-you-opened>:5055`, and the UI loads
# while every call fails. OPEN_NOTEBOOK_PASSWORD is not made redundant by the
# chain: it is what keeps the API honest for everything else on this host, and
# with it unset the API answers anyone, including the unauthenticated
# episode-audio download.
#
# VOLUMES. `/app/data` is a named volume: uploads, the LangGraph SQLite
# checkpoints and the extraction caches the image keeps under it
@@ -81,7 +85,7 @@ Pull=never
Network=dankclaw_default
Network=open-notebook.network
PublishPort=0.0.0.0:8502:8502
PublishPort=0.0.0.0:5056:5055
PublishPort=127.0.0.1:5056:5055
EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Volume=open-notebook-data:/app/data
@@ -27,7 +27,7 @@ After=network.target
[Container]
ContainerName=open-webui
Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main
Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
Environment=ANONYMIZED_TELEMETRY=False
+1
View File
@@ -45,3 +45,4 @@
192.168.0.143 pihole.franksdank.cloud
192.168.0.143 seerr.franksdank.cloud
192.168.0.143 marreta.franksdank.cloud
192.168.0.143 notebook.franksdank.cloud
+1 -1
View File
@@ -24,7 +24,7 @@ Wants=network-online.target
[Container]
ContainerName=pihole
Image=docker.io/pihole/pihole:latest
Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
Network=slirp4netns
PublishPort=0.0.0.0:53:53/udp
PublishPort=0.0.0.0:53:53/tcp
+1 -1
View File
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
[Container]
ContainerName=gluetun
Network=searxng.network
Image=docker.io/qmcgaw/gluetun:latest
Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
AddCapability=NET_ADMIN
AddDevice=/dev/net/tun:/dev/net/tun
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
ContainerName=searxng-valkey
Network=searxng.network
NetworkAlias=valkey
Image=docker.io/valkey/valkey:9-alpine
Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
Volume=/opt/searxng/valkey:/data
[Service]
+1 -1
View File
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
[Container]
ContainerName=searxng
Network=gluetun.container
Image=ghcr.io/searxng/searxng:latest
Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
Volume=/opt/searxng/config:/etc/searxng
+1 -1
View File
@@ -29,7 +29,7 @@ Wants=network-online.target
[Container]
ContainerName=seerr
Image=ghcr.io/seerr-team/seerr:v3.4.1
Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
Network=dankclaw_default
PublishPort=127.0.0.1:5055:5055
Volume=/home/dank/Containers/seerr/data:/app/config
+26
View File
@@ -0,0 +1,26 @@
# /etc/udev/rules.d/98-halogen-npu-access.rules
#
# Why the NPU node is world-writable: `frank-dragoon` runs rootless podman under a
# unit with `User=dank`, and the container's user namespace maps exactly one uid and
# one gid (dank). `/dev/accel/accel0` is `root:render 0660`, so inside the container
# the node reads `nobody:nogroup` and the engine dies with its own diagnosis:
#
# halogen npu: /dev/accel/accel0 is here, but this process cannot open it
# (crw-rw---- nobody:nogroup).
#
# `--group-add keep-groups` is the vendor's advice and it does NOT work in this
# shape: measured 2026-10-19 from the host as a `dank` shell that was already in
# `render`, with and without the flag, the node still read `nobody:nogroup` and the
# open was refused — an unmapped gid cannot be granted to a process inside a
# namespace that does not map it. Same conclusion `99-comfyui-gpu-access.rules`
# reached for `/dev/kfd` from the systemd-user-spawn direction: the mode is the only
# spawn-context-independent door short of running the stack rootful.
#
# Blast radius: any process in this host's single uid domain (every rootless
# container here shares it) can submit to the XDNA2 NPU. The NPU holds no user data
# and no privileged context — the same trade already accepted for the GPU compute
# nodes beside it. Installed by `deploy-remote.sh` when the NPU lane is armed and left
# in place afterwards — the mode costs nothing while the lane is idle. `serve-frank-dragoon.sh`
# refuses to arm the lane on a node that is not other-accessible, so a host that lost this
# file fails loudly at boot instead of crash-looping.
KERNEL=="accel[0-9]*", SUBSYSTEM=="accel", MODE="0666"
+44 -6
View File
@@ -19,11 +19,14 @@
# is recorded in the parent tree, so cloning the parent commit recursively is
# reproducible without a second pin.
#
# THE BUILD LINE IS THE ONE PROVEN ON DANKCLAW, verbatim:
# THE BUILD LINE IS THE ONE PROVEN ON DANKCLAW, verbatim — with the Vulkan door
# open since 2026-10-19. That configure was run in this exact base image and
# answers `Found Vulkan: /usr/lib/x86_64-linux-gnu/libvulkan.so (found version
# "1.3.275") found components: glslc` then `Including Vulkan backend`:
#
# cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CCACHE=OFF \
# -DGGML_BACKEND_DL=OFF -DGGML_CPU_ALL_VARIANTS=OFF -DGGML_OPENMP=ON \
# -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=OFF \
# -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=ON \
# -DGGML_METAL=OFF -DGGML_SYCL=OFF -DGGML_RPC=OFF \
# && cmake --build build -j 8
#
@@ -42,8 +45,12 @@
# defaults it ON outside wasm/cross-compiles, i.e. -march=native for THIS
# host — correct while build host == run host, wrong if anyone ever builds
# this image elsewhere, which is why the fleet builds it on Dankclaw.
# CUDA/HIP/VULKAN/METAL/SYCL/RPC all OFF: this is a CPU endpoint (the trial
# ran on CPU; the box's GPUs belong to the text endpoints).
# CUDA/HIP/METAL/SYCL/RPC stay OFF, VULKAN is ON: the endpoint computes on the
# gfx1151 iGPU (decision and co-tenancy gates:
# report/tts-npu-spike-verdict-2026-10-19.md, shape in [[Frank TTS Engine]]).
# Nothing else is available to this image: no ROCm userland, no NVIDIA
# userland, no macOS, no oneAPI, and RPC would add a second process to
# supervise on a box that already has one engine per card.
#
# BUILD SHARED LIBS, RUNTIME SMALL. BUILD_SHARED_LIBS defaults ON on Linux, so
# tts-server links libggml*.so rather than swallowing them; the runtime stage
@@ -67,9 +74,28 @@ ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:68ee22004a9109b44973313b02f
# ------------------------------------------------------------------ build
FROM ${BASE_IMAGE} AS build
# The three Vulkan build deps are not optional and not interchangeable:
# libvulkan-dev vulkan/vulkan.h + the VulkanHeaders/VulkanLoader CMake
# configs, i.e. what find_package(Vulkan) resolves to.
# glslc ggml compiles its shaders with glslc, not glslangValidator.
# Ubuntu's glslang-tools ships glslangValidator and NOT glslc,
# so without this package `find_package(Vulkan COMPONENTS
# glslc REQUIRED)` fails and there is no Vulkan backend.
# The archive's glslc is old (2023.8): it compiles
# GL_KHR_cooperative_matrix — the path ggml's quantized
# matmul needs — but reports cooperative_matrix2,
# integer_dot_product, bfloat16 and the micro-float
# extensions as unsupported, so those shader families are
# compiled out. If a Vulkan prefill ever reads as
# inexplicably slow, suspect this compiler before the device;
# the fix is a pinned glslang build passed as
# -DVulkan_GLSLC_EXECUTABLE=<path>, not a different runtime.
# spirv-headers find_package(SPIRV-Headers CONFIG REQUIRED); the distro
# package ships /usr/share/cmake/SPIRV-Headers/.
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
ca-certificates cmake g++ git make \
libvulkan-dev glslc spirv-headers \
&& rm -rf /var/lib/apt/lists/*
# Checked out detached at a full SHA, then the recorded submodule. No branch
@@ -90,7 +116,7 @@ RUN cmake -B build -DCMAKE_BUILD_TYPE=Release \
-DGGML_BLAS=OFF \
-DGGML_CUDA=OFF \
-DGGML_HIP=OFF \
-DGGML_VULKAN=OFF \
-DGGML_VULKAN=ON \
-DGGML_METAL=OFF \
-DGGML_SYCL=OFF \
-DGGML_RPC=OFF \
@@ -102,9 +128,21 @@ FROM ${BASE_IMAGE}
# libgomp1: the OpenMP runtime the build above links against. curl: the unit's
# HealthCmd. ca-certificates: so that curl can speak https if anyone ever
# points a probe at a URL with a scheme this host terminates elsewhere.
#
# libvulkan1 + mesa-vulkan-drivers are what make `--device Vulkan0` mean
# anything at runtime: libvulkan1 is the loader libggml-vulkan.so dlopens, and
# mesa-vulkan-drivers is the RADV ICD that claims the gfx1151. Loader without an
# ICD enumerates ZERO devices, and tts-server then dies in device lookup AFTER
# both GGUFs have been read — the journal reads as a model fault, not a missing
# driver. Enumerated from this base on 2026-10-19 with `--device /dev/dri
# --group-add keep-groups` plus the host's ComfyUI udev rule: "Radeon 8060S
# Graphics (RADV GFX1151)", API 1.4.318 (the heap figure is owned by
# [[Frank TTS Engine]]; it is the GTT-backed aperture, not the ~2 GiB DRM
# carve-out). No NVIDIA ICD is installed on purpose: one Vulkan device in the
# enumeration is what makes the NAME Vulkan0 safe to write down.
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
ca-certificates curl libgomp1 \
ca-certificates curl libgomp1 libvulkan1 mesa-vulkan-drivers \
&& rm -rf /var/lib/apt/lists/*
# Same unprivileged uid as the frank-tts façade (10000): the voice packs and the
@@ -40,12 +40,28 @@
# speaks to the server from inside its own namespace, which is where the address
# is valid.
#
# MEMORY / NO GPU. Peak RSS measured 5.7 GiB with both GGUFs resident
# (2026-10-18 trial); no memory limit is set because the box has the room and a
# limit would convert a busy moment into an OOM kill mid-utterance. No
# /dev/kfd, no /dev/dri, no keep-groups: every GPU backend is compiled out of
# the image, the endpoint is CPU-only by decision, and the GPUs on this host
# belong to the text endpoints.
# MEMORY / THE RENDER NODE. Host RSS is ~1.7 GiB, not the size of the GGUFs:
# the Vulkan backend copies both tensors into device memory at load, so they
# stop counting in the process's RSS (measured after the 2026-10-19 cutover —
# `podman stats` / `ps -o rss`). No memory limit is set: the box has the room
# and a limit would convert a busy moment into an OOM kill mid-utterance.
#
# `AddDevice=/dev/dri` is what makes the iGPU visible; without it the Vulkan
# ICD enumerates nothing and the server dies on device lookup after it has
# already read both GGUFs. `GroupAdd=keep-groups` is belt-and-braces, not the
# thing that works here: the units podman spawns carry no supplementary groups
# at all, and this host grants render access a different way — the udev rule
# that makes the render nodes world-writable (same rule ComfyUI rides, see
# dankclaw/comfyui). Measured: `id` inside this container lists only the
# container's own group, and `tts-server` still holds renderD129.
# The node is NOT named: this host is hybrid, renderD128 is the NVIDIA card and
# renderD129 the amdgpu iGPU, so a node number in the unit is a guess that a
# re-probe can invalidate, while the directory hands the container every render
# node and the driver picks by name.
#
# /dev/kfd stays absent on purpose: that is the ROCm compute interface, owned by
# the engines that use HIP. A Vulkan endpoint has no use for it, and handing a
# container a second way onto the GPU is surface for nothing.
#
# THE VOICE MOUNT IS READ-ONLY AND IS FOR THE WARMER, NOT THE ENGINE. A voice
# becomes known to the engine only through POST /v1/audio/voices, and the
@@ -98,6 +114,9 @@ Network=frank-tts-engine.network
EnvironmentFile=/home/dank/serverAppConfigs/scripts/frank-tts-engine.env
Volume=/home/dank/models/Qwen3-TTS-GGUF:/models:ro
Volume=/home/dank/voices:/voices:ro
# Render access for the iGPU (see MEMORY / THE RENDER NODE above).
AddDevice=/dev/dri
GroupAdd=keep-groups
HealthCmd=curl -fsS http://127.0.0.1:8880/health
HealthInterval=30s
HealthTimeout=10s
+54 -19
View File
@@ -7,7 +7,7 @@
# supervision, no retries, no logging of its own. PID 1 stays the server, so
# `podman stop` and the unit's Restart=always drive it directly.
#
# THE FLAGS THAT ARE NOT HERE, and must stay absent:
# THE FLAG THAT MUST STAY ABSENT:
#
# --threads / -t does not exist in this build. src/backend.h computes the
# CPU thread count as std::thread::hardware_concurrency()/2
@@ -18,13 +18,19 @@
# placement — pinning is deliberately not done anywhere in
# this stack. See the env file header for the same warning
# aimed at the operator.
# --device left at its default. Every GPU backend is compiled out of
# the image, so the default device pick can only land on
# CPU; naming a device would add a way to be wrong.
#
# Everything defaults to the shape the 2026-10-18 trial measured, so an empty
# environment still boots the decided configuration: 1.7 B talker Q4_K_M with
# the Q8_0 tokenizer, one request at a time, on the private bridge.
# --device IS here, but conditionally: it reaches the binary only when
# QWENTTS_DEVICE is non-empty, and empty is the CPU rollback — the binary then
# takes its own default device, which is CPU because the CPU backend is always
# built. One line in scripts/frank-tts-engine.env moves the endpoint between the
# two shapes, with no rebuild. What must never reach argv is a device NAME the
# loader does not enumerate: that is not refused by the argument parser, it dies
# during device lookup with the loader's own error, so the journal reads as a
# load that failed rather than a configuration refusal.
#
# Everything defaults to the shape this endpoint runs: 1.7 B talker Q4_K_M with
# the Q8_0 tokenizer, one request at a time, on the private bridge, on the
# gfx1151 iGPU. An empty environment still boots that shape.
set -euo pipefail
# The binary path. Overridable for one reason: the bats seam runs a stub
@@ -60,13 +66,33 @@ set -euo pipefail
: "${QWENTTS_LANG:=auto}"
# One request at a time. The server-side queue is a serialised FIFO at this
# setting, which is what keeps the measured per-voice timings true; batching
# on a CPU build splits the same cores between two syntheses and the warm RTF
# margin (0.55-0.67 against real time 1.0) disappears. The engine's own
# default is also 1; it is written here because it is a decision, not an
# accident.
# setting, which is what keeps the per-voice timings predictable. A second
# concurrent synthesis shares the device with the first — and on this box that
# device is also carrying frank-dragoon — so it buys no throughput and it
# doubles the wait for the caller behind it. The engine's own default is also 1;
# written here because it is a decision, not an accident.
: "${QWENTTS_MAX_BATCH:=1}"
# The compute device, NAMED and not numbered: `Vulkan0` is how ggml enumerates
# the gfx1151 iGPU on this host. An index would follow whatever order the
# loader's device list returns; a name says which device this endpoint means.
# The container sees exactly one Vulkan device because the image carries the
# Mesa ICD and no NVIDIA one, so the 3090 is not in the enumeration at all.
#
# EMPTY IS THE CPU ROLLBACK: a value that is set but EMPTY passes no `--device`,
# the binary picks its own default, and the CPU backend is always built — which
# is why moving this endpoint between the iGPU and the CPU is one line in
# scripts/frank-tts-engine.env and not a rebuild.
#
# That is also why the default below is NOT `${QWENTTS_DEVICE:=Vulkan0}`: `:=`
# fires on a set-but-empty variable as well as an unset one, so the rollback line
# `QWENTTS_DEVICE=` would be rewritten back to Vulkan0 and the endpoint would sit
# on the shared GPU with no way out short of a rebuild. Only UNSET takes the
# default.
if [[ ! -v QWENTTS_DEVICE ]]; then
QWENTTS_DEVICE=Vulkan0
fi
# Fail loudly BEFORE binding, so the journal says which artifact is missing
# instead of the unit reporting a container that starts and dies. This is the
# self-healing path: Restart=always re-runs it every RestartSec, and the same
@@ -79,11 +105,20 @@ for artifact in "$QWENTTS_MODEL" "$QWENTTS_CODEC"; do
fi
done
exec "$QWENTTS_BIN" \
--model "$QWENTTS_MODEL" \
--codec "$QWENTTS_CODEC" \
--alias "$QWENTTS_ALIAS" \
--host "$QWENTTS_HOST" \
--port "$QWENTTS_PORT" \
--lang "$QWENTTS_LANG" \
ARGS=(
--model "$QWENTTS_MODEL"
--codec "$QWENTTS_CODEC"
--alias "$QWENTTS_ALIAS"
--host "$QWENTTS_HOST"
--port "$QWENTTS_PORT"
--lang "$QWENTTS_LANG"
--max-batch "$QWENTTS_MAX_BATCH"
)
# Only when a device is named. An empty QWENTTS_DEVICE has to produce NO flag,
# because "the rollback shape" is argv identical to the pre-iGPU endpoint.
if [[ -n "$QWENTTS_DEVICE" ]]; then
ARGS+=(--device "$QWENTTS_DEVICE")
fi
exec "$QWENTTS_BIN" "${ARGS[@]}"
@@ -43,7 +43,7 @@ After=network.target
[Container]
ContainerName=calibre-server
Image=lscr.io/linuxserver/calibre:9.15.0
Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
Network=pasta
EnvironmentFile=/home/dank/Containers/calibre-server/env
Environment=PUID=1000
@@ -16,7 +16,7 @@ After=network.target
[Container]
ContainerName=calibre-web
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
Network=pasta
Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
Environment=LANG=en_US.UTF-8
+6 -28
View File
@@ -94,34 +94,12 @@ services:
timeout: 10s
retries: 3
# ═══════════════════════════════════════════════
# Jellyfin — media server (movies, TV, music)
# ═══════════════════════════════════════════════
jellyfin:
image: jellyfin/jellyfin:latest
container_name: jellyfin
restart: unless-stopped
ports:
- "8096:8096/tcp"
- "8920:8920/tcp"
- "7359:7359/udp"
- "1900:1900/udp"
environment:
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
volumes:
- /opt/jellyfin/config:/config
- /opt/jellyfin/cache:/cache
- /mnt/entropy:/data:ro
devices:
- /dev/dri:/dev/dri
group_add:
- "992"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
interval: 30s
timeout: 10s
retries: 3
# Jellyfin left this file when it became a Quadlet: the container is owned by
# dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
# /home/dank/Containers/jellyfin, and its library arrives through the SSD
# overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
# was here named /opt/jellyfin/* and a read-only array bind — a second owner for
# the container name `jellyfin`, wrong on every path. One owner per container.
# ═══════════════════════════════════════════════
# Calibre-Web — e-book web frontend
@@ -21,7 +21,13 @@ After=network.target
[Container]
ContainerName=copyparty
Image=ghcr.io/9001/copyparty-ac:latest
# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
# the pin rides :latest@digest (the tag in the reference is cosmetic; the
# digest is the pull). Pinned 2026-10-21 at the build whose label is
# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
# (1.20.26); the previously running build was the rolling :latest at
# 1.20.24 (issue #73 on command-center).
Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
Network=dankserver_appnet
EnvironmentFile=/home/dank/Containers/copyparty/env
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
+1
View File
@@ -0,0 +1 @@
entropy UUID=cd86c728-459e-4f3a-bcc2-41ecf84685ec /etc/entropy nofail,luks
+18
View File
@@ -0,0 +1,18 @@
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/nvme0n1p2 during curtin installation
/dev/disk/by-uuid/4f3731d2-fb7b-48e7-a79f-0e33cb8a4162 / ext4 defaults 0 1
# /boot/efi was on /dev/nvme0n1p1 during curtin installation
/dev/disk/by-uuid/183C-442D /boot/efi vfat defaults 0 1
#/dev/disk/by-label/archive-one /mnt/archive-one auto nosuid,nodev,nofail,x-gvfs-show 0 0
#/dev/disk/by-label/archive-two /mnt/archive-two auto nosuid,nodev,nofail,x-gvfs-show 0 0
# /dev/md0 is a LUKS container, not a filesystem; its ext4 is mounted via
# /dev/mapper/entropy on the next line. Mounting /dev/md0 directly fails.
#/dev/md0 /mnt/md0 ext4 auto,nofail,nosuid,nodev 0 0
/dev/mapper/entropy /mnt/entropy ext4 auto,nosuid,nofail,nodev,x-gvfs-show 0 0
/swapfile none swap sw 0 0
@@ -0,0 +1,30 @@
# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
#
# Installed by `scripts/media-view.sh install` — the repo copy is the source of
# truth (fleet law: never edit host state by hand).
#
# An OverlayFS merge that shows the media library as one tree while splitting
# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
# written), upperdir is a directory on the NVMe that receives every write. The
# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
# sidecars and artwork are SSD bytes — which is what makes them fast to browse
# and what keeps Plex's NFO agent off the spinners — while the media itself stays
# on the array untouched.
#
# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
# empty directory and the view would present an empty library as truth.
[Unit]
Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
RequiresMountsFor=/mnt/entropy
[Mount]
What=overlay
Where=/var/lib/media-meta/view
Type=overlay
Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
[Install]
WantedBy=local-fs.target
+10 -4
View File
@@ -4,8 +4,9 @@
# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
#
# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
# anywhere writes to the entropy volume. Every
# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
# while the sidecars this server writes land on the NVMe. Every
# creation-time environment variable of the compose-era container is kept as
# an Environment= line (zero drift); none of them are secrets, so this unit
# needs no EnvironmentFile.
@@ -16,7 +17,7 @@ After=network.target
[Container]
ContainerName=jellyfin
Image=ghcr.io/jellyfin/jellyfin:latest
Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
Network=dankserver_appnet
Environment=DEBIAN_FRONTEND=noninteractive
Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
PublishPort=1900:1900/udp
Volume=/home/dank/Containers/jellyfin/config:/config
Volume=/home/dank/Containers/jellyfin/cache:/cache
Volume=/mnt/entropy:/data:ro
# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
# fail-closed guard — starting against a plain, unmounted view directory would let
# a scan conclude the library is empty.
Volume=/var/lib/media-meta/view:/data
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
AddDevice=/dev/dri
GroupAdd=keep-groups
HealthCmd=curl -f http://localhost:8096/health
@@ -0,0 +1,39 @@
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
#
# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
# Unit name: cadvisor.service (container name: cadvisor)
#
# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
# Same rootless shape, same --cgroupns=host requirement (see that file for the
# full story), only the LAN address differs: this host publishes on
# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
# network (cross-host, so the host.containers.internal trick does not apply).
# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
# tautulli) by their quadlet unit names via the cgroup path.
[Unit]
Description=cAdvisor container metrics (Dankserver)
After=network-online.target
Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.235:8080:8080
Volume=/sys:/sys:ro
Volume=/:/rootfs:ro
Exec=--housekeeping_interval=30s
HealthCmd=wget --spider -q http://localhost:8080/healthz
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=40s
[Service]
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
@@ -37,7 +37,7 @@ Wants=network-online.target
[Container]
ContainerName=node-exporter
Image=docker.io/prom/node-exporter:latest
Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
Network=host
PodmanArgs=--pid=host
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
+7 -2
View File
@@ -4,14 +4,19 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
ContainerName=plex
Network=dankserver_appnet:alias=plex
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
Volume=/var/lib/plexmediaserver:/config
Volume=/mnt/entropy:/mnt/entropy:ro
# The container path stays /mnt/entropy on purpose: every path Plex recorded in
# its own database is written against it, so swapping the HOST source of this bind
# to the overlay view re-points Plex at the sidecars without touching its DB.
# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
Volume=/var/lib/media-meta/view:/mnt/entropy:ro
UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
PublishPort=32400:32400
[Service]
@@ -14,7 +14,7 @@ After=network.target
[Container]
ContainerName=romm-mariadb
Image=docker.io/library/mariadb:11
Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
Network=dankserver_appnet:alias=mariadb
EnvironmentFile=/home/dank/Containers/romm/env
Environment=MARIADB_DATABASE=romm
+1 -1
View File
@@ -12,7 +12,7 @@ After=network.target
[Container]
ContainerName=romm-redis
Image=docker.io/library/redis:7-alpine
Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
Network=dankserver_appnet:alias=redis
Volume=/home/dank/Containers/romm/redis:/data
HealthCmd=redis-cli ping
+1 -1
View File
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
[Container]
ContainerName=romm
Image=ghcr.io/rommapp/romm:latest
Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
Network=dankserver_appnet
EnvironmentFile=/home/dank/Containers/romm/env
Environment=DB_HOST=mariadb
@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
ContainerName=tautulli
Network=dankserver_appnet:alias=tautulli
Environment=PUID=0
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
[Container]
ContainerName=gluetun
Image=docker.io/qmcgaw/gluetun:latest
Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
AddCapability=NET_ADMIN
AddDevice=/dev/net/tun:/dev/net/tun
Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -15,7 +15,7 @@ After=gluetun.service
[Container]
ContainerName=lidarr
Image=lscr.io/linuxserver/lidarr:latest
Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -24,7 +24,7 @@ After=gluetun.service
[Container]
ContainerName=prowlarr
Image=lscr.io/linuxserver/prowlarr:latest
Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -29,7 +29,7 @@ Network=gluetun.container
# Pinned to the last 4.x line: Readarr's retired qB client requires the
# qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
# empty body, which fails Readarr's auth check. All other clients accept 4.x.
Image=lscr.io/linuxserver/qbittorrent:4.6.7
Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -15,7 +15,7 @@ After=gluetun.service
[Container]
ContainerName=radarr
Image=lscr.io/linuxserver/radarr:latest
Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -19,7 +19,7 @@ After=gluetun.service
[Container]
ContainerName=readarr
Image=lscr.io/linuxserver/readarr:amd64-1.0.980
Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
@@ -16,7 +16,7 @@ After=gluetun.service
[Container]
ContainerName=sonarr
Image=lscr.io/linuxserver/sonarr:latest
Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
Environment=PUID=0
Environment=PGID=0
Environment=TZ=America/Los_Angeles
+28 -17
View File
@@ -136,25 +136,36 @@ TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that
ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
3090 slot.
2. **Contention gate.** `frank-dragoon` is the endpoint the operator session runs on, so the
honest measurement with both loaded is dragoon's own TTFT and decode with a synthesis in
flight — one request each, no ladder. Expect compute-unit sharing rather than memory
pressure: the aperture has room, the shaders do not.
2. **Contention gate — PASSED, measured 2026-10-19 after the cutover.** With a 16.9 s
synthesis occupying the device, `frank-dragoon` answered a streamed 124-token request at
**17 ms to first token and ~35 tok/s decode**, against 17 ms and ~33 tok/s for the same
request with the card idle. Compute-unit sharing showed up as nothing. One request per
arm, no ladder: these are the gate's evidence, not a throughput record, and the engine's
own shape lives in [[Frank TTS Engine]].
**Change surface** — the whole edit, so nobody re-derives it:
**Change surface — as landed** (PR #34, `6248f02`; the CPU shape stays reachable as the
rollback):
1. `Containerfile.engine` build stage: add `libvulkan-dev glslang-dev`. It installs only
`ca-certificates cmake g++ git make` today, and ggml's Vulkan backend compiles its
shaders at build time — the flag alone fails at configure.
2. `-DGGML_VULKAN=OFF` → `ON` in the same cmake block; the flag list is written out
explicitly, so this is a one-word diff.
3. Runtime stage: add `mesa-vulkan-drivers`. Without a Vulkan **ICD inside the image** the
device node buys nothing — there is no `radeon_icd.json` in the container today.
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — **not**
`renderD128`. On this hybrid box that node is the NVIDIA card; the iGPU is
`renderD129` (`readlink -f /sys/class/drm/renderD129/device/driver` → `amdgpu`).
5. Engine argv: `--device Vulkan0` — qwentts.cpp's tools take `--device <name>` at the
pinned commit.
1. `Containerfile.engine` build stage: add **`libvulkan-dev glslc spirv-headers`**. The
prediction above (`libvulkan-dev glslang-dev`) is wrong on two counts and the configure
says so: ggml looks for `glslc`, which Ubuntu's `glslang-tools` does **not** ship (it
ships `glslangValidator`), and `find_package(SPIRV-Headers CONFIG REQUIRED)` needs the
distro's `spirv-headers`, which does carry the CMake config. `glslc` is its own source
package on noble, at 2023.8 — old enough that only `GL_KHR_cooperative_matrix` compiles,
so the coopmat2 / integer-dot / bf16 / micro-float shader families are built out. Suspect
that compiler before the device if a Vulkan prefill ever reads as slow.
2. `-DGGML_VULKAN=OFF` → `ON`: one word in the written-out flag list.
3. Runtime stage: `mesa-vulkan-drivers` **and** `libvulkan1` — the ICD and its loader. An
image with a loader and no ICD enumerates zero devices.
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — the
DIRECTORY, never the node: on this hybrid box `renderD128` is the NVIDIA card and
`renderD129` the amdgpu iGPU (`readlink -f /sys/class/drm/renderD129/device/driver` →
`amdgpu`). On this host the udev rule already makes the nodes world-writable, so the
group is belt-and-braces, not the gate.
5. Engine argv: `--device Vulkan0`, reached through `QWENTTS_DEVICE` in
`scripts/frank-tts-engine.env`. **An empty value passes no flag at all** — that is the
CPU rollback, and it is why the entrypoint's default is an `-v` test rather than `:=`,
which also fires on a set-but-empty value and would silently undo the rollback.
Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
+32 -14
View File
@@ -1,27 +1,45 @@
#!/usr/bin/env bash
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
# derive-protected-experts.py over the calib dir, splice its cast lines into
# the marked section of the recipe in place, keep the pre-edit recipe in the
# campaign trash. Refuses to run twice (marker text says derived).
# campaign trash. Refuses to run twice (marker text says derived), and refuses
# when a requested target count was not reached. Run it on a COPY of the
# template so the committed recipe stays the clean, reviewable baseline.
set -euo pipefail
CALIB=${1:-/home/dank/corpora/calib}
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
PY=/home/dank/venvs/radcorpus/bin/python
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
TARGET_COUNT=${3:-0}
PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
TS=$(date +%Y%m%d-%H%M%S)
# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
# scratch directory; on the box they are the campaign's usual places.
OUT=${OUT:-/tmp/protected-cast-$TS.txt}
TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
ARGS=("$DERIVE" "$CALIB")
if [ "$TARGET_COUNT" -gt 0 ]; then
ARGS+=(--target-count "$TARGET_COUNT")
fi
"$PY" "${ARGS[@]}" > "$OUT"
N=$(grep -c 'cast dtype=bf16' "$OUT")
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
echo " (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
exit 1
fi
echo "derived $N protected experts -> $OUT"
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
mkdir -p "$TRASH_DIR"
cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
import sys
recipe, castfile = sys.argv[1], sys.argv[2]
lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
elif 'PROTECTED-END' in l:
insec = False
out.append(l)
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
elif not insec:
out.append(l)
open(recipe, 'w').write('\n'.join(out) + '\n')
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
#
# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
# Two DB-side facts (the provider was created by hand via `ak shell`, never
# repo-managed) silently broke that login, and BOTH had previously been
# mis-recorded here as a single "the property-mapping evaluator is broken, so
# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
#
# 1. provider.authorization_flow was NULL. The authorize view builds a flow
# planner from it and crashes with
# `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
# "Server Error — Powered by authentik", before any claim is touched.
# Fix: point it at default-provider-authorization-implicit-consent, the
# flow every other provider in this instance already uses.
# 2. The `grafana-email-scope` ScopeMapping's expression was the bare string
# `user.email`. authentik merges a scope mapping's result into the token as
# a MAPPING: a returned dict adds its keys as claims; a returned string is
# keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
# claim is ever emitted. Grafana, finding no email, falls back to
# GET <api_url>/emails, authentik 404s that path, and the login dies with
# "Login failed: InternalError". Fix: return
# {"email": user.email, "email_verified": True} — the shape the stock
# OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
# mapping has always worked, which is the proof the evaluator is fine.
#
# This is authentik DB state. It is NOT in any deployable file: a blueprint
# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
# facts and the symptom returns verbatim. Re-run this after any such rebuild.
# Every step is idempotent.
#
# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
# dankclaw/observability/quadlet/grafana.container and deploys normally.
#
# Usage: authentik-grafana-sso-fix.sh (override host: AUTHENTIK_HOST=…)
set -euo pipefail
HOST="${AUTHENTIK_HOST:-dankclaw}"
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
from authentik.flows.models import Flow
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.filter(name="grafana").first()
if not p:
raise SystemExit("no OAuth2Provider named 'grafana'")
# 1. The authorization flow every other provider uses.
want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
if not want:
raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
if p.authorization_flow_id != want.pk:
p.authorization_flow = want
p.save()
print(f"set grafana.authorization_flow -> {want.slug}")
else:
print(f"grafana.authorization_flow already {want.slug}")
# 2. The email scope mapping must RETURN A DICT, not a bare string.
m = p.property_mappings.filter(name="grafana-email-scope").first()
if not m:
raise SystemExit("grafana-email-scope mapping is missing")
fixed = 'return {"email": user.email, "email_verified": True}'
if m.expression.strip() != fixed:
m.expression = fixed
m.save()
print("repaired grafana-email-scope -> returns {email, email_verified}")
else:
print("grafana-email-scope expression already correct")
# 3. The id_token must carry the scope claims; Grafana reads identity from it.
if not p.include_claims_in_id_token:
p.include_claims_in_id_token = True
p.save()
print("enabled include_claims_in_id_token")
else:
print("include_claims_in_id_token already on")
PYEOF
echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
echo "the round trip must land on / and /api/user must answer 200 with an email."
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Set the authentik login-session TTL on Dankclaw.
#
# The login-session lifetime is governed by the `default-authentication-login`
# stage's `session_duration`, NOT a global setting and NOT SESSION_COOKIE_AGE
# (that is the pre-login session, sessions.unauthenticated_age). Authentik does
# not expose SESSION_SAVE_EVERY_REQUEST at all, so the session is an absolute
# window from login, not a sliding one.
#
# The field is a STRING in authentik's own idiom ("days=7", "seconds=30",
# "hours=12") — not a Python timedelta. Writing str(timedelta(days=7)) stores
# "7 days, 0:00:00", which authentik's timedelta_from_string parser REJECTS
# (ValueError), and login then crashes computing the session expiry. Always pass
# the "key=value" form; this script validates it before writing.
#
# Existing sessions keep their old expiry; the new TTL applies from the next
# login. This is DB state — it is NOT repo-managed and a blueprint re-import or
# rebuild-from-scratch reverts it. Run this again after such a rebuild.
#
# Usage: authentik-set-login-session-duration.sh ["days=7"] (default days=7)
set -euo pipefail
DUR="${1:-days=7}"
STAGE="default-authentication-login"
HOST="${AUTHENTIK_HOST:-dankclaw}"
# Refuse anything that is not the safe key=value duration idiom.
if ! printf '%s' "$DUR" | grep -Eq '^[A-Za-z]+=[0-9]+([ ]+[A-Za-z]+=[0-9]+)*$'; then
echo "invalid duration '$DUR' — use authentik form, e.g. 'days=7' or 'seconds=30'" >&2
exit 2
fi
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
from django.apps import apps
from authentik.lib.utils.time import timedelta_from_string
dur = "$DUR"
timedelta_from_string(dur) # validate the form before touching the row
ULS = apps.get_model("authentik_stages_user_login", "UserLoginStage")
s = ULS.objects.get(name="$STAGE")
s.session_duration = dur
s.save(update_fields=["session_duration"])
back = ULS.objects.get(name="$STAGE").session_duration
assert back == dur, "persisted value mismatch: %r" % back
print("session_duration set to", repr(back), "->", timedelta_from_string(back))
PYEOF
echo "Login-session TTL for '$STAGE' is now $DUR (applies on next login)."
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env bash
# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
# no rule matches keeps the checkpoint's encoding, so every tensor stays the
# bf16 it is in the safetensors and the container is the checkpoint's own
# numbers behind a .rad header.
#
# This file is the campaign's measuring stick, twice over: scripts/
# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
# arm's --kld-ref is scored against that record) and the same container serves
# with the calibration tap to write the Grams that derive the protected-expert
# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
# --reuse, and the four refusal gates below.
#
# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
# served from one; rad-convert also bakes the tokenizer. The convert runs on
# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
# kernels on CPU and production serving is untouched for the whole run (the
# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
#
# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
# written, cheapest first — the verifier runs last of them because its Hub gate
# hashes 343 GiB and must not be paid for by a run that was going to refuse on
# the output file anyway.
# checkpoint the tree must be on disk; an absent dir names the command that
# makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
# free space the convert writes ~343,323 MiB next to the ~343,323 MiB
# checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
# plus 6.5 GiB of slack) is the floor. The volume that cannot
# hold both is discovered at minute 2, not hour 5.
# output the campaign converts the reference ONCE. An existing
# qwen38-bf16.rad is refused; --force retires it to
# ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
# deleted, and the move is a rename on the same volume).
# verified scripts/download-qwen38-bf16.sh's verify mode must pass. A
# failed verify exits 1, not 2: the tree being the wrong payload
# is a result, a missing environment is a refusal.
# plan rad-convert --plan-only -v first — the oracle for "does this
# shape boot". A refused plan costs seconds and saves a convert.
#
# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
# --dry-run print the argv it would exec (podman wrapping rad-convert),
# one shell-quoted argument a line, and exit 0. No gate runs, no
# container starts, nothing is written — the offline test seam.
# It does resolve the checkpoint and output paths through the
# /models mount, because an argv that carries host paths would
# not run either.
# --plan-only run the plan and every gate except the disk and output ones —
# those two protect bytes the plan never writes, and this is the
# probe to run when the disk gate itself is what refuses
# --force retire an existing output container to the trash dir first
#
# knobs: FRANK_QWEN38_MODELS the host dir mounted at /models (default
# /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
# Qwen3.8-Flash-Next, the same knob the verifier reads);
# FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
# QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
# verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
# reach it through the environment untouched.
# exit: 0 converted (or planned), 1 the checkpoint failed verification,
# 2 refused before anything was written
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ONE owner of the engine pin: the digest frank-archon serves on. The reference
# must be built by the SAME engine build that scores it, or a KLD delta is
# partly a version delta.
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-archon.env"
IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
LOG="/home/dank/logs/qwen38_bf16_convert.log"
DRY=0
PLAN_ONLY=0
FORCE=0
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY=1 ;;
--plan-only) PLAN_ONLY=1 ;;
--force) FORCE=1 ;;
*) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
exit 2 ;;
esac
shift
done
# The convert sees the tree only through the /models mount, so both paths must
# be under it; the container-side spelling is what goes into the argv.
case "$CKPT" in
"$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
*) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
exit 2 ;;
esac
case "$OUT" in
"$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
*) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
exit 2 ;;
esac
# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
# bound this container will actually be served at. Both flags come off the pinned
# image's own help: --kv-cache-dtype at convert "only picks the kernels it
# checks" and the container is identical either way, so naming fp8 is free
# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
# convert refuses here instead of the boot failing on a drained window.
# --max-ctx is the bound declare is run at, whose default is the checkpoint's
# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
# machine, so the default is not an option). 32,768 is set by the corpus, not by
# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
# corpus — so the window the rig boots at clears every record 2.6x over and the
# reference boot cannot die on a sequence that will not fit. The rig's window is
# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
# 32,768) and that script refuses a corpus its window cannot hold, so raising one
# without the other is caught before the drained window, not inside it. The
# payload containers the gate scores declare at the training context because
# production must hold the 200k shape; a declared bound is a declare-time check,
# not a weight change, so the reference and the payloads carry the same numbers
# whatever each declared at — which is why one can be measured against the other.
CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
if [ "$DRY" = "1" ]; then
echo "dry run: no gate checked, no container started, nothing written"
echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
echo "# plan argv:"
printf '%q\n' "${PLAN_CMD[@]}"
echo "# convert argv:"
printf '%q\n' "${CONV_CMD[@]}"
exit 0
fi
# ---- gate: the checkpoint is on disk ---------------------------------------
if [ ! -d "$CKPT" ]; then
echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
exit 2
fi
# ---- gate: peak disk --------------------------------------------------------
# The deepest existing ancestor is the right thing to ask: a fresh
# /home/dank/models/rad has no free space of its own to report.
OUT_DIR="$(dirname "$OUT")"
PROBE="$OUT_DIR"
while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
exit 2
fi
if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
exit 2
fi
# ---- gate: one conversion ---------------------------------------------------
if [ -e "$OUT" ]; then
if [ "$PLAN_ONLY" = "1" ]; then
echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
elif [ "$FORCE" = "1" ]; then
mkdir -p "$TRASH"
DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
mv "$OUT" "$DEST"
echo "TRASHED: retired the existing reference container $OUT -> $DEST"
else
echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
exit 2
fi
fi
# ---- gate: the tree really is the checkpoint (the full verify) -------------
echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
rc=0
bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
if [ "$rc" -ne 0 ]; then
if [ "$rc" -eq 2 ]; then
echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
exit 2
fi
echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
exit 1
fi
# ---- plan, then convert ------------------------------------------------------
mkdir -p "$(dirname "$LOG")"
# rad-convert opens -o during declare as well, so the output directory exists
# before the plan; that empty directory is all this creates ahead of it.
mkdir -p "$OUT_DIR"
echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
if [ "$rc" -ne 0 ]; then
echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
exit 2
fi
if [ "$PLAN_ONLY" = "1" ]; then
echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
exit 0
fi
echo "== convert: $CKPT -> $OUT ($(date -Is))"
"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
if [ "$rc" -ne 0 ]; then
echo "rad-convert rc=$rc; log: $LOG" >&2
exit "$rc"
fi
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"
+149 -18
View File
@@ -22,6 +22,17 @@ Mechanics notes:
prompts do not bias the Hessians (the corpus is drawn round-robin over
source classes, so loss is class-neutral as long as failures are rare;
a systematic 4xx is loud in the log and must be fixed, not ignored).
- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
the run, and a run that never sees the engine's completion line exits non-zero.
The first version failed on a flat round, which aborted exactly one round too
early (2026-10-29).
- before any prompt, the tap must carry the rig's own marker
(`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
container, and that container must be running. Measured 2026-10-29:
with neither check, four rounds of 10M tokens went into restored
production after the rig's boot had died, and recorded nothing.
Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
[--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
import argparse
import concurrent.futures
import json
import os
import subprocess
import sys
import time
import urllib.request
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import qwen38_common as qc # noqa: E402 - sibling module, same directory
DONE_MARK = "calibration wrote"
@@ -49,9 +67,9 @@ def tap_done(container):
return DONE_MARK in (out.stdout + out.stderr)
def push_one(url, prompt):
def push_one(url, prompt, model):
body = json.dumps({
"model": "frank-archon",
"model": model,
"prompt": prompt,
"max_tokens": 1,
"temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
return False
def main():
def push_round(url, prompts, model, parallel):
"""Push the whole corpus once; returns how many requests were accepted."""
ok = 0
with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
for good in ex.map(lambda p: push_one(url, p, model), prompts):
ok += 1 if good else 0
return ok
def tap_bytes(path):
"""Bytes the tap has sealed so far; 0 when the directory is absent.
A tap whose directory is wrong, full or read-only stops writing while the
engine keeps answering 200 — the driver then re-pushes the whole corpus for
every round it is allowed and the loss is invisible until the hours are gone.
"""
total = 0
try:
for root, _, files in os.walk(path):
for f in files:
try:
total += os.path.getsize(os.path.join(root, f))
except OSError:
pass
except OSError:
return 0
return total
def round_verdict(before, after, done):
"""What one round's tap bytes mean.
'done' the engine logged that it sealed the tap — the run is over
'shrink' the tap LOST bytes: reset, pruned or replaced under us
'grew' sealed bytes this round
'flat' sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
pooled floor, and a round that does not reach it has nothing to
write yet (the campaign's corpus is 10.03M tokens against a 12M
floor, so round 1 is always flat). Failing on it aborted the run
exactly one round too early.
"""
if done:
return "done"
if after < before:
return "shrink"
if after == before:
return "flat"
return "grew"
def require_rig(a):
"""Refuse unless the rig that opened this tap is up and serving right now.
Three facts, in order: the runner wrote its marker into the tap dir, the
marker names the container this pusher targets, and that container is
running. A marker without a live container is a boot that died; a live
container without a marker is some other engine on the port — on 2026-10-29
it was restored production, which took four rounds of 10M tokens before the
old driver admitted it had recorded nothing.
"""
marker = os.path.join(a.tap_dir, a.tap_marker)
if not os.path.isfile(marker):
qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
"run-qwen38-bf16.sh calib, so a push here would feed whatever "
"answers the port and record nothing")
try:
lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
except OSError as exc:
qc.refuse(f"cannot read the rig marker {marker}: {exc}")
meta = dict(line.split("=", 1) for line in lines if "=" in line)
who = meta.get("container", "")
if who != a.container:
qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
f"{a.container!r} — wrong tap, or a stale marker")
try:
p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
capture_output=True, text=True, timeout=30)
except Exception as exc:
qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
if p.returncode != 0 or p.stdout.strip() != "true":
qc.refuse(f"the rig container {a.container} is not running — whatever is "
"answering the port, it is not the calibration rig")
def parse_args():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=7770)
ap.add_argument("--parallel", type=int, default=64)
ap.add_argument("--container", default="arbench")
ap.add_argument("--model", default="frank-archon")
ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
ap.add_argument("--tap-marker", default=".rig-boot",
help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
ap.add_argument("--max-rounds", type=int, default=4)
a = ap.parse_args()
url = f"http://127.0.0.1:{a.port}/v1/completions"
return ap.parse_args()
def load_prompts(a) -> list[str]:
if not os.path.isfile(a.corpus):
qc.refuse(f"no corpus at {a.corpus}")
prompts = []
with open(a.corpus) as f:
for line in f:
line = line.strip()
if line:
prompts.append(json.loads(line)["prompt"])
print(f"corpus: {len(prompts)} prompts", flush=True)
for lineno, rec in qc.iter_jsonl(a.corpus):
if not isinstance(rec.get("prompt"), str):
qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
prompts.append(rec["prompt"])
if not prompts:
qc.refuse(f"the corpus at {a.corpus} holds no prompts")
return prompts
def main():
a = parse_args()
url = f"http://127.0.0.1:{a.port}/v1/completions"
prompts = load_prompts(a)
require_rig(a)
chars = sum(len(p) for p in prompts)
print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
flush=True)
for rnd in range(1, a.max_rounds + 1):
if tap_done(a.container):
break
before = tap_bytes(a.tap_dir)
t0 = time.time()
ok = 0
with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
for good in ex.map(lambda p: push_one(url, p), prompts):
ok += 1 if good else 0
dt = time.time() - t0
ok = push_round(url, prompts, a.model, a.parallel)
dt = max(time.time() - t0, 1e-9)
print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
flush=True)
f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
if ok == 0:
print("FAIL: not one prompt was accepted; fix the engine or the argv "
"before spending another round", flush=True)
sys.exit(1)
if tap_done(a.container):
break
after = tap_bytes(a.tap_dir)
verdict = round_verdict(before, after, False)
if verdict == "shrink":
print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
f"({before} B before, {after} B after) — it was reset or replaced "
"under us", flush=True)
sys.exit(1)
if verdict == "flat":
print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
f"{before} B) — continuing", flush=True)
else:
print("max rounds exhausted, tap not done", flush=True)
sys.exit(1)
+1 -1
View File
@@ -21,7 +21,7 @@
# deployed knob has since dropped to 262144.
set -euo pipefail
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
CUR=/home/dank/models/halogen-flashnext-v2
NEW=/home/dank/models/halogen-v2-abliterated
CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
+236
View File
@@ -0,0 +1,236 @@
#!/usr/bin/env bash
# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
#
# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
# converter runs on the host backend and the fleet can keep serving through the whole script.
# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
# boundary between measurement and serving.
#
# Three arms, because the campaign's question is which search knob pays for its hours
# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
#
# cd3 the recipe as written — coordinate descent three sweeps. The control.
# cd6 six sweeps: the same search, twice the passes.
# ao1 act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
# stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
# cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
# act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
# the two knobs cannot be combined into a best-of-both container.
#
# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
# expert line and the other 60 lines cannot drift between arms.
#
# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
#
# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
# inputs checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
# directory converts UNCALIBRATED with one stderr note, which is silently worse
# than failing here.
# reference the 335.3 GiB BF16 reference container must be GONE before the converts start.
# It and the checkpoint together eat the volume; its job (the KLD recording and
# the calibration grams) is finished by then, and both survive on disk without the
# container beside it. The FP8-native anchor is FETCHED after the converts (disk
# sequencing), so nothing here waits on it.
# plan --plan-only runs FIRST, per arm, and prints "about X would be written". The
# number is the bill: free space must cover it plus the 128k-prompt overshoot
# margin, and the finished artifact is checked against it.
# outputs no arm overwrites anything; move the old one to trash.
#
# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
# [--target-count N] [--keep-recipe]
set -u -o pipefail
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
# the same image the tap producer boots: a tap must be converted by the line that
# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
# it. Production stays on its own pin — this is the campaign's toolchain.
IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
CALIB=${CALIB:-$CAMPAIGN/calib}
IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
OUTDIR=${OUTDIR:-/home/dank/models/rad}
BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad} # the reference container, not an arm output
STATE=${STATE:-$CAMPAIGN} # the dir the container mounts read-only: the
# spliced recipe and the imatrix live here
TARGET_COUNT=${TARGET_COUNT:-48}
MIN_GRAMS=${MIN_GRAMS:-24576} # per-expert DOWN grams: 48 recorded layers x 512
# experts. The checkpoint has 49 MoE layers, but the
# MTP layer is never recorded, so its 512 experts are
# absent from the tap's output by design
MARGIN_MIB=${MARGIN_MIB:-40960} # the long-prompt overshoot, off the plan number
TS=$(date +%Y%m%d-%H%M%S)
ARMS=()
DRY=0
PLAN_ONLY=0
KEEP_RECIPE=0
while [ "$#" -gt 0 ]; do
case "$1" in
--arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
ARMS+=("$2"); shift 2 ;;
--dry-run) DRY=1; shift ;;
--plan-only) PLAN_ONLY=1; shift ;;
--target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
TARGET_COUNT="$2"; shift 2 ;;
--keep-recipe) KEEP_RECIPE=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
for a in "${ARMS[@]}"; do
case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
done
# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
# it for empty).
expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
arm_quant() {
case "$1" in
# The control is the template itself: no override, so the recipe's own line decides.
cd3) return 0 ;;
cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
# act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
# columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
# this arm is the interleave knob alone, not cd=3 plus it.
ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
esac
}
# --- gates ------------------------------------------------------------------------------------
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
# converter fail halfway through a planned arm.
[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
if [ -e "$BF16_RAD" ]; then
echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
"the calibration grams are already taken, so move it to trash — the converts need its" \
"space (peak-disk law)" >&2
exit 2
fi
for a in "${ARMS[@]}"; do
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
# A killed run leaves its container behind (--rm only fires when podman itself
# exits), and the next arm then dies on "name already in use" AFTER the plan
# phase; refuse here, where the fix is one `podman rm -f`.
podman container exists "qwen38-convert-$a" && {
echo "container qwen38-convert-$a still exists (a killed run?);" \
"'podman rm -f qwen38-convert-$a' and re-run" >&2
exit 2; }
done
# --- the protected plane, spliced into a per-run copy of the template --------------------------
mkdir -p "$STATE"
WORK="$STATE/recipe-$TS.recipe"
cp "$RECIPE" "$WORK"
if [ "$DRY" = 1 ]; then
echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
else
# Positional, because the tool reads its recipe and calib dir by position: passing them as
# environment variables would be silently ignored and it would rewrite the COMMITTED
# template in place instead of this run's copy.
"$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
|| { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
fi
run_convert() { # run_convert ARM [extra flags...]
local arm="$1"; shift
local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
local argv=(--rm --name "qwen38-convert-$arm"
-e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
-v "$(dirname "$CKPT"):/models:ro"
-v "$CALIB:/data/calib:ro"
-v "$STATE:/campaign:ro"
-v "$OUTDIR:/out")
local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
--tokenizer "/models/${CKPT##*/}/tokenizer.json"
--imatrix /campaign/"${IMATRIX##*/}"
# No --max-ctx: the context declare then runs at the checkpoint's training context
# (262144), which is what the shipped container did — rad-info --meta on it carries
# no override. A container declared at a smaller bound cannot be expected to serve
# the 200k window D2 holds, and the convert is not where to discover that.
# --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
# checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
# to by the convert, not by a boot on a drained window.
--kv-cache-dtype fp8 -o "/out/$(basename "$out")")
local q; q=$(arm_quant "$arm")
[ -n "$q" ] && conv+=(--quant "$q")
if [ "$DRY" = 1 ]; then
echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
return 0
fi
podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
2>&1 | tee "$log"
return "${PIPESTATUS[0]}"
}
# --- plan, then bill, then convert -------------------------------------------------------------
overall=0
for arm in "${ARMS[@]}"; do
echo "== arm $arm =="
if [ "$DRY" = 1 ]; then
# Both lines, in order: the plan runs before the convert, and the convert is what the
# operator is about to pay an afternoon for.
run_convert "$arm" --plan-only
run_convert "$arm"
continue
fi
rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
[ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
echo "$plan_log"
# "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
# hardcoding a size, so the gate still works after any recipe edit.
want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
[ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
overall=1; continue
fi
if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
rc=0; run_convert "$arm" || rc=$?
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
[ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
# count. An arm that came out materially smaller lost a plane somewhere.
lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
overall=1; continue
fi
echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
done
if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
mkdir -p "$CAMPAIGN/.AGENTTRASH"
mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
fi
exit "$overall"
+48 -5
View File
@@ -34,12 +34,16 @@ SERVICE_ORDER = (
"hermes",
"hermes-dashboard",
"hermes-webui",
"prometheus",
"grafana",
"loki",
"alloy",
"node-exporter",
"cadvisor",
"gpu-exporter",
"alertmanager",
# The prober must answer before prometheus re-runs the services job.
"blackbox-exporter",
"prometheus",
"authentik-postgres",
"authentik-server",
"authentik-worker",
@@ -92,7 +96,10 @@ DANKCLAW_SERVICES = {
"loki",
"alloy",
"node-exporter",
"cadvisor",
"gpu-exporter",
"alertmanager",
"blackbox-exporter",
"authentik-postgres",
"authentik-server",
"authentik-worker",
@@ -124,6 +131,8 @@ DANKSERVER_SERVICES = {
"readarr",
"frank-archon",
"node-exporter",
"cadvisor",
"gpu-exporter",
}
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -146,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
# Quadlet unit now — the last compose services were the hermes trio, whose units
# have owned their containers since 2026-09-12 and whose compose blocks were
# dead weight that could only create a duplicate container.
# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
# has owned the container since the cutover, so a compose edit must not restart it.
DANKSERVER_COMPOSE_SERVICES = {
"mariadb",
"redis",
"romm",
"copyparty",
"jellyfin",
"calibre-web",
}
COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
@@ -271,6 +281,8 @@ def services_for_path(path: str, target: str) -> set[str]:
# Provisioned datasources and dashboards: git is the source of
# truth, so a change here must restart the service that reads them.
_add(services, "grafana")
elif group == "blackbox":
_add(services, "blackbox-exporter")
elif group == "homepage":
# Homepage is a Dankclaw-owned deployment. Shared homepage files
# are intentionally not copied during a Dankserver-only run.
@@ -293,6 +305,17 @@ def services_for_path(path: str, target: str) -> set[str]:
if len(parts) >= 2 and parts[1] in {"caddy", "comfyui", "loki", "alloy", "hermes", "anubis", "seerr"}:
_add(services, parts[1])
return services
if len(parts) >= 3 and parts[1] == "host":
# Host state a service needs at boot lives under dankclaw/host/.
# The fabric-clock hold is frank-dragoon's NPU precondition (the
# engine refuses to start with HALOGEN_NPU_MODELS set while the
# clock moves) and the deployer installs and gates it, so changing
# it re-dispatches that endpoint. `grub` is the tracked boot line,
# which moves only through scripts/set-dankclaw-bootline.sh and is
# a reboot decision — it plans no service.
if parts[2].startswith("halogen-fabric-clock"):
_add(services, "frank-dragoon")
return services
if len(parts) >= 4 and parts[1] == "authentik" and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"authentik-postgres", "authentik-server", "authentik-worker", "authentik-outpost", "authentik-outpost-ldap"}:
@@ -363,7 +386,7 @@ def services_for_path(path: str, target: str) -> set[str]:
# consumer (prometheus before grafana, loki before alloy).
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
_add(services, name)
return services
if len(parts) >= 2 and parts[1] == "gitea":
@@ -399,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
"tautulli": {"tautulli"},
}
_add(services, *quadlet_services.get(parts[1], set()))
if len(parts) >= 2 and parts[1] == "host":
# dankserver/host/systemd/media-meta-view.mount is the overlay view the
# media servers bind (scripts/media-view.sh installs it on the host),
# so editing it restarts jellyfin and plex and nothing else.
_add(services, "jellyfin", "plex")
if len(parts) >= 2 and parts[1] == "observability":
# Dankserver's node-exporter became a Quadlet unit on 2026-09-16
# (serverAppConfigs-docs/Infrastructure/Observability.md): its
@@ -406,8 +434,8 @@ def services_for_path(path: str, target: str) -> set[str]:
# node-dankserver job had nothing to scrape.
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name == "node-exporter":
_add(services, "node-exporter")
if name in {"node-exporter", "cadvisor"}:
_add(services, name)
return services
# frank-archon's quadlet-era directory left the tree on 2026-09-26
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -438,6 +466,21 @@ def services_for_path(path: str, target: str) -> set[str]:
# change to it redeploying alertmanager is the only path that
# reaches it on the host.
_add(services, "alertmanager")
if name in {"gpu-exporter.py", "gpu-exporter.service"}:
# The host-process GPU exporter is installed directly by the
# gpu-exporter deploy on BOTH hosts, so its script and unit select
# that service (target_services then keeps it only on the host being
# planned). Editing the collector restarts the running exporter.
_add(services, "gpu-exporter")
if name == "media-view.sh":
# The installer is what creates, mounts and reaps the view both media
# servers read, so a change to it is a restart of that pair.
_add(services, "jellyfin", "plex")
if name == "jellyfin-providers.sh":
# The deployer runs this before jellyfin restarts, so a change to
# the installer is a change to the provider state Jellyfin loads at
# start-up: it selects jellyfin and nothing else.
_add(services, "jellyfin")
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-zealot only. The ik_llama.cpp
+358 -34
View File
@@ -86,7 +86,7 @@ service_is_allowed() {
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:comfyui|dankclaw:hermes|\
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
dankserver:frank-archon|dankserver:node-exporter|\
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
@@ -252,16 +252,24 @@ install_signal_relay() {
# The relay binds the host's LAN IP (the one address a bridge container
# can reach as host.containers.internal), so keep the unauthenticated
# endpoint off the rest of the LAN: admit loopback, the host itself
# (its own health probe) and the podman bridge subnet only. Idempotent:
# -C before -I.
# (its own health probe) and the podman bridge subnet only.
# ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
# shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
# ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
# (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
# dankclaw drain aborted at the alertmanager gate twice before the cause
# was read off `nft list ruleset`). Rebuild the tail instead of inserting
# it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
# at the head (-C before -I, unchanged), then APPEND the DROP so it is
# last and the ACCEPTs decide first. Converges on poisoned hosts.
run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
|| run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
# The interpreter needs a moment to bind after restart; a one-shot curl
# raced the socket on 2026-09-17 and failed a healthy deploy.
local i
@@ -275,14 +283,81 @@ install_signal_relay() {
return 1
}
# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
# it runs python3 from the checkout on the host so it can read the DRM/cgroup
# sysfs and shell out to nvidia-smi. Same two-address bind posture as
# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
deploy_gpu_exporter() {
local unit_dir="$HOME/.config/systemd/user" lan dropin i
case "$TARGET" in
dankclaw) lan=192.168.0.143 ;;
dankserver) lan=192.168.0.235 ;;
*) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
esac
mkdir -p "$unit_dir"
cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
dropin="$unit_dir/gpu-exporter.service.d"
mkdir -p "$dropin"
printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
> "$dropin/bind.conf"
systemctl --user daemon-reload
systemctl --user enable gpu-exporter.service
systemctl --user restart gpu-exporter.service
# The interpreter needs a moment to bind after restart (the same race that
# once failed a healthy signal-relay deploy).
for i in $(seq 1 15); do
curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
sleep 1
done
echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
return 1
}
copy_loki_config() {
run_privileged mkdir -p /opt/loki/config
run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
}
copy_alloy_config() {
run_privileged mkdir -p /opt/alloy/config
run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
# The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
# carries the host's name, so Loki streams say WHICH machine's journal a
# line came from without a second config file per host.
# *_CONF_ROOT are test seams: the bats cases execute these copiers for
# real against a sandbox with a stubbed run_privileged.
local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
tmp="$(mktemp)"
sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
echo "alloy config: host-label substitution failed" >&2
rm -f "$tmp"
return 1
fi
# Syntax gate: alloy's config language has no schema — a misplaced
# attribute (e.g. `on_error` outside the json stage) parses only at RUN
# time, and the unit then crash-loops with a healthy-looking deploy.
# Validate against the very image that will run it, before installing.
# Skipped only in the bats sandbox (no image there).
if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
if ! podman run --rm -v "$tmp:/c.alloy:ro" \
--entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
validate /c.alloy; then
echo "alloy config: validate FAILED against the pinned image" >&2
rm -f "$tmp"
return 1
fi
fi
run_privileged mkdir -p "$root/config"
run_privileged cp "$tmp" "$root/config/config.alloy"
rm -f "$tmp"
}
copy_blackbox_config() {
# Single-file mount target: the unit binds this exact path into
# /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
# BLACKBOX_CONF_ROOT is the same test seam.
local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
run_privileged mkdir -p "$root"
run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
}
copy_grafana_provisioning() {
@@ -291,14 +366,25 @@ copy_grafana_provisioning() {
# datasource would be invisible to review, lost with the data volume, and
# unknown to anyone reading the repository. The unit already mounts
# /opt/grafana/provisioning read-only, so this only has to populate it.
run_privileged mkdir -p /opt/grafana/provisioning/datasources \
/opt/grafana/provisioning/dashboards
# GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
# copy runs unprivileged (the bats case executes this function for real).
local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
run_privileged mkdir -p "$root/provisioning/datasources" \
"$root/provisioning/dashboards"
run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
/opt/grafana/provisioning/datasources/prometheus.yml
"$root/provisioning/datasources/prometheus.yml"
run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
/opt/grafana/provisioning/dashboards/provider.yml
run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
/opt/grafana/provisioning/dashboards/llm-fleet.json
"$root/provisioning/dashboards/provider.yml"
# EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
# turns each file in the directory into a dashboard, so a copier that names
# files one by one silently drops new boards (measured 2026-10-21: the
# containers board existed in the repo since its commit and was NEVER
# provisioned — the panel-census browser check could not find it while the
# hand-placed older boards happened to be on disk). Same rule as the quadlet
# tier: the directory is the contract, the copy is generated state.
for dash in shared/grafana/provisioning/dashboards/*.json; do
run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
done
}
render_caddyfile() {
@@ -487,13 +573,15 @@ seed_grafana_env() {
|| { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; }
}
# The five observability containers left compose on 2026-09-11 (they were
# started by hand with podman-compose and never came back after a reboot).
# They are ordinary rootless Quadlet units now: same images, same mounts, same
# arguments, same container names, so nothing that addresses them by name —
# Caddy, Prometheus' own scrape config, Alloy's loki.write endpoint — changed.
# Only the published ports did: the three HTTP UIs bind 127.0.0.1, because the
# compose services published on 0.0.0.0 and none of them authenticates.
# The observability containers left compose on 2026-09-11 (they were started by
# hand with podman-compose and never came back after a reboot). Five are ordinary
# rootless Quadlet units: same images, same mounts, same arguments, same
# container names, so nothing that addresses them by name — Caddy, Prometheus'
# own scrape config — changed. Only the published ports did: the three HTTP UIs
# bind 127.0.0.1, because the compose services published on 0.0.0.0 and none of
# them authenticates. Alloy is the exception: a ROOT system service on
# Network=host, because reading the host journal needs privileges a rootless
# container cannot get. It is handled by deploy_alloy, not the loop below.
deploy_observability() {
local service="$1"
# Observability units live under <target>/observability/quadlet: Dankclaw
@@ -512,16 +600,17 @@ deploy_observability() {
copy_loki_config
build_compose_image loki
;;
alloy)
copy_alloy_config
build_compose_image alloy
;;
grafana)
copy_grafana_provisioning
seed_grafana_env || return 1
;;
node-exporter)
;;
cadvisor)
;;
blackbox-exporter)
copy_blackbox_config
;;
*)
echo "unknown observability service: $service" >&2
return 2
@@ -534,11 +623,75 @@ deploy_observability() {
alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
# Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
# is not bindable from the host netns — see the unit header).
blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
esac
}
# Mirror an image the deployer built rootless into the ROOT image store. A root
# quadlet reads root storage; a root build (build_compose_image is rootless) is
# not possible on this host — the build container's resolver fails DNS — so the
# only way to get a root-readable copy is save-and-load. Compares image ids so a
# rebuilt alloy actually re-lands in root storage instead of leaving a stale tag.
mirror_image_to_root() {
local ref="$1" rid
rid="$(podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" || {
echo "ERROR: rootless image $ref is not built; cannot mirror to root" >&2
return 1
}
if [[ "$(run_privileged podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" == "$rid" ]]; then
return 0
fi
local tar; tar="$(mktemp /var/tmp/alloy-mirror.XXXXXX.tar)"
podman save -o "$tar" "$ref"
run_privileged podman load -i "$tar"
rm -f "$tar"
}
# Install a quadlet into the SYSTEM generator dir (root manager) instead of the
# user dir. Boot persistence is the unit's own [Install] section (multi-user
# target); the v1 system generator creates the .wants on daemon-reload, so —
# like the user units — no `systemctl enable` (generated units cannot be enabled
# here).
install_system_quadlet() {
local source_dir="$1" unit="$2"
run_privileged mkdir -p /etc/containers/systemd
run_privileged cp "$source_dir/$unit.container" /etc/containers/systemd/
run_privileged systemctl daemon-reload
}
# Alloy as a ROOT system service on Network=host. See the unit file: a rootless
# container cannot read the 0640 journal files, and it cannot reach Loki's
# 127.0.0.1:3100 publish from a bridge netns. Root + host network fixes both;
# the price is that this one container runs as root and on the host network.
deploy_alloy() {
copy_alloy_config
build_compose_image alloy # rootless build (resolver works)
mirror_image_to_root localhost/franksdank/alloy:latest
# Retire the rootless unit BEFORE the system one starts: both managers
# generate alloy.service, and two containers named "alloy" both binding
# 127.0.0.1:12345 cannot coexist. Move the user quadlet out of the generator
# dir (it regenerates from there) and stop the user unit.
if [[ -f "$quadlet_unit_dir/alloy.container" ]] ||
systemctl --user is-active --quiet alloy.service 2>/dev/null; then
systemctl --user stop alloy.service 2>/dev/null || true
if [[ -f "$quadlet_unit_dir/alloy.container" ]]; then
local trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-alloy-root-cutover"
mkdir -p "$trash"
mv -- "$quadlet_unit_dir/alloy.container" "$trash/alloy.container"
echo "retired rootless alloy.container -> $trash"
fi
systemctl --user daemon-reload
fi
install_system_quadlet dankclaw/observability/quadlet alloy
run_privileged systemctl restart alloy.service
wait_for_health alloy http://127.0.0.1:12345/-/healthy 60
}
build_compose_image() {
local service="$1" image cfhash oldhash containerfile
# Every file the image COPIES IN, beside the Containerfile. The skip below
@@ -1049,7 +1202,42 @@ deploy_romm_stack() {
systemctl --user restart "$service.service"
}
ensure_media_view() {
# Both media servers read their library through the SSD-backed overlay view
# (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
# array below stays read-only and every sidecar Jellyfin writes lands on the
# NVMe.
# Raise it BEFORE either container starts — a start against a bare view
# directory would let a scan clear the library database. The units also carry
# ConditionPathIsMountPoint, so a missing view skips the start rather than
# damaging anything. A relayout must be followed by `media-view.sh reap`
# (see the Media Relayout page), which this is not.
run_privileged bash scripts/media-view.sh install
bash scripts/media-view.sh verify
}
ensure_provider_keys() {
# Jellyfin's metadata-provider keys live in the container-owned plugin
# configuration under the data root, never in git; scripts/jellyfin-providers.sh
# is what places them, reading the per-service host env file (the fleet
# convention: grafana, caddy, pihole, copyparty, romm, seerr) that holds
# TMDB_API_KEY= and TVDB_API_KEY=, mode 600.
# Missing keys mean a keyless provider — a degraded library, not a broken
# deploy — so this warns rather than failing the restart. But when the keys
# ARE placed it must happen before the restart: the plugin and its keys are
# read at start-up, so a restart first would boot the old state.
local keys=/home/dank/Containers/jellyfin/provider-keys
if [ -f "$keys" ]; then
bash scripts/jellyfin-providers.sh install
else
echo "WARN: $keys is absent - provider keys NOT placed"
echo "WARN: create it (TMDB_API_KEY=, TVDB_API_KEY=, chmod 600) and re-deploy jellyfin"
fi
}
deploy_jellyfin() {
ensure_media_view
ensure_provider_keys
cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload
if ! systemctl --user is-active -q jellyfin.service; then
@@ -1137,6 +1325,7 @@ deploy_copyparty() {
}
deploy_plex() {
ensure_media_view
cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
systemctl --user daemon-reload
if ! systemctl --user is-active -q plex.service; then
@@ -1209,11 +1398,17 @@ deploy_authentik() {
# generated". Boot persistence comes from [Install] WantedBy=default.target
# inside each .container file, which the generator honors at user-manager
# start (linger must stay on for the deploy user).
systemctl --user start authentik-postgres.service
# RESTART, not start: `start` on an already-active unit is a no-op, so an
# image bump would install the new digest and keep serving the old
# container while every gate stayed green (measured 2026-10-30: server and
# worker stayed on the previous release beside an outpost that moved,
# because only the outpost handler restarts). The generator has already
# re-rendered the unit from the .container, so a restart is what applies it.
systemctl --user restart authentik-postgres.service
wait_container_healthy authentik-postgres 120 || return 1
systemctl --user start authentik-server.service
systemctl --user restart authentik-server.service
wait_container_healthy authentik-server 300 || return 1
systemctl --user start authentik-worker.service
systemctl --user restart authentik-worker.service
}
deploy_authentik_outpost() {
@@ -1310,9 +1505,10 @@ deploy_anubis() {
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
# run a 200,000 window with its ~45k fit gap; the pool still carries two
# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -1328,7 +1524,10 @@ deploy_anubis() {
# artifact would leave a restart-looping unit and a dark endpoint. The
# port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released.
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
# The payload dir is its own twin because the NPU lane's containment gate below
# reads it; every artifact path under it stays a full literal as usual.
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
LLM_DRAGOON_CKPT="${LLM_DRAGOON_CKPT:-/home/dank/models/halogen-v2-abliterated/qwen38-flash-next-v2-abliterated.hgn}"
LLM_DRAGOON_CKPT_BYTES="${LLM_DRAGOON_CKPT_BYTES:-66687678432}"
LLM_DRAGOON_NGRAM="${LLM_DRAGOON_NGRAM:-/home/dank/models/halogen-v2-abliterated/qwen38-flash-next-ngram.hgn}"
@@ -1337,6 +1536,18 @@ LLM_DRAGOON_VISION="${LLM_DRAGOON_VISION:-/home/dank/models/halogen-v2-abliterat
LLM_DRAGOON_VISION_BYTES="${LLM_DRAGOON_VISION_BYTES:-897916416}"
LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-v2-abliterated/tokenizer}"
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
# NPU lane (2026-10-19). HALOGEN_NPU_MODELS is the whole switch and the parity
# twin of the env file's line: empty here AND in the env = the pre-lane
# endpoint. The gates below mirror serve-frank-dragoon.sh, and the vendor's
# fabric-clock host unit (dankclaw/host/, verbatim at tag v0.16.2) is installed
# ONLY while the lane is named — on Strix Halo the iGPU and the NPU co-running
# hang the machine while the fabric clock moves, so halogen refuses to boot with
# the lane armed unless that clock is held.
LLM_DRAGOON_NPU_MODELS="${LLM_DRAGOON_NPU_MODELS:-qwen3-embedding-0.6b}"
LLM_DRAGOON_NPU_DIR="${LLM_DRAGOON_NPU_DIR:-/home/dank/models/halogen-v2-abliterated/npu}"
LLM_DRAGOON_DEV_ACCEL="${LLM_DRAGOON_DEV_ACCEL:-/dev/accel/accel0}"
LLM_DRAGOON_XRT_DIR="${LLM_DRAGOON_XRT_DIR:-/usr/lib/x86_64-linux-gnu}"
LLM_DRAGOON_NETWORK="${LLM_DRAGOON_NETWORK:-dankclaw_default}"
retire_halogen_dragoon_quadlet() {
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1415,6 +1626,77 @@ deploy_llm_dragoon() {
fi
done
# The container network has to exist before the run line names it (a typo
# there is a restart-looping unit, not a config error anyone reads).
if [[ -n "$LLM_DRAGOON_NETWORK" ]] && ! podman network exists "$LLM_DRAGOON_NETWORK"; then
echo "ERROR: podman network $LLM_DRAGOON_NETWORK does not exist" >&2
echo "frank-dragoon: create it or blank FRANK_DRAGOON_NETWORK (pasta default)" >&2
return 1
fi
# --- NPU lane: the same predicates the serve script checks --------------
if [[ -n "$LLM_DRAGOON_NPU_MODELS" ]]; then
# One door into the container: the read-only /models mount of the
# payload dir. A staging dir outside it needs a nested bind whose
# mountpoint runc cannot create through a read-only parent, so the
# parity twin of the serve gate refuses the shape before the restart.
if [[ "$LLM_DRAGOON_NPU_DIR/" != "$LLM_DRAGOON_MODEL_DIR"/* ]]; then
echo "ERROR: NPU staging dir $LLM_DRAGOON_NPU_DIR is outside the payload dir $LLM_DRAGOON_MODEL_DIR — the lane's files reach the container only through the read-only /models mount" >&2
return 1
fi
if [[ ! -e "$LLM_DRAGOON_DEV_ACCEL" ]]; then
echo "ERROR: NPU node $LLM_DRAGOON_DEV_ACCEL absent — amdxdna is not bound (is iommu=pt still on the boot line?)" >&2
return 1
fi
# Access, not existence: a rootless container cannot open a root:render
# 0660 node (its namespace maps one uid/gid, and keep-groups cannot grant
# an unmapped gid — measured 2026-10-19). Install the mode rule, re-trigger
# the node, then PROVE the node is other-accessible before the restart.
run_privileged install -m 0644 dankclaw/udev/98-halogen-npu-access.rules \
/etc/udev/rules.d/98-halogen-npu-access.rules
run_privileged udevadm control --reload-rules
run_privileged udevadm trigger --sysname-match="$(basename "$LLM_DRAGOON_DEV_ACCEL")"
# (`-o FILE` is not "other-writable" — it means "exists", and `-o` is also
# the OR operator inside test), so the bit is read straight off the mode.
if [[ $(( 8#$(stat -c '%a' "$LLM_DRAGOON_DEV_ACCEL") & 8#2 )) -eq 0 ]]; then
echo "ERROR: $LLM_DRAGOON_DEV_ACCEL is still $(stat -c '%a %U:%G' "$LLM_DRAGOON_DEV_ACCEL") after installing the access rule — the lane cannot open it rootless" >&2
return 1
fi
for xl in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
if [[ ! -e "$LLM_DRAGOON_XRT_DIR/$xl" ]]; then
echo "ERROR: XRT NPU library $LLM_DRAGOON_XRT_DIR/$xl absent (Ubuntu libxrt2 + libxrt-npu2)" >&2
return 1
fi
done
for nm in ${LLM_DRAGOON_NPU_MODELS//,/ }; do
if [[ ! -d "$LLM_DRAGOON_NPU_DIR/$nm" ]]; then
echo "ERROR: NPU model $nm not staged in $LLM_DRAGOON_NPU_DIR" >&2
echo "frank-dragoon: stage it: bash scripts/download-frank-dragoon-npu.sh" >&2
return 1
fi
done
# The hold must exist BEFORE the engine starts and survive every boot
# (a resume hands the clock back too — the unit's WantedBy covers it).
run_privileged install -m 0755 dankclaw/host/halogen-fabric-clock \
/usr/local/sbin/halogen-fabric-clock
run_privileged install -m 0644 dankclaw/host/halogen-fabric-clock.service \
/etc/systemd/system/halogen-fabric-clock.service
run_privileged systemctl daemon-reload
run_privileged systemctl enable --now halogen-fabric-clock.service
if ! run_privileged /usr/local/sbin/halogen-fabric-clock status; then
echo "ERROR: fabric clock is not held — halogen refuses to start with HALOGEN_NPU_MODELS set" >&2
echo "frank-dragoon: journalctl -u halogen-fabric-clock.service -n 40" >&2
return 1
fi
elif [[ -x /usr/local/sbin/halogen-fabric-clock ]]; then
# Lane rolled back: hand the clock back to the driver and drop the boot
# hook. `disable` deletes the INSTALLED copy of this unit — unlike the
# tracked LLM unit symlinks the fleet's disable law protects, this file
# is repo-owned and the next lane-on deploy reinstalls it.
run_privileged systemctl disable halogen-fabric-clock.service 2>/dev/null || true
run_privileged /usr/local/sbin/halogen-fabric-clock release || true
fi
retire_halogen_dragoon_quadlet
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
# the container) before waiting on the port. The stray sweep is keyed
@@ -1434,6 +1716,33 @@ deploy_llm_dragoon() {
run_privileged systemctl enable frank-dragoon.service
run_privileged systemctl restart frank-dragoon.service
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
# The lane's own gate. /health proves the engine is up, nothing about the
# NPU programs; one real embedding is the observable output. The first
# request after a boot is a warmup artifact (fleet law), so retry before
# calling a booted lane dead.
if [[ -n "$LLM_DRAGOON_NPU_MODELS" ]]; then
for nm in ${LLM_DRAGOON_NPU_MODELS//,/ }; do
lane_ok=0
for _try in 1 2 3 4 5; do
if curl -fsS -m 60 -H 'content-type: application/json' \
-d "{\"model\":\"$nm\",\"input\":[\"frank-dragoon npu lane gate\"]}" \
http://127.0.0.1:7770/v1/embeddings 2>/dev/null \
| python3 -c 'import json,sys
v = json.load(sys.stdin)["data"][0]["embedding"]
raise SystemExit(0 if isinstance(v, list) and len(v) and all(isinstance(x, (int, float)) for x in v) else 1)'; then
lane_ok=1
break
fi
sleep 10
done
if [[ "$lane_ok" != 1 ]]; then
echo "ERROR: NPU model $nm answered no vector on /v1/embeddings" >&2
echo "frank-dragoon: read the boot ledger: tail -100 /home/dank/logs/frank_dragoon.log" >&2
return 1
fi
echo "frank-dragoon: NPU lane verified — $nm embedded"
done
fi
}
# deploy_llm_zealot — frank-zealot is llama.cpp (PrismML fork build) on the
@@ -1463,6 +1772,12 @@ LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism-b10
# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger)
# lands AFTER the engine's fit check, so it is not inside the q8_0 budget.
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
# The refusal adapter is part of the payload identity now, so the deployer
# checks it beside the other two files — but it has its OWN fetcher, so a
# missing adapter must not send the operator to the bonsai9 script. Integrity
# (bytes + sha256) is the fetcher's job at download time; this gate only asks
# whether the file the env names exists, exactly like the two above.
LLM_ZEALOT_LORA="${LLM_ZEALOT_LORA:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf}"
deploy_llm_zealot() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
@@ -1484,6 +1799,13 @@ deploy_llm_zealot() {
return 1
fi
done
if [[ ! -f "$LLM_ZEALOT_LORA" ]]; then
echo "ERROR: frank-zealot refusal adapter absent at $LLM_ZEALOT_LORA" >&2
echo "frank-zealot: run scripts/download-frank-zealot-uncensored-lora.sh first;" >&2
echo "frank-zealot: the env names the adapter, so the unit would refuse to boot" >&2
echo "frank-zealot: without it — starting the unit anyway is a crash loop." >&2
return 1
fi
relink_unit frank-zealot
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-zealot.service
@@ -1815,7 +2137,9 @@ deploy_service() {
copyparty) deploy_copyparty ;;
plex) deploy_plex ;;
tautulli) deploy_tautulli ;;
prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
alloy) deploy_alloy ;;
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
gpu-exporter) deploy_gpu_exporter ;;
comfyui) deploy_comfyui ;;
*) deploy_compose_service "$service" ;;
esac
+6 -4
View File
@@ -133,13 +133,15 @@ fi
valid_service() {
local service="$1"
case "$TARGET:$service" in
# dankclaw:searxng-network is client-only: a network quadlet with no
# remote dispatch handler, so the remote gate does not (must not) list it.
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
dankclaw:gitea-server|dankclaw:homepage|\
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:alertmanager|dankclaw:blackbox-exporter|\
dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:open-notebook|\
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-archon|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
dankserver:prowlarr|dankserver:frank-archon|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
+151 -24
View File
@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
[--min-route-mult 3.0]
[--min-route-mult 3.0] [--target-count 0]
--target-count N fills the byte bill: the bars pick the base set exactly as
the bar-only pass does, then the energy ranking admits experts one at a time
(each with its parity partner) until the count lands. The energy bar NEVER
relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
walks down only when the ranking cannot produce that many candidates.
0 takes the bars as given.
"""
import argparse
import collections
import glob
import os
import re
@@ -32,6 +39,7 @@ import sys
import numpy as np
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
return k, rank, int(rows), tr
def main():
ap = argparse.ArgumentParser()
ap.add_argument("calib_dir")
ap.add_argument("--min-share", type=float, default=0.03)
ap.add_argument("--min-route-mult", type=float, default=3.0)
a = ap.parse_args()
# layer -> global expert -> (rank, rows, energy)
data = {}
for path in sorted(glob.glob(os.path.join(
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
m = NAME_RE.match(os.path.basename(path))
if not m:
continue
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
k, _, rows, tr = read_gram(path)
data.setdefault(layer, {})[expert] = (rank, rows, tr)
if not data:
raise SystemExit("no per-expert down grams found")
def select(data, min_share, min_route_mult):
"""(cast lines, report lines, count) for one pair of bars."""
lines, report = [], []
total = 0
for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
for e, (rank, rows, tr) in experts.items():
es, rs = tr / tot_e, rows / max(tot_r, 1)
rec = (e, rank, es, rs)
if es >= a.min_share and es >= a.min_route_mult * rs:
if es >= min_share and es >= min_route_mult * rs:
qual.append(rec)
elif es > 0:
pool.append(rec)
@@ -107,10 +96,148 @@ def main():
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
f"ratio {es/max(rs,1e-12):6.1f}")
total += len(chosen)
return lines, report, total
def parity_repair(chosen, by_half, report, only=None):
"""Add the next-best candidate to every half — or to `only` — holding an odd count.
The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
of bf16 experts on a half is not a style problem: the two tables would not match.
`by_half` is energy-descending, so the first unchosen candidate is the next most
energetic, which is the expert worth keeping. Returns how many were added.
"""
added = 0
for key in ([only] if only is not None else sorted(by_half)):
layer, half = key
n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
if n % 2 == 0:
continue
for e, es, rs in by_half.get(key, []):
if (layer, e) not in chosen:
chosen[(layer, e)] = (es, rs)
added += 1
report.append(f"blk.{layer}: half {half} odd -> added e{e} "
f"(energy {es*100:.2f}%)")
break
else:
report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
f"left to make it even")
return added
def select_to_count(data, min_share, min_route_mult, target):
"""(lines, report, total, cutoff) for the set that LANDS the count.
The bars decide the base set; the count is then filled from the energy ranking,
one expert at a time, each addition taking its half's next-best candidate with it
so every rank's half stays even. Only the RATIO bar walks down to find candidates
— it is the law that says an expert is disproportionate and not merely busy, and
the caller's byte bill has to be filled — while the ENERGY bar holds, because that
bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
the set bottoms out at: the bar this campaign actually converted with.
"""
report = []
shares, ranked = {}, []
for layer in sorted(data):
experts = data[layer]
tot_e = sum(v[2] for v in experts.values())
tot_r = sum(v[1] for v in experts.values())
if tot_e <= 0 or tot_r <= 0:
report.append(f"blk.{layer}: empty recording, skipped")
continue
for e, (_rank, rows, tr) in experts.items():
es, rs = tr / tot_e, rows / max(tot_r, 1)
if es <= 0:
continue
shares[(layer, e)] = (es, rs)
ranked.append((es, layer, e, rs))
ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
by_half = {}
for es, layer, e, rs in ranked:
by_half.setdefault((layer, e // 256), []).append((e, es, rs))
chosen = {k: v for k, v in shares.items()
if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
total = len(chosen)
total += parity_repair(chosen, by_half, report)
mult = min_route_mult
while True:
pool = [(es, l, e, rs) for es, l, e, rs in ranked
if es >= mult * rs and (l, e) not in chosen]
if len(pool) + total >= target or mult <= 0.0:
break
mult -= 0.5
report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
f"for {target - total} slots; relaxing toward 0")
for es, layer, e, rs in pool:
if total >= target:
break
if (layer, e) in chosen:
continue
chosen[(layer, e)] = (es, rs)
total += 1
report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
lines, cutoff = [], None
for (layer, e) in sorted(chosen):
es, rs = chosen[(layer, e)]
cutoff = es if cutoff is None else min(cutoff, es)
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight cast dtype=bf16")
return lines, report, total, cutoff
def main():
ap = argparse.ArgumentParser()
ap.add_argument("calib_dir")
ap.add_argument("--min-share", type=float, default=0.03)
ap.add_argument("--min-route-mult", type=float, default=3.0)
ap.add_argument("--target-count", type=int, default=0)
a = ap.parse_args()
# layer -> global expert -> (rank, rows, energy)
data = {}
for path in sorted(glob.glob(os.path.join(
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
m = NAME_RE.match(os.path.basename(path))
if not m:
continue
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
k, _, rows, tr = read_gram(path)
data.setdefault(layer, {})[expert] = (rank, rows, tr)
if not data:
raise SystemExit("no per-expert down grams found")
if a.target_count > 0:
lines, report, total, cutoff = select_to_count(
data, a.min_share, a.min_route_mult, a.target_count)
bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
else:
lines, report, total = select(data, a.min_share, a.min_route_mult)
bar = f"min-share {a.min_share:.3g}"
# The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
# every layer must hold an even number of bf16 experts. Judged on the output and
# not on the internal sets, so a slip in either pass is caught here rather than at
# engine load time.
halves = collections.Counter()
for rule in lines:
m = CAST_RE.match(rule)
if m:
halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
odd = {key: n for key, n in halves.items() if n % 2}
if odd:
print("PARITY LAW BROKEN: " + ", ".join(
f"blk.{lay} half {half} holds {n} protected experts"
for (lay, half), n in sorted(odd.items())), file=sys.stderr)
raise SystemExit(2)
print("\n".join(lines))
print("\n".join(report), file=sys.stderr)
print(f"total protected: {total}", file=sys.stderr)
print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
if __name__ == "__main__":
+134
View File
@@ -0,0 +1,134 @@
#!/usr/bin/env bash
# download-frank-dragoon-npu.sh — stage the NPU half of frank-dragoon: the
# vendor-precompiled AIE programs halogen-flash-server runs on the XDNA2 NPU
# when HALOGEN_NPU_MODELS names them. These are NOT model weights you can
# quantize yourself: XDNA2 executes only programs compiled by AMD's AIE
# compiler, so the only source is peonist-ai's per-model NPU repo, and the
# engine checks every file's size and sha256 against its own catalog at boot
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
# whose files do not match.
#
# The manifest below IS that catalog block (dumped from the pinned 0.18.20
# image on 2026-10-30; the 0.16.2 block this replaced named different device
# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
# stays unset; the vendor's own README: with it unset the container makes no
# outbound connections at all).
#
# The fetch law, same as every sibling script: curl -f (an access error must
# not become a "successful" 153-byte artifact), the exact byte count and the
# sha256 from the manifest, --verify-only to re-check a staged set.
#
# Adding a second NPU model: read its block out of the image's catalog and add
# a case below —
# podman exec frank-dragoon-halogen grep "^file <model-id> " /opt/halogen/npu/models.txt
# podman exec frank-dragoon-halogen grep "^model <model-id> " /opt/halogen/npu/models.txt
# and add the model id to HALOGEN_NPU_MODELS in frank-dragoon.env.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
# The env file owns the staging path (it must sit inside the payload dir, which
# is the container's only /models mount); an exported FRANK_DRAGOON_NPU_DIR wins
# so the bats seam can stage somewhere disposable.
DEST="${FRANK_DRAGOON_NPU_DIR:-}"
if [ -z "$DEST" ] && [ -f "$ENV_FILE" ]; then
DEST="$(sed -nE 's/^FRANK_DRAGOON_NPU_DIR=(.*)$/\1/p' "$ENV_FILE" | tail -1)"
fi
: "${DEST:?FRANK_DRAGOON_NPU_DIR is not set and $ENV_FILE names no FRANK_DRAGOON_NPU_DIR}"
HF="https://huggingface.co"
MODELS="${HALOGEN_NPU_MODELS:-}"
if [ -z "$MODELS" ] && [ -f "$ENV_FILE" ]; then
MODELS="$(sed -nE 's/^HALOGEN_NPU_MODELS=(.*)$/\1/p' "$ENV_FILE" | tail -1)"
fi
VERIFY_ONLY=0
case "${1:-}" in
--verify-only) VERIFY_ONLY=1 ;;
"") ;;
*) echo "usage: ${0##*/} [--verify-only] (models come from HALOGEN_NPU_MODELS)" >&2; exit 2 ;;
esac
[ -n "$MODELS" ] || { echo "no NPU model named — set HALOGEN_NPU_MODELS or run with it in the environment" >&2; exit 1; }
# manifest <model-id> -> repo revision + one "path bytes sha256" per file.
manifest() {
case "$1" in
qwen3-embedding-0.6b)
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
FILES=(
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
"devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
"devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
"devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
"devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
"devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
"devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
"devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
"devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
"devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
"devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
"devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
"devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
"devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
"devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
"devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
"devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
) ;;
*)
echo "unknown NPU model '$1' — add its catalog block to manifest() in this script" >&2
return 1 ;;
esac
}
for MODEL in ${MODELS//,/ }; do
manifest "$MODEL" || exit 1
echo "== $MODEL ($NPU_REPO@$NPU_REV) =="
for entry in "${FILES[@]}"; do
read -r f want sha <<<"$entry"
path="$DEST/$MODEL/$f"
mkdir -p "$(dirname "$path")"
if [ "$VERIFY_ONLY" -eq 0 ]; then
if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
echo "HAVE $f ($want bytes) — skipping fetch"
else
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl
# DROPS the Range header across redirects, so resolve the
# effective URL first and resume against the CDN (the sibling
# 62 GiB fetch hit this on 2026-10-18).
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
# Resume ONLY into a genuinely partial file. A staged file of a
# DIFFERENT size is an artifact of another revision (a version
# bump re-cuts the device programs), and `-C -` against it
# appends the tail to the old bytes: the result is the right
# BYTE COUNT with the wrong content, which reads as a corrupt
# program, not as a fetch fault (measured 2026-10-30 on
# consts.hnpb). Fetch those from zero.
resume=()
if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
resume=(-C -)
fi
curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
if [ -L "$path" ]; then
echo "SYMLINK $path — the serve gate refuses symlinks; replace with a real file" >&2
exit 1
fi
got="$(stat -c %s "$path")"
[ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
got="$(sha256sum "$path" | cut -d' ' -f1)"
[ "$got" = "$sha" ] || { echo "SHA256 $path: $got != $sha" >&2; exit 1; }
echo "OK $f ($want bytes, sha256)"
done
done
echo "DOWNLOAD_DONE $DEST ($MODELS)"
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env bash
# Fetch the refusal-direction adapter that frank-zealot applies at load:
# AtomicChat/Ternary-Bonsai-2-9B-Uncensored-LoRA-GGUF — a **3.2 MB rank-1 LoRA**
# for prism-ml/Ternary-Bonsai-2-9B, not a checkpoint. Public repo: no auth
# header, and never a token on the command line (a Bearer token is
# world-readable in /proc/<pid>/cmdline).
#
# IT IS AN EDIT INSIDE THE COMPUTE GRAPH, NOT A SECOND MODEL. The card: B = -r
# (the refusal direction, row 23 of its own probe matrix), A = P^T(s *
# H(r^T W_fold)) built from the base pack's folded input basis, applied at every
# transformer layer (`target_layers = all`, `sites = 64`), with the embedding
# deliberately untouched because an embedding LoRA crashes this fork's graph.
# `--lora-scaled <file>:<scale>` is the flag pair: scale 1 is the card's
# "held-out refusals 0 %" point, scale 0 is the plain model again.
#
# THE RUNTIME IS A GATE, NOT A PREFERENCE. The card verified the adapter on
# `prism-b10777-e3de715` — the exact build frank-zealot runs — and says older
# forks (e.g. `prism-b10709`) refuse these packs with `unsupported
# prism.hadamard.version: 2`. The fork is already pinned for the payload's
# custom tensor type 142; this adapter adds a second reason it cannot move.
#
# PROVENANCE ASYMMETRY, STATED HONESTLY. The GGUF's own metadata names the pack
# the direction was fitted against:
# prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf
# prism.abliterate.base_sha256 = 3561df7ac7d26a71ffa991ac2db740b10841c3a350aa2676ccd3ff5d428b4e8b
# frank-zealot serves the PQ2_0 + MTP + Q8_0-embedding packing (its sha256 is
# c79fcdfa…, see download-frank-zealot-bonsai9.sh), which is NOT that file and is
# not on this host. The adapter still applies — same tensor names, and the
# metadata is provenance rather than a checksum gate — but its strength was
# calibrated on a sibling quantization. What proves the effect here is the
# behaviour of the endpoint (a refusal probe before and after the swap), not the
# metadata and not the card's table, which was measured on PTQ1_0.
#
# Fresh downloads, no resume; -f/--fail-with-body so an HTTP error is not a
# success, then byte count and sha256 (this repo is NOT LFS — the file is a
# plain git object, so the sha below is the content hash of the Hub's bytes as
# fetched, not an LFS oid). Idempotent: a complete file is skipped.
#
# Usage:
# scripts/download-frank-zealot-uncensored-lora.sh [DEST_DIR] [--verify-only]
set -euo pipefail
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Bonsai2-9B-MTP}"
REPO="AtomicChat/Ternary-Bonsai-2-9B-Uncensored-LoRA-GGUF"
# name | bytes | sha256 of the Hub's bytes
FILES=(
"Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf|3155424|861b9c099fb5cd81438ba85bad30590847c777b1e18112586be7f4f9ae5fb07b"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
IFS='|' read -r name size sha <<<"$entry"
url="https://huggingface.co/${REPO}/resolve/main/${name}"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name (from $REPO)"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$url"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+249
View File
@@ -0,0 +1,249 @@
#!/usr/bin/env bash
# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
# converts, and the source the KLD reference is recorded from.
#
# Why the base and not a finetune: the campaign converts to compare against the
# publisher's own iq4r container, and that container was built from this
# checkpoint. Base-in, base-out; anything else confounds the gate. (The
# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
# set is identical to this repo's but whose weights are not — none of its
# calibration grams, KLD reference or protected list carries over.)
#
# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
# verified the key set against this repo, so no gate letter is needed).
#
# Verification is four gates deep, because the failure this repo actually lived
# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
# HTTP 401/403 and writes the error body as the artifact — three shards of
# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
#
# 1. shard census: the file count must equal the expected shard count, so a
# partially fetched fragment tree (the .partNofM trap) cannot pass.
# 2. per-shard completeness, from the shard's OWN safetensors header: the
# header declares where the tensor data ends, so every shard's true size is
# known offline. A torn tail is named here, by file, instead of surfacing
# as a load error five hours later.
# 3. index coverage: every shard model.safetensors.index.json references
# exists and is named.
# 4. total bytes against the number the Hub declared, then, unless --no-hub,
# every file's sha256 against the Hub's LFS record — including the shards
# already on disk, so a previous partial download is never trusted.
#
# The token never enters a command line (the 2026-09-16 leak law): it is read
# into HF_TOKEN in the environment, where a local process listing is the only
# reader. This repo is ungated, so the fetch simply runs without one when no
# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
#
# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
# [--shards N] [--bytes B] [--repo ID]
# --fetch download first (resumable: re-run to continue), then verify
# --no-hub skip the Hub-side sha256 sweep; the other gates always run
# exit: 0 verified, 1 verification failed, 2 environment / usage error
set -euo pipefail
REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
# shards only, so a config file edited on the Hub cannot fail it.
EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
FETCH=0
HUB=1
while [ $# -gt 0 ]; do
case "$1" in
--fetch) FETCH=1 ;;
--no-hub) HUB=0 ;;
--dir) LOCAL_DIR="$2"; shift ;;
--shards) EXPECTED_SHARDS="$2"; shift ;;
--bytes) EXPECTED_BYTES="$2"; shift ;;
--repo) REPO="$2"; shift ;;
*) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
exit 2 ;;
esac
shift
done
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
FAILS=0
note() { echo "$@"; FAILS=$((FAILS + 1)); }
if [ "$FETCH" = "1" ]; then
# Every fetch gate is a hard stop: skipping the download silently and then
# verifying a half tree is how a campaign ships a stale payload.
HFGET="$SELF_DIR/hfget.sh"
if [ ! -f "$HFGET" ]; then
echo "REFUSED: the fetch helper is not here: $HFGET" >&2
exit 2
fi
if ! command -v jq >/dev/null 2>&1; then
echo "REFUSED: hfget.sh needs jq on PATH" >&2
exit 2
fi
mkdir -p "$LOCAL_DIR"
# The campaign's peak-disk law starts at the fetch: the checkpoint and the
# reference container built from it are ~335 GiB each, and a tree that does
# not fit is discovered hours into a transfer, when it costs the most.
free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
echo " (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
echo " nothing is left half-written)" >&2
exit 2
fi
echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
if [ -z "${HF_TOKEN:-}" ]; then
CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
if [ -f "$CREDSTORE" ]; then
HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
elif [ -f "$HOME/.cache/huggingface/token" ]; then
# The legacy single-line token file the hub library writes.
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
elif [ -f "$HOME/.hf_token" ]; then
HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
fi
fi
if [ -n "${HF_TOKEN:-}" ]; then
export HF_TOKEN
fi
bash "$HFGET" "$REPO" "$LOCAL_DIR" main
unset HF_TOKEN
fi
if [ ! -d "$LOCAL_DIR" ]; then
echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
exit 1
fi
INDEX="$LOCAL_DIR/model.safetensors.index.json"
if [ ! -f "$INDEX" ]; then
echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
exit 1
fi
echo "== gate 1: shard census ($LOCAL_DIR)"
SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
else
echo " ok: $SHARDS shards"
fi
echo "== gate 2: every shard is whole, judged by its own header"
# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
# declares where the data ends, so the file's true size is computable offline:
# 8 + header + the largest end offset. Truncation and error bodies both fall
# here, and each is reported with the file that caused it.
while IFS= read -r f; do
state=$(python3 - "$f" <<'PY'
import json, os, struct, sys
path = sys.argv[1]
size = os.path.getsize(path)
try:
with open(path, "rb") as fh:
(hlen,) = struct.unpack("<Q", fh.read(8))
if hlen == 0 or hlen > size:
print("NOT_SAFETENSORS")
raise SystemExit
header = json.loads(fh.read(hlen))
except Exception:
print("NOT_SAFETENSORS")
raise SystemExit
if not header:
print("NOT_SAFETENSORS")
raise SystemExit
end = max((v["data_offsets"][1] for v in header.values()
if isinstance(v, dict) and "data_offsets" in v), default=0)
want = 8 + hlen + end
print("OK" if want == size else f"TRUNCATED want={want} have={size}")
PY
)
case "$state" in
OK) ;;
NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
esac
done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
if [ "$FAILS" -eq 0 ]; then
echo " ok: every shard's data extent matches its size on disk"
fi
echo "== gate 3: the index's shards are all present"
INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
import json, sys
map_ = json.load(open(sys.argv[1]))["weight_map"]
print("\n".join(sorted(set(map_.values()))))
PY
)
while IFS= read -r name; do
[ -n "$name" ] || continue
if [ ! -f "$LOCAL_DIR/$name" ]; then
note "FAIL MISSING: $name is named by the index and is not on disk"
fi
done <<< "$INDEX_SHARDS"
if [ "$FAILS" -eq 0 ]; then
echo " ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
fi
echo "== gate 4: total bytes vs the Hub's declaration"
TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
else
echo " ok: $TOTAL bytes"
fi
if [ "$HUB" = "1" ]; then
echo "== gate 5: every file's sha256 vs the Hub's LFS record"
# size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
TMPD="$(mktemp -d)"
trap 'rm -rf "$TMPD"' EXIT
MANIFEST="$TMPD/tree.json"
PLAN="$TMPD/plan"
curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
-o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
python3 - "$MANIFEST" > "$PLAN" <<'PY'
import json, sys
for e in json.load(open(sys.argv[1])):
if e.get("type") != "file":
continue
sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
if sha:
print(e["path"], e["size"], sha)
PY
BAD=0
while read -r rel size sha; do
[ -n "${rel:-}" ] || continue
if [ ! -f "$LOCAL_DIR/$rel" ]; then
note "FAIL MISSING: $rel is on the Hub and not on disk"
continue
fi
have=$(stat -c %s "$LOCAL_DIR/$rel")
if [ "$have" != "$size" ]; then
note "FAIL SIZE: $rel is $have bytes, Hub says $size"
continue
fi
got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
if [ "$got" != "$sha" ]; then
note "FAIL SHA256: $rel got $got want $sha"
BAD=$((BAD + 1))
fi
done < "$PLAN"
if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
echo " ok: sha256 matches for every file"
fi
rm -rf "$TMPD"
trap - EXIT
fi
if [ "$FAILS" -gt 0 ]; then
echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
exit 1
fi
echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"
+57 -31
View File
@@ -46,31 +46,42 @@ FRANK_ARCHON_TP=2
# stays EXACT for single-sequence decode and any step of <= 25 rows — see
# MAX_SEQS x (1 + SPEC_TOKENS) below.
FRANK_ARCHON_TP_WIRE=wht6
# Decode slots. A slot costs KV, never decode; the pool is sized to this
# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
# conversation answers 400 while /health and every shorter chat work fine
# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
# instead of advertising a third slot that only exists at shorter prompts.
FRANK_ARCHON_MAX_SEQS=2
# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
# fourth conversations are backed by the cache instead of over-committing it.
FRANK_ARCHON_MAX_SEQS=4
# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
# limit, and models.yml contextWindow MUST quote this number (over-length is a
# refusal, not a truncation). The pool is sized for seqs x maxlen, not one
# sequence.
FRANK_ARCHON_MAX_MODEL_LEN=200000
FRANK_ARCHON_PLACEMENT=expert_tiered
# Pinned host pool for expert rows that do not fit VRAM. Under the budget
# resolver this is ALSO the lever that decides which experts stay on the card:
# the card's leftover goes to the expert plane first and the KV cache takes the
# rest, so growing the pool moves experts off the card and shrinking it spills
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
# refused a boot). The ledger's coverage line is the referee, not this file.
# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
# host RAM above what the expert plane's footprint needs buys no cache. Back at
# 44,928, the size that boots clean.
FRANK_ARCHON_HOST_POOL_MIB=44928
# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
# host must pin tp x this number, and the refusal that names it is
# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
# MiB of experts off the card and into the cache. Derived from the boot ledger (a
# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
# x8 link when the router picks it. The ledger's coverage line is the referee, not
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
# 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
# endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
# vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
# the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
# step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
# by this pool, so the refused calibration documents were fixed by cutting them to the
# window (scripts/rechunk-kld-corpus.py), not by widening the pool.
FRANK_ARCHON_HOST_POOL_MIB=49152
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
# floor is built from it.
FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +104,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# Step sharing. A prefill chunk that shares a step with decoding sequences holds
# the device for the chunk's whole time, and every decode row in that step waits
# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
# `auto` leaves each decode row behind the chunk a row plus its draft window and
# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
# chunk took the step and admission never ran. Interleave inserts N decode-only
# steps after a chunk step taken beside decoders: it protects their cadence and
# holds a waiting request out of those steps, so it stays 0 until the post-migration
# decode-vs-prefill buckets say it is wanted.
# that long for its next token: measured at this shape, decode-only steps 15.5 ms
# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
# while another session prefills (law and census: [[frank-archon]]).
# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
# draft window per decoding sequence (3 tokens) and an eighth of the budget per
# waiting request or waiting prefill, and the source states it will not halve for
# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
# steps. A number is the only value that applies whenever the chunk shares a step,
# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
# because below it every routed layer reads its pooled experts synchronously, so
# 512 is a trap and the vendor measures the whole job slower there. Interleave
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
# their cadence and holds a waiting request out of those steps. That is the decoder
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
# price falls on the prefilling prompt (its chunks lose every held step) and on any
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
# ALONE, so it cannot buy this back while decoders run. Meter:
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
# before N rises.
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
FRANK_ARCHON_PREFILL_INTERLEAVE=0
FRANK_ARCHON_PREFILL_INTERLEAVE=1
# Prefix cache: pinned host memory only. The disk tier was removed outright
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +142,11 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
# itself before refusing by name, so recovery is a boot, not a visit.
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
FRANK_ARCHON_TTM_MIN_MIB=150000
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
# 66,560 pool the 2026-10-29 A/B rejected).
FRANK_ARCHON_TTM_MIN_MIB=162000
FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
# this unquoted on purpose, so keep multi-flag values as one string.
+68 -10
View File
@@ -10,8 +10,11 @@
# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
# engine moves with the payload, never ahead of it.
# Image digest-pinned (a tag pull silently re-points; the digest is what was
# qualified — 0.16.2 is the version the model card names as its quickstart pin).
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
# pin) — the vendor has moved two minor versions and the payload lock is what
# the deploy's health gate plus the boot ledger decide; rollback is the
# previous digest, 0c61bf84…, on this same line.
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
# Container identity kept from the 27B era: the unit's ExecStop names it.
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
# Predecessor identity the serve script clears if it is still holding the GPUs.
@@ -49,14 +52,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
# stays at the image default, which is what was qualified) ------------------
# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
HALOGEN_MODEL_ID=frank-dragoon
# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
# the page) — a window equal to the ceiling makes every long request a 400.
HALOGEN_CTX=200704
# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
# at once. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
# RAM the pool takes is RAM the n-gram table's page cache loses.
# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
# longer prompt is a 400 naming the limit, never a truncation; stretching
# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
# page): with the window at 200,000 the gap is 62,144, which absorbs the
# chars/4 under-count. A window at or near the ceiling makes every long
# request a guaranteed 400.
HALOGEN_CTX=262144
# ONE pool behind both slots. The harness window is 200,000, so 401,408
# still carries TWO maximum harness-length conversations at once
# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
# ceiling-length requests would queue instead — buying those means raising
# the pool, and RAM the pool takes is RAM the n-gram table's page cache
# loses. Pool positions buy concurrency; the per-request window never moves
# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
HALOGEN_KV_POOL_POSITIONS=401408
# Two concurrent conversations: the MTP drafter is not slot-locked on this
# engine, so the second slot is capacity, not a speculation tax.
@@ -80,3 +90,51 @@ HALOGEN_TOP_K=20
# Weights stay pinned: the engine pins by default and only the retired w4b build
# honored HALOGEN_FLASH_PIN_TRUNK=0, which this payload rejects.
HALOGEN_WEIGHTS_LOCK=1
# --- NPU lane (XDNA2) -------------------------------------------------------
# HALOGEN_NPU_MODELS is the whole switch: EMPTY or unset = the endpoint exactly
# as it ran before this lane existed (no accel device, no XRT mount, no
# fabric-clock dependency, no /v1/embeddings). Rolling the lane back is blanking
# this one line and re-deploying — nothing else moves.
# Names are the vendor catalog's folder names (`/opt/halogen/npu/models.txt`
# inside the image) and the engine reads a model ONLY at /models/npu/<name>/ —
# HALOGEN_NPU_DIR is gone from the image and setting it is a boot refusal.
# The engine cannot fetch anything here: /models is read-only and
# HALOGEN_DOWNLOAD stays unset so the container opens no outbound connection at
# boot, so scripts/download-frank-dragoon-npu.sh must stage the files first
# (the serve gate and the deployer both refuse by name when a named model is
# missing). FRANK_DRAGOON_NPU_DIR therefore lives INSIDE the payload dir: the
# payload mount is the container's only handle on /models, and runc cannot
# mkdir a nested /models/npu mountpoint through a read-only bind — the lane's
# first boot died on precisely that (2026-10-19). Swap the payload dir and this
# path moves with it.
# Cost: ~1.5 GiB extra pinned host RAM per named model, and a hard dependency on
# the host holding the GPU fabric clock — on Strix Halo the iGPU and the NPU
# together hang the machine (and corrupt NPU results) while FCLK changes speed,
# so halogen refuses to start with this variable set unless the clock is held
# (dankclaw/host/halogen-fabric-clock; the deployer installs and gates it).
HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
# Node paths and the XRT directory are variables so the bats seam can drive
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
# searches that third prefix; without it the lane dies naming it and the
# supervisor takes the container down — 2026-10-21 boot loop). The basename of
# this directory IS the multiarch name. The vendor forbids mounting an
# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
# sysfs root the fabric-clock preflight reads (same knob the vendor's
# halogen-fabric-clock status reads; the seam points it at a fixture).
FRANK_DRAGOON_SYSFS=/sys
# Container network. Empty = podman's rootless default (`pasta`): a private
# netns that nothing else can join — podman refuses a second network while the
# primary is pasta, the published port is refused from dankclaw_default
# containers, and host.containers.internal resolves link-local, which
# open-notebook's URL guard refuses to store. Named = the app bridge, so
# open-notebook can dial http://frank-dragoon-halogen:8731 for the NPU
# endpoints. The engine's unauthenticated 8730 stays private either way (it
# binds the container's own loopback); only the API port becomes name-resolvable.
FRANK_DRAGOON_NETWORK=dankclaw_default
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
After=network-online.target
Wants=network-online.target
+21
View File
@@ -27,6 +27,16 @@
# below stop describing the process you are running. Nothing in this fleet
# pins this container; that is a decision, not an oversight.
#
# THE DEVICE LAW (since the 2026-10-19 Vulkan cutover). Synthesis computes on
# the gfx1151 iGPU — the SAME card frank-dragoon serves. It is a shared device,
# never a dedicated one, so two numbers matter and neither is what `rocm-smi`
# prints first: the memory the container may use is the RADV heap inside the
# GTT-backed aperture (the ~2 GiB DRM carve-out is not the working set), and the
# timing that matters is dragoon's, measured WITH a synthesis in flight. The
# rollback is one line: set QWENTTS_DEVICE empty, and the entrypoint passes no
# --device at all — the pre-iGPU CPU shape, no rebuild, no restart of anything
# else on the box.
#
# THE REGISTRY LAW (why the unit has an ExecStartPost at all). Registrations live
# in memory only, so a restart loses them, and re-registration alone does NOT warm
# a voice — the warm pass has to synthesize something. Without it the façade
@@ -103,6 +113,17 @@ QWENTTS_LANG=auto
# round-trip outside it) is what absorbs the queue.
QWENTTS_MAX_BATCH=1
# The compute device, by ggml device NAME. `Vulkan0` is the gfx1151 iGPU: the
# image carries the Mesa (RADV) ICD and no NVIDIA ICD, so this container's
# Vulkan enumeration has exactly one entry and it is the iGPU. The unit hands
# the container /dev/dri and keeps the render group; on this hybrid host the
# iGPU's render node is renderD129 and renderD128 is the NVIDIA card, which is
# why the unit passes the directory and lets the driver resolve the node.
#
# ROLLBACK: an empty value means no --device flag at all — the CPU shape this
# endpoint shipped with until 2026-10-19. One line, no rebuild.
QWENTTS_DEVICE=Vulkan0
# --- the boot warmer (dankclaw/voiceservice/tts-engine-warm.sh) --------------
# These three are read by the warm hook, not by tts-server. They live here
# because the hook is exec-ed into this container and therefore inherits this
+2 -2
View File
@@ -7,7 +7,7 @@
# whichever TTS engine wins the engine decision (dankclaw/voiceservice/app.py
# is a single stdlib-only file in a locally built image). Consumers — omp's
# TTS device, OpenWebUI, open-notebook — address a VOICE BY NAME
# (voice=narrator1, or the harness' voice_id=narrator1) and never see a path:
# (voice=wilwheaton, or the harness' voice_id=wilwheaton) and never see a path:
# a name is real exactly when /voices/<name>/reference.wav exists, and
# anything else is a 400 that lists the names that are.
#
@@ -65,7 +65,7 @@ TTS_UPSTREAM_MODEL=voxcpm2
# name still 400s with the list that exists — this covers the absent case only —
# and the journal marks such a line `(default)` so an unrequested voice never
# reads as a requested one. Any pack directory in /home/dank/voices is legal.
DEFAULT_VOICE=narrator1
DEFAULT_VOICE=wilwheaton
# CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
# configured": the endpoint answers a 500 that says so instead of running
+17 -4
View File
@@ -1,11 +1,12 @@
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
# the refusal-direction adapter applied at load.
#
# ONE OWNER: shape, payload identity, eras, laws and every measured number
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
# knobs and only the reason a reader editing THIS line cannot recover
# elsewhere. Where the two disagree, this file wins on values.
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
# metadata dequantizes wrong, silently.
@@ -13,6 +14,17 @@ MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.g
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
# under concurrent image requests, which is exactly --parallel 3.
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
# Refusal-direction adapter (a rank-1 LoRA, 3.2 MB) applied at load; it edits
# the compute graph, it is not a second model. Scale 1 is the card's "held-out
# refusals 0 %" point. LORA_FILE="" is the documented opt-out — the plain pack
# again, no rebuild — and LORA_SCALE=0 is the same result through the flag.
# PROVENANCE ASYMMETRY: the adapter's own prism.abliterate.base_gguf names
# Ternary-Bonsai-2-9B-PTQ1_0.gguf as the pack its direction was fitted against,
# which is NOT this packing and is not on this host. The tensor names match, so
# it applies; the strength was calibrated on a sibling quantization, so the
# effect is proven by the endpoint's behaviour, never by the metadata.
LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf
LORA_SCALE=1
# Served id — the harness/family name, not a fact about the payload.
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
ALIAS=frank-zealot
@@ -52,7 +64,8 @@ CACHE_TYPE_V=q8_0
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
# disk as the rollback path).
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
# every --spec-draft-*); half a set has segfaulted at load.
SPEC_TYPE=draft-mtp
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 2x262144 slots over one 524288 pool, unquantized f16 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF + refusal adapter (qwen35 hybrid, 3x200704 slots over one 602112 pool, f16 K + q8_0 V, MTP armed, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
After=network-online.target
Wants=network-online.target
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
gpu_utilization_percent gpu_memory_used_bytes gpu_memory_total_bytes
gpu_temperature_celsius gpu_power_watts
each labelled {gpu, vendor, model}. The scrape job adds `host`.
Two sources, both host-native — no container, no GPU passthrough:
* AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
the R9700s, so those cards legitimately have no gpu_power_watts line; a
missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
would plot as a real reading.
* NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
fails; that is tolerated and simply yields no NVIDIA series.
Default mode prints one scrape to stdout and exits (the test harness, and any
`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
wildcard, matching node-exporter's posture on these boxes.
"""
import argparse
import glob
import os
import subprocess
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
FAMILIES = [
("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
("gpu_power_watts", "gauge", "Board power draw in watts."),
]
DEFAULT_NVIDIA_CMD = (
"nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
"memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
)
def _read(path):
try:
with open(path) as fh:
return fh.read().strip()
except OSError:
return ""
def _pci_slot(device_dir):
uevent = _read(os.path.join(device_dir, "uevent"))
for line in uevent.splitlines():
if line.startswith("PCI_SLOT_NAME="):
return line.split("=", 1)[1]
return ""
def _pci_id(device_dir):
uevent = _read(os.path.join(device_dir, "uevent"))
for line in uevent.splitlines():
if line.startswith("PCI_ID="):
return "amd:" + line.split("=", 1)[1].lower()
return "amd"
def collect_amd(sys_dir):
for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
device = os.path.join(card, "device")
busy = _read(os.path.join(device, "gpu_busy_percent"))
if busy == "":
continue # not an amdgpu compute node (e.g. a render node w/o metrics)
labels = {"gpu": _pci_slot(device) or os.path.basename(card),
"vendor": "amd", "model": _pci_id(device)}
yield "gpu_utilization_percent", labels, busy
for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
("gpu_memory_total_bytes", "mem_info_vram_total")):
v = _read(os.path.join(device, leaf))
if v != "":
yield name, labels, v
temp = ""
for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
temp = _read(hwmon)
if temp != "":
break
if temp != "":
yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
power = _read(os.path.join(device, "power1_average"))
if power != "": # empty on the R9700s -> metric absent, not zero
yield "gpu_power_watts", labels, str(int(power) / 1e6)
def collect_nvidia(cmd):
if not cmd:
return
try:
out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
timeout=10, check=True).stdout
except (subprocess.SubprocessError, OSError):
return # no NVIDIA GPU / no driver: tolerate, emit nothing
for line in out.splitlines():
parts = [p.strip() for p in line.split(",")]
if len(parts) < 7:
continue
idx, name, util, mused, mtotal, temp, power = parts[:7]
labels = {"gpu": idx, "vendor": "nvidia", "model": name}
yield "gpu_utilization_percent", labels, util
yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
if temp not in ("", "[N/A]"):
yield "gpu_temperature_celsius", labels, temp
if power not in ("", "[N/A]"):
yield "gpu_power_watts", labels, power
def collect(sys_dir, nvidia_cmd):
samples = {}
for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
for name, labels, value in source:
try:
value = str(float(value)) # normalise; drop trailing junk
except ValueError:
continue
samples.setdefault(name, []).append((labels, value))
return samples
def _fmt(value):
f = float(value)
if f == int(f) and abs(f) < 1e15:
return str(int(f))
return repr(f)
def render(samples):
out = []
for name, kind, help_text in FAMILIES:
out.append("# HELP %s %s" % (name, help_text))
out.append("# TYPE %s %s" % (name, kind))
for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
out.append("%s{%s} %s" % (name, lset, _fmt(value)))
return "\n".join(out) + "\n"
class Handler(BaseHTTPRequestHandler):
sys_dir = "/sys/class/drm"
nvidia_cmd = DEFAULT_NVIDIA_CMD
def do_GET(self): # noqa: N802 (http.server API)
if self.path.split("?")[0] not in ("/metrics", "/"):
self.send_error(404)
return
body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
def main(argv):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
ap.add_argument("--port", type=int, default=9401)
ap.add_argument("--bind", action="append", default=None,
help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
"comma-list if unset. NEVER a wildcard.")
args = ap.parse_args(argv)
if not args.serve:
sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
return 0
binds = args.bind
if not binds:
env = os.environ.get("GPU_EXPORTER_BINDS", "")
binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
Handler.sys_dir = args.amdgpu_sysfs
Handler.nvidia_cmd = args.nvidia_cmd
servers = []
for b in binds:
if b in ("0.0.0.0", "::", "*"):
sys.stderr.write("refusing wildcard bind %s\n" % b)
return 2
servers.append(ThreadingHTTPServer((b, args.port), Handler))
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
# EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
# healthy in `ss` yet times out every scrape once its SYN backlog fills —
# and the loopback deploy gate cannot see it. One thread per extra bind.
for s in servers[1:]:
threading.Thread(target=s.serve_forever, daemon=True).start()
servers[0].serve_forever()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+35
View File
@@ -0,0 +1,35 @@
# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
#
# Install path: ~/.config/systemd/user/gpu-exporter.service
# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
#
# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
# updates it with no copy step), a hand-written user unit can be enabled where
# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
# it must run on the host — a container would need GPU/DRM passthrough for
# nothing gained.
#
# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
# drop-in adding THIS host's LAN address (the same two-address posture
# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
# the Dankserver instance over the LAN from Dankclaw.
[Unit]
Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
After=network-online.target
Wants=network-online.target
[Service]
# Loopback by default; the deploy drop-in appends the host LAN address.
Environment=GPU_EXPORTER_BINDS=127.0.0.1
ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
+418
View File
@@ -0,0 +1,418 @@
#!/usr/bin/env python3
"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
beside the recording. Python 3 stdlib only: numpy must NOT be required, because
this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
WHY. `rad-convert` takes its per-column activation second moments and its
expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
Only the GGUF form carries a per-expert dimension — with the legacy flat form,
or with none, every expert in a layer is weighted by the layer's aggregate,
which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
already measured everything; this tool only re-voices it, so the imatrix and the
protected-expert census (derive-protected-experts.py) are the SAME recording:
one measurement, two consumers, neither able to disagree about which experts are
hot.
INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
(arch/common/rad_arch.h, calib_write). Header law, same parse as
scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
numbering (rad_block_moe_fp8.h: ge = expert_base + e):
gram.r<rank>.blk.L.ffn_down_exps.bin the layer's pooled gram
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin expert e's own gram
`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
rows, so **the diagonal is exactly the per-column activation second moment an
imatrix wants**, and `rows` is the count the reader hands back from `count()`.
Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
put a full matrix, and the quantiser asks for columns.
OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
layer's experts folded into ONE entry, so expert 9 is slice 9 of
`blk.L.ffn_down_exps`):
blk.L.ffn_down_exps in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
gate_up input is the same hidden-state rows as its down
input, so the second moment over them is identical and
so is the routing count. Two tensor infos over one
payload: the reader maps each half independently
(imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
by base name), and copying ~200 MB to state one fact
twice buys no correctness.
blk.L.ssm_inz [k] and [1], from the pooled gram: that row's input is
the same hidden state, so its columns take the pooled
diagonal and its count is the layer's pooled rows.
Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
entry's cols do not equal the weight's width and the convert proceeds without
it: an entry the reader silently ignores is a lie in the artifact, worse than
an absent one. The corollary to check before trusting the two hidden-width
entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
expert width, so the gram's k must equal the checkpoint's hidden width (its
config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
its count stays its own routing rows — the rule the engine itself applies when it
picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
is worse than round-to-nearest, and those experts carry a fraction of a percent
of the routing). The count never inherits: popularity is `count`'s job. The count
of inherited experts is reported on stderr.
RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
so the ranks describe DISJOINT experts and the fold is a UNION into the one
[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
calib_rows). The same global expert in two per-expert files is legal only when
the numbers agree exactly; otherwise the recording is corrupt and the entry would
be whichever file sorted last, so refuse.
GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
relative to the aligned data section. Every tensor offset is padded up to
`general.alignment` and the data section itself starts aligned. That is the
classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
arithmetic point into the middle of the previous tensor's floats, and it reads
them without complaint.
chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
row total, not a sum over layers: every layer's tap accumulated over the whole
corpus, so adding the layers counts each token once per layer and inflates the
figure by the layer count. chunk_count = that row total // chunk_size, the
"chunks of chunk_size" line imatrix.cpp prints for itself.
REFUSALS, each exit 2 with a message naming what is missing:
NO_GRAMS the directory holds no RADGRAM2 down-projection gram
NOT_RADGRAM2 a gram whose magic is not RADGRAM2
SHORT a gram whose length is not 32 + k*k*4
NO_POOLED a layer with expert grams and no pooled gram to inherit from
K_MISMATCH a layer whose files disagree about k (rank pooled grams, or an
expert gram against its layer's pooled gram)
EXPERT_CONFLICT a global expert recorded twice with different numbers
EXISTS the -o path already holds a file (the trash law: move it)
CHUNK_COUNT the derived chunk count does not fit the u32 metadata field
USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
[--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
"""
import argparse
import os
import re
import struct
import sys
MAGIC = b"RADGRAM2"
HDR = 32 # 8s I I Q I + pad, rad_arch.h
KOWN_ROWS_PER_K = 4 # libquant/lq_hess.h kOwnRowsPerK
ALIGNMENT = 32 # general.alignment, and the pad unit
POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
GT_U32, GT_STR, GT_ARR = 4, 8, 9
GGML_F32 = 0
def refuse(msg):
sys.stderr.write("REFUSED %s\n" % msg)
sys.exit(2)
def read_gram(path):
"""(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
try:
with open(path, "rb") as f:
raw = f.read()
except OSError as e:
refuse("UNREADABLE %s: %s" % (path, e))
if len(raw) < HDR:
refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
% (path, len(raw), HDR))
magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
if magic != MAGIC:
refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
if k <= 0:
refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
need = HDR + k * k * 4
if len(raw) != need:
refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
% (path, len(raw), need, k))
# The diagonal only, read by stride out of the row-major matrix.
diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
for i in range(k)]
return k, rank, rows, domain, diag
def load_dir(d, verbose):
"""{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
Files outside the ffn_down_exps stem are not ours to fold: the tap names a
pooled gram after every projection it taps, and those carry other widths."""
if not os.path.isdir(d):
refuse("NO_GRAMS %s: not a directory" % d)
layers, rotated = {}, 0
for name in sorted(os.listdir(d)):
m = POOLED_RE.match(name) or EXPERT_RE.match(name)
if not m:
continue
path = os.path.join(d, name)
if not os.path.isfile(path):
continue
pooled = bool(POOLED_RE.match(name))
layer = int(m.group(2))
k, rank, rows, domain, diag = read_gram(path)
if domain != 0:
rotated += 1
L = layers.setdefault(layer, {"pooled": None, "experts": {}})
if pooled:
P = L["pooled"]
if P is None:
L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
elif P["k"] != k:
refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
" disagree about the width"
% (layer, P["paths"][0], P["k"], path, k))
else:
# calib_sum: each rank summed the same stem over its own rows.
P["rows"] += rows
P["paths"].append(path)
for i in range(k):
P["diag"][i] += diag[i]
if verbose:
sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
% (layer, name, k, rows))
continue
e = int(m.group(3))
own = L["experts"].get(e)
if own is None:
L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
"path": path, "ranks": [rank]}
if verbose:
sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
% (layer, rank, e, rows))
continue
if own["rows"] != rows or own["diag"] != diag:
refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
" (rows=%d) record the same global expert with different"
" numbers" % (layer, e, own["path"], own["rows"], path, rows))
own["ranks"].append(rank)
sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
" and %s; folded once\n"
% (layer, e, own["path"], name))
if not layers:
refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
"[.<expert>].bin files in the directory" % d)
return layers, rotated
def build_entries(layers, min_rows, verbose):
"""([ (name, dims, payload|None, key) ], census, layers emitted)."""
entries = []
census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
for layer in sorted(layers):
P, E = layers[layer]["pooled"], layers[layer]["experts"]
if E and P is None:
refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
" expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
" under the row threshold would have nothing to inherit"
% (layer, len(E), layer))
if P is None:
continue
k = P["k"]
census["layers"] += 1
census["rows"] = max(census["rows"], P["rows"])
dn = "blk.%d.ffn_down_exps" % layer
# A staging key names ONE payload. A gate_up entry reuses the down
# layer's bytes by carrying the same key with no payload of its own;
# in_sum2 and counts are different bytes and must never share a key.
ksum, kcnt = (layer, "sum2"), (layer, "counts")
kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
# One entry holds one width: an expert gram recorded at another k would
# inherit (or be inherited by) a diagonal of the wrong length, and the
# reader would then answer `cols` from whichever file set dims[0].
for e in sorted(E):
if E[e]["k"] != k:
refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
" layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
E[e]["k"], k))
if not E:
# No per-expert recording for this layer: emit the dense form
# ([k], [1]) rather than 512 identical slices, which would claim a
# per-expert measurement that was never taken.
entries.append((dn + ".in_sum2", [k],
struct.pack("<%df" % k, *P["diag"]), ksum))
entries.append((dn + ".counts", [1],
struct.pack("<f", float(P["rows"])), kcnt))
sys.stderr.write("note: layer %d has no per-expert grams; %s is"
" dense over the pooled gram\n" % (layer, dn))
else:
floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
n_expert = max(E) + 1
sum2, counts = bytearray(), bytearray()
for e in range(n_expert):
own = E.get(e)
if own is not None and own["rows"] >= floor:
src = own["diag"]
rows = own["rows"]
else:
src = P["diag"]
rows = own["rows"] if own is not None else 0
census["inherited"] += 1
sum2 += struct.pack("<%df" % k, *src)
counts += struct.pack("<f", float(rows))
census["experts"] += 1
census["routing"] += rows
if min(E) != 0:
sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
" those slices carry the pooled gram at count 0\n"
% (layer, min(E) - 1))
entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
gu = "blk.%d.ffn_gate_up_exps" % layer
entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
entries.append((gu + ".counts", [1, n_expert], None, kcnt))
if verbose:
sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
" own-gram floor=%d\n"
% (layer, k, n_expert, P["rows"], floor))
inz = "blk.%d.ssm_inz" % layer
entries.append((inz + ".in_sum2", [k],
struct.pack("<%df" % k, *P["diag"]), kpgm))
entries.append((inz + ".counts", [1],
struct.pack("<f", float(P["rows"])), kpcnt))
return entries, census
def enc_str(v):
b = v.encode("utf-8")
return struct.pack("<Q", len(b)) + b
def write_gguf(path, entries, datasets, chunk_size, chunk_count):
"""Write GGUF v3 and return its size in bytes.
One payload per key: entries that share a key share the bytes and the
offset, so a second tensor info states the same fact without copying it."""
kv = bytearray()
kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", ALIGNMENT)
kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", chunk_size)
kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+ struct.pack("<I", chunk_count)
kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+ struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
for d in datasets:
kv += enc_str(d)
head = bytearray()
head += b"GGUF" + struct.pack("<I", 3)
head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
head += kv
payloads = {}
for name, _dims, payload, key in entries:
if payload is None:
continue
prev = payloads.get(key)
if prev is not None and prev != payload:
# Two different payloads on one key would ship whichever was
# appended last while BOTH tensor infos pointed at it: the counts
# payload landing over the in_sum2 payload is exactly how a
# hand-rolled GGUF writer loses half its data without an error.
refuse("INTERNAL %s: two different payloads share one staging key"
% name)
payloads.setdefault(key, payload)
infos = bytearray()
offsets, order, size = {}, [], 0
for name, dims, payload, key in entries:
if key not in payloads:
refuse("INTERNAL %s: no payload staged for its key" % name)
if key not in offsets:
offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
size = offsets[key] + len(payloads[key])
order.append(key)
infos += enc_str(name) + struct.pack("<I", len(dims))
for d in dims:
infos += struct.pack("<Q", d)
infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
body = bytes(head) + bytes(infos)
data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
with open(path, "wb") as f:
f.write(body)
f.write(b"\0" * (data_start - len(body)))
# The offsets in the tensor infos are the file's law, not a hint: each
# payload must land at its own padded offset. Writing them back to back
# instead leaves every reader that trusts the info — imatrix.cpp does —
# reading its neighbour's floats: plausible values, no error, wrong
# quantisation.
here = data_start
for key in order:
f.write(b"\0" * (data_start + offsets[key] - here))
f.write(payloads[key])
here = data_start + offsets[key] + len(payloads[key])
return f.tell()
def main():
ap = argparse.ArgumentParser(
description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
ap.add_argument("calib_dir")
ap.add_argument("-o", "--out", required=True)
ap.add_argument("--datasets", default="",
help="comma-separated names for imatrix.datasets")
ap.add_argument("--chunk-size", type=int, default=4096,
help="rows per chunk for imatrix.chunk_size (default 4096)")
ap.add_argument("--min-rows", type=int, default=0,
help="own-gram row threshold (default 4*k, the engine's"
" kOwnRowsPerK law)")
ap.add_argument("-v", "--verbose", action="store_true")
a = ap.parse_args()
if os.path.exists(a.out):
# The repo's deletion law: an artifact that exists is moved to trash by
# a human, never silently overwritten by a re-run of the tool that made
# it, so a second convert cannot destroy the imatrix the first one fed.
refuse("EXISTS %s: an imatrix is already there; move it to"
" .AGENTTRASH/ or name another -o" % a.out)
if a.chunk_size <= 0:
refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
layers, rotated = load_dir(a.calib_dir, a.verbose)
entries, census = build_entries(layers, a.min_rows, a.verbose)
if not entries:
refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
if rotated:
sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
" diagonal is the rotated activation's second moment,"
" not the model's own\n" % rotated)
chunk_count = census["rows"] // a.chunk_size
if chunk_count > 0xFFFFFFFF:
# imatrix.chunk_count is a u32 in the reader's grammar, so there is no
# legal way to record a bigger one. Reaching this means either a chunk
# size picked for a far larger run or a row total read out of the wrong
# header offset — rows is a u64 and a u32 parse is off by 2**32 — and
# the reader would trust whatever number landed in the field.
refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
" imatrix.chunk_count field; raise --chunk-size"
% (chunk_count, census["rows"], a.chunk_size))
nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
a.chunk_size, chunk_count)
sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
" rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
% (len(entries), census["layers"], census["experts"],
census["inherited"], census["rows"], census["routing"],
chunk_count, a.chunk_size, nbytes, a.out))
if __name__ == "__main__":
main()
+391
View File
@@ -0,0 +1,391 @@
#!/usr/bin/env bash
# Metadata-provider keys for Jellyfin on Dankserver, placed from the service's
# host env file with the plugin artifact pinned by content hash.
#
# Jellyfin 12 ships TMDb, OMDb, StudioImages and MusicBrainz as CORE plugins:
# they have no DLL, only a configuration file under the data root, and that
# file is created by the running container. So the fleet's provider state lives
# in two places the repo does not track —
#
# plugins/configurations/Jellyfin.Plugin.Tmdb.xml <TmdbApiKey /> (empty = keyless)
# plugins/configurations/Jellyfin.Plugin.Tvdb.xml <ApiKey /> (needs the plugin)
#
# and both are owned by the container user. This tool is the only thing that
# writes them: it is declarative, idempotent, and it never prints a key, so it
# is safe to run while someone watches.
#
# Keys never enter git and never enter argv (a value in argv is world-readable
# through /proc/<pid>/cmdline). They live in the per-service host env file that
# is the fleet's convention for container credentials (grafana, caddy, pihole,
# copyparty, romm, seerr), mode 600:
#
# /home/dank/Containers/jellyfin/provider-keys
# TMDB_API_KEY=<key>
# TVDB_API_KEY=<key>
# TVDB_PIN=<pin> # optional: only a subscriber-shaped TVDB key
# # needs one; a licensed key does not
#
# TVDB itself is not core in 12 (the metadata provider was cut in 10.9), so it
# is installed from the official catalog, pinned three ways: sha256 of the
# artifact, the plugin's own guid, and its version. A matching sha proves only
# that the bytes are the bytes fetched — the guid check is what says it is the
# official plugin, the version check is what says it is the one this file was
# reviewed against.
#
# Subcommands
# plan print the resolved shape and every pending change, touch nothing
# install fetch+verify the plugin, then write the keys (backing up first)
# verify prove the plugin and both keys are actually in place
#
# Nothing here restarts Jellyfin: the deployer owns that
# (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
# start-up, so a key written while the server runs is not live until then.
set -euo pipefail
# Jellyfin's data root on Dankserver (the rootless quadlet's config volume).
JF_ROOT="${JF_ROOT:-/home/dank/Containers/jellyfin/config}"
PROV_KEYS="${PROV_KEYS:-/home/dank/Containers/jellyfin/provider-keys}"
TRASH_DIR="${TRASH_DIR:-$HOME/.AGENTTRASH/$(date +%Y%m%d)-jellyfin-providers}"
PLUGINS="$JF_ROOT/plugins"
CONF="$PLUGINS/configurations"
TMDB_XML="$CONF/Jellyfin.Plugin.Tmdb.xml"
TVDB_XML="$CONF/Jellyfin.Plugin.Tvdb.xml"
# repo.jellyfin.org catalog entry for TheTVDB (owner: jellyfin), version 24.0.0.0,
# targetAbi 12.0.0.0 — published 2026-09-09, verified against Jellyfin 12.4.0.
TVDB_PLUGIN_URL="${TVDB_PLUGIN_URL:-https://repo.jellyfin.org/files/plugin/thetvdb/thetvdb_24.0.0.0.zip}"
TVDB_PLUGIN_SHA256="${TVDB_PLUGIN_SHA256:-6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319}"
TVDB_PLUGIN_GUID="${TVDB_PLUGIN_GUID:-a677c0da-fac5-4cde-941a-7134223f14c8}"
TVDB_PLUGIN_VERSION="${TVDB_PLUGIN_VERSION:-24.0.0.0}"
PLUGIN_DIR="thetvdb_${TVDB_PLUGIN_VERSION}"
TVDB_DLL="Jellyfin.Plugin.Tvdb.dll"
die() {
printf 'jellyfin-providers: %s\n' "$*" >&2
exit 1
}
usage() {
printf 'usage: %s plan|install|verify\n' "$0" >&2
exit 2
}
# check_keys — refuse anything that is not a readable private file. An open
# mode is an error, not a warning: a provider key readable by every user on the
# box is the failure this check exists to catch.
check_keys() {
[ -f "$PROV_KEYS" ] ||
die "$PROV_KEYS not found (create it with TMDB_API_KEY= and TVDB_API_KEY=, then chmod 600)"
local mode
mode="$(stat -c '%a' "$PROV_KEYS")"
case "$mode" in
600 | 400) ;;
*) die "$PROV_KEYS is mode $mode; provider keys must be 600" ;;
esac
}
# keys_value <VAR> — the value of one variable in the env file, or nothing.
keys_value() {
awk -v k="$1" -F= '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$PROV_KEYS"
}
# keys_len <VAR> — the length, and a refusal naming the variable when absent.
# The value itself is never printed.
keys_len() {
local v
v="$(keys_value "$1")"
[ -n "$v" ] || die "$PROV_KEYS has no $1= line"
printf '%s' "${#v}"
}
# xml_value <file> <element> — the element's text, empty when it is absent or
# self-closing.
xml_value() {
python3 - "$1" "$2" <<'PYEOF'
import re, sys
src = open(sys.argv[1], encoding="utf-8").read()
m = re.search(r"<%s(?:\s*/>|>(.*?)</%s>)" % (sys.argv[2], sys.argv[2]), src, re.S)
print(m.group(1).strip() if (m and m.group(1)) else "")
PYEOF
}
# xml_secret <file> <element> <var> — set <element> to that variable from the
# env file. The python process reads the env file itself, so the value never
# appears in argv.
xml_secret() {
python3 - "$1" "$2" "$3" "$PROV_KEYS" <<'PYEOF'
import os, re, sys, tempfile
from xml.sax.saxutils import escape
xml_path, elem, var, keys_path = sys.argv[1:5]
val = ""
for line in open(keys_path, encoding="utf-8"):
key, sep, rest = line.partition("=")
if sep and key.strip() == var:
val = rest.strip()
break
if not val:
raise SystemExit(f"{keys_path} has no {var}= line")
src = open(xml_path, encoding="utf-8").read()
pat = re.compile(r"([ \t]*)<%s(?:\s*/>|>.*?</%s>)" % (elem, elem), re.S)
text = escape(val)
if pat.search(src):
out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
else:
out = src.replace("</Configuration>", f" <{elem}>{text}</{elem}>\n</Configuration>", 1)
if out == src:
raise SystemExit(f"{xml_path}: no place to write <{elem}>")
d, base = os.path.split(xml_path)
fd, tmp = tempfile.mkstemp(prefix=base + ".", dir=d)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(out)
os.chmod(tmp, os.stat(xml_path).st_mode & 0o777)
os.replace(tmp, xml_path)
PYEOF
}
# plugin_meta <zip> — "guid version" out of the artifact's own meta.json.
plugin_meta() {
python3 - "$1" <<'PYEOF'
import json, sys, zipfile
meta = json.loads(zipfile.ZipFile(sys.argv[1]).read("meta.json").decode("utf-8"))
print(meta.get("guid", ""), meta.get("version", ""))
PYEOF
}
check_root() {
# Never fabricate a data root: creating plugins/ beside a typo'd path would
# install a plugin Jellyfin never loads and hide the operator's mistake.
[ -d "$CONF" ] || die "no Jellyfin configuration directory at $CONF (is JF_ROOT right?)"
}
check_keys_report() {
check_keys
# A refusal inside a command substitution used as a printf argument would
# be swallowed, so the guard runs as its own statement.
keys_len TMDB_API_KEY >/dev/null
keys_len TVDB_API_KEY >/dev/null
printf '= keys read from %s (mode %s)\n' "$PROV_KEYS" "$(stat -c '%a' "$PROV_KEYS")"
}
# The plugin's state, as one line in the shared vocabulary.
plugin_state() {
if [ -d "$PLUGINS/$PLUGIN_DIR" ]; then
printf '= %s already installed\n' "$PLUGIN_DIR"
return 0
fi
local other
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
if [ -n "$other" ]; then
printf '+ install %s (a different version is present: %s)\n' "$PLUGIN_DIR" "$other"
return 0
fi
printf '+ install %s from %s\n' "$PLUGIN_DIR" "${TVDB_PLUGIN_URL#*://}"
}
tmdb_state() {
if [ ! -f "$TMDB_XML" ]; then
printf '! %s is missing — save the TMDb plugin settings once in the UI, then re-run\n' "$TMDB_XML" >&2
return 1
fi
local now want
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
want="$(keys_value TMDB_API_KEY)"
if [ -n "$now" ] && [ "$now" = "$want" ]; then
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
elif [ -n "$now" ]; then
printf '+ replace the TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
else
printf '+ set TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
fi
}
tvdb_state() {
local now
if [ -f "$TVDB_XML" ]; then
now="$(xml_value "$TVDB_XML" ApiKey)"
if [ -n "$now" ]; then
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
else
printf '+ fill the empty ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
fi
else
printf '+ create Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
fi
}
write_plugin() {
local tmp guid version other
other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
[ -z "$other" ] ||
die "another TheTVDB version is already installed ($other); move it to the trash first — two copies of one plugin means two loads"
tmp="$(mktemp "${TMPDIR:-/tmp}/thetvdb.XXXXXX.zip")"
# The pins are checked before anything is unpacked, so a bad artifact never
# reaches the plugins directory.
curl -fsSL --max-time 300 -o "$tmp" "$TVDB_PLUGIN_URL" ||
{ rm -f "$tmp"; die "could not fetch $TVDB_PLUGIN_URL"; }
local got
got="$(sha256sum "$tmp" | cut -d' ' -f1)"
if [ "$got" != "$TVDB_PLUGIN_SHA256" ]; then
rm -f "$tmp"
die "artifact sha256 is $got, pinned $TVDB_PLUGIN_SHA256 — refusing to install"
fi
read -r guid version <<<"$(plugin_meta "$tmp")"
if [ "$guid" != "$TVDB_PLUGIN_GUID" ]; then
rm -f "$tmp"
die "artifact guid is $guid, expected the official $TVDB_PLUGIN_GUID — refusing to install"
fi
if [ "$version" != "$TVDB_PLUGIN_VERSION" ]; then
rm -f "$tmp"
die "artifact version is $version, pinned $TVDB_PLUGIN_VERSION — refusing to install"
fi
mkdir -p "$PLUGINS/$PLUGIN_DIR"
python3 -m zipfile -e "$tmp" "$PLUGINS/$PLUGIN_DIR/"
rm -f "$tmp"
[ -f "$PLUGINS/$PLUGIN_DIR/$TVDB_DLL" ] ||
die "$PLUGIN_DIR has no $TVDB_DLL — the artifact does not match this plugin"
printf '+ installed %s\n' "$PLUGIN_DIR"
}
backup_to_trash() {
local f="$1"
mkdir -p "$TRASH_DIR"
local dest="$TRASH_DIR/$(basename "$f").$(date +%H%M%S)"
cp -p "$f" "$dest"
printf '+ previous copy kept at %s\n' "$dest"
}
cmd_write_keys() {
local now want
KEYS_CHANGED=0
# TMDb: the running container owns this file and every setting in it. Only
# the key element is touched, so country, language and adult settings
# survive the edit.
[ -f "$TMDB_XML" ] ||
die "$TMDB_XML is missing — save the TMDb plugin settings once in the UI, then re-run"
now="$(xml_value "$TMDB_XML" TmdbApiKey)"
keys_len TMDB_API_KEY >/dev/null
want="$(keys_value TMDB_API_KEY)"
if [ "$now" = "$want" ]; then
printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
else
backup_to_trash "$TMDB_XML"
xml_secret "$TMDB_XML" TmdbApiKey TMDB_API_KEY
printf '+ set TmdbApiKey (%s chars)\n' "${#want}"
KEYS_CHANGED=1
fi
# TVDB: never overwrite a config the operator saved through the UI.
if [ -f "$TVDB_XML" ]; then
now="$(xml_value "$TVDB_XML" ApiKey)"
if [ -n "$now" ]; then
printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
else
keys_len TVDB_API_KEY >/dev/null
backup_to_trash "$TVDB_XML"
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
if [ -n "$(keys_value TVDB_PIN)" ] && [ -z "$(xml_value "$TVDB_XML" Pin)" ]; then
xml_secret "$TVDB_XML" Pin TVDB_PIN
fi
printf '+ filled the ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
KEYS_CHANGED=1
fi
else
keys_len TVDB_API_KEY >/dev/null
cat >"$TVDB_XML" <<XML
<?xml version="1.0" encoding="utf-8"?>
<Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ApiKey />
</Configuration>
XML
xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
if [ -n "$(keys_value TVDB_PIN)" ]; then
xml_secret "$TVDB_XML" Pin TVDB_PIN
fi
printf '+ created Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
KEYS_CHANGED=1
fi
}
cmd_plan() {
check_root
check_keys_report
printf 'root %s\n' "$JF_ROOT"
plugin_state
tmdb_state
tvdb_state
}
cmd_install() {
check_root
check_keys_report
KEYS_CHANGED=0
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
write_plugin
KEYS_CHANGED=1
else
printf '= %s already installed\n' "$PLUGIN_DIR"
fi
cmd_write_keys
if [ "$KEYS_CHANGED" -eq 1 ]; then
# Nothing is restarted here on purpose: this host may be playing media
# right now, and both the plugin and its keys are read at start-up.
printf 'next: scripts/deploy.sh dankserver --service jellyfin (the plugin and keys load at start-up)\n'
else
printf '= no change\n'
fi
}
cmd_verify() {
check_root
local bad=0
if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
printf 'MISSING plugin directory %s (thetvdb not installed)\n' "$PLUGIN_DIR" >&2
bad=1
else
local dlls abi
dlls="$(find "$PLUGINS/$PLUGIN_DIR" -maxdepth 1 -name '*.dll' | wc -l)"
[ "$dlls" -ge 1 ] || {
printf 'MISSING %s inside %s\n' "$TVDB_DLL" "$PLUGIN_DIR" >&2
bad=1
}
abi="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("targetAbi","?"))' \
"$PLUGINS/$PLUGIN_DIR/meta.json" 2>/dev/null || echo '?')"
printf 'plugin %s: %s dll(s), targetAbi %s\n' "$PLUGIN_DIR" "$dlls" "$abi"
fi
local pair elem n
for pair in "$TMDB_XML|TmdbApiKey" "$TVDB_XML|ApiKey"; do
elem="${pair##*|}"
if [ ! -f "${pair%%|*}" ]; then
printf 'MISSING %s\n' "${pair%%|*}" >&2
bad=1
continue
fi
if ! python3 -c 'import sys, xml.dom.minidom; xml.dom.minidom.parse(sys.argv[1])' "${pair%%|*}"; then
printf 'UNPARSEABLE %s\n' "${pair%%|*}" >&2
bad=1
continue
fi
n="$(xml_value "${pair%%|*}" "$elem")"
if [ -z "$n" ]; then
printf 'MISSING key: %s has no %s value\n' "$(basename "${pair%%|*}")" "$elem" >&2
bad=1
else
printf '%s %s chars\n' "$elem" "${#n}"
fi
done
[ "$bad" -eq 0 ] || return 1
printf 'providers configured\n'
}
cmd="${1:-}"
case "$cmd" in
plan) cmd_plan ;;
install) cmd_install ;;
verify) cmd_verify ;;
*) usage ;;
esac
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env bash
# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
# recording.
#
# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
# kldref, in the same drained window that records the calibration grams). Every scored
# container is measured against that one recording, so the KL scale runs from 0 = the
# checkpoint itself downward in faithfulness.
#
# TWO MODES, because the disk decides how many anchors can exist at once:
#
# gate IQ4R.json CAND.json candidate vs the served payload
# gate FP8.json IQ4R.json CAND.json the same, inside the absolute interval
#
# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
# release, not half of an invented distance to the checkpoint. Without it the gate still
# answers the campaign's question — does the candidate beat what is served — and says so.
#
# The law, on the summary of one slice (--slice all | a corpus source):
#
# candidate vs served (2 reports)
# 1. mean KL(cand) < mean KL(iq4r) the served payload is beaten on the corpus
# 2. p99.9(cand) <= p99.9(iq4r) and the tail is no worse — a mean can hide a few
# positions taking the whole hit
# 3. top-1(cand) >= top-1(iq4r) the agreement the harness keys on does not fall
# 4. every source slice passes 1-3 a win bought by regressing one distribution
# (code, prose, tools) is not a win
#
# inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
# recovery fraction, as agreed 2026-10-27) plus:
# 5. p99.9(cand) <= p99.9(iq4r) the tail law
# 6. every source slice passes the slice law
#
# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
# same step size, --deterministic, no speculation) that makes the numbers comparable.
# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
#
# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
# kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
set -uo pipefail
if [ "$#" -lt 2 ]; then
echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
echo " kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
exit 2
fi
python3 - "$@" <<'PY'
import json, sys
a = sys.argv[1:]
opts, files = [], []
i = 0
while i < len(a):
if a[i].startswith("--"):
opts += [a[i], a[i + 1]]
i += 2
else:
files.append(a[i])
i += 1
if len(files) not in (2, 3):
sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
" kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
o = dict(zip(opts[0::2], opts[1::2]))
slice_ = o.get("--slice", "all")
recover = float(o.get("--recover", 0.5))
if not 0.0 < recover <= 1.0:
sys.exit(f"--recover must be in (0,1], got {recover}")
interval = len(files) == 3
def die(msg):
print(msg, file=sys.stderr)
raise SystemExit(2)
def load(path):
try:
return json.load(open(path))
except Exception as e:
die(f"cannot read the KLD report {path}: {e}")
def summary(path, d):
"""The slice this gate grades: 'all', or one corpus source."""
if slice_ == "all":
g = d.get("all")
else:
g = (d.get("by_source") or {}).get(slice_)
if g is None:
die(f"{path} has no slice {slice_!r}; it has: "
f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
if g is None or "kld" not in g or "top1_agreement" not in g:
die(f"{path} has no {slice_!r} summary — wrong report version?")
return g
docs = {label: load(path) for label, path in
zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
rows = []
for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
g = summary(path, docs[label])
rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
print(f"reference: BF16 container; slice: {slice_}; "
+ ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8} positions")
for label, path, model, kl, p999, top1, n in rows:
print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}% {n}")
r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
kl_cand, p_cand, t_cand = r["cand"]
kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
ok = True
def law(n, text, passed):
global ok
ok = ok and passed
print(f" [{'PASS' if passed else 'FAIL'}] {n}. {text}")
def tail_law(n):
law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
p_cand <= p_iq4r)
# --- the slice law: every corpus source must pass the metric laws -------------
def slice_law(n):
"""A win that regresses one source class is not a win: check them all."""
mine = docs["cand"].get("by_source") or {}
theirs = docs["iq4r"].get("by_source") or {}
if not mine or not theirs:
law(n, "no per-source summaries to check", True)
return
missing = sorted(set(theirs) - set(mine))
extra = sorted(set(mine) - set(theirs))
if missing or extra:
law(n, "same source slices in both reports"
+ (f" (missing from cand: {', '.join(missing)})" if missing else "")
+ (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
return
bad = []
for src in sorted(theirs):
c, q = mine[src], theirs[src]
worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
worse_top1 = c["top1_agreement"] < q["top1_agreement"]
if worse_mean or worse_tail or worse_top1:
why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
("top-1", worse_top1)) if w)
bad.append(f"{src} ({why})")
law(n, "every source slice passes (mean, p99.9, top-1)"
+ (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
if interval:
kl_fp8 = r["fp8"][0]
gap = kl_iq4r - kl_fp8
law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
gap > 0)
law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
kl_fp8 < kl_cand < kl_iq4r)
need = kl_iq4r - recover * gap
law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
if gap > 0 else "gap is not positive; cannot score recovery",
gap > 0 and kl_cand <= need)
law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
tail_law(5)
slice_law(6)
else:
law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
kl_cand < kl_iq4r)
tail_law(2)
law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
t_cand >= t_iq4r)
slice_law(4)
print("KLD GATE " + ("PASS" if ok else "FAIL"))
sys.exit(0 if ok else 1)
PY
+175 -112
View File
@@ -34,14 +34,25 @@ import json
import os
import random
import re
import sys
from jinja2 import Environment, FileSystemLoader
from tokenizers import Tokenizer
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import qwen38_common as qc # noqa: E402 - sibling module, same directory
CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
".yaml", ".yml", ".bats", ".service", ".container", ".toml")
JSONL_EXTS = (".jsonl", ".json")
DOC_DIR = "serverAppConfigs-docs"
FRAGMENT_TOKENS = 2400 # what one fragment may contribute
CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
POOL_CYCLES = 40 # how many times a class pool may repeat before giving up
KLD_GUARD = 5000 # rendered conversations tried before declaring the pool short
FLEET_CMDS = [
"systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
return out[:cap]
def main() -> None:
def parse_args() -> argparse.Namespace:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo", required=True)
ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
ap.add_argument("--hf-code-parquet", default="")
ap.add_argument("--hf-prose-parquet", default="")
ap.add_argument("--seed", type=int, default=20261006)
a = ap.parse_args()
return ap.parse_args()
rng = random.Random(a.seed)
tokenizer = Tokenizer.from_file(a.tokenizer_json)
env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
def load_template(template: str) -> tuple[Environment, str]:
"""The chat template as the engine will render it, plus its short name."""
env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
trim_blocks=True, lstrip_blocks=True)
env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
ValueError(m))
tname = os.path.basename(a.template)
return env, os.path.basename(template)
def classify(repo: str) -> dict[str, list[str]]:
"""Walk the checkout into the three source classes the mix draws from."""
code, prose, jsonl = [], [], []
for root, dirs, files in os.walk(a.repo):
for root, dirs, files in os.walk(repo):
dirs[:] = [d for d in dirs if not d.startswith(".")]
for f in sorted(files):
p = os.path.relpath(os.path.join(root, f), a.repo)
p = os.path.relpath(os.path.join(root, f), repo)
if DOC_DIR in p and f.endswith(".md"):
prose.append(p)
elif p.startswith(DOC_DIR):
@@ -180,137 +195,185 @@ def main() -> None:
code.append(p)
elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
jsonl.append(p)
# file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
pools: dict[str, tuple[list, list]] = {}
for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
[p for p in lst if fnv(p) % 5 != 0])
return {"code": code, "prose": prose, "tools": jsonl}
def text_of(p: str) -> str:
path = os.path.join(a.repo, p)
raw = open(path, encoding="utf-8", errors="replace").read()
if p.endswith(JSONL_EXTS):
lines = [json.loads(l)["content"] for l in raw.splitlines()
if l.strip() and "content" in l]
raw = "\n".join(lines)
toks = tokenizer.encode(raw, add_special_tokens=False).ids
return tokenizer.decode(toks[:2400], skip_special_tokens=True)
def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
"""File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
return {cls: ([p for p in lst if fnv(p) % 5 == 0],
[p for p in lst if fnv(p) % 5 != 0])
for cls, lst in classes.items()}
def jsonl_content(raw: str) -> str:
"""The content fields of a content-keyed JSONL, or "" for anything else.
A .json in this repo is not necessarily a content-keyed JSONL: the substring
test passes on any line that merely mentions "content", and a list-shaped
document raises TypeError — both used to crash the whole build. Skip what
does not fit instead.
"""
lines = []
for line in raw.splitlines():
if not line.strip() or "content" not in line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(rec, dict) and isinstance(rec.get("content"), str):
lines.append(rec["content"])
return "\n".join(lines)
def fragment_text(repo: str, rel: str, tok) -> str:
"""One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
raw = open(os.path.join(repo, rel), encoding="utf-8",
errors="replace").read()
if rel.endswith(JSONL_EXTS):
raw = jsonl_content(raw)
ids = tok.encode(raw, add_special_tokens=False).ids
return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
def build_frags(a: argparse.Namespace, tok,
rng: random.Random) -> dict[str, dict[str, list[str]]]:
"""Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
pools = split_pools(classify(a.repo))
cmds = harvest_cmds(a.repo, rng)
# fragment pools per class, corpus-split first
frags: dict[str, dict[str, list[str]]] = {}
for cls in pools:
kld_files, cal_files = pools[cls]
fr = {}
src = {"kld": kld_files, "cal": cal_files}
for cls, (kld_files, cal_files) in pools.items():
if cls == "tools":
fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
"cal": [tool_call(rng, cmds) for _ in range(40000)]}
else:
fr["kld"] = [text_of(p) for p in src["kld"]]
fr["cal"] = [text_of(p) for p in src["cal"]]
frags[cls] = fr
frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
"cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
# per-class fragment caps: prose rows are short (~66 tok) so need many;
# codeparrot files are long (~2.6k tok) so a small cap suffices
hfcap = {"code": 40000, "prose": 60000, "tools": 0}
for cls, spec in (("code", a.hf_code_parquet),
("prose", a.hf_prose_parquet)):
for path in [p for p in spec.split(",") if p]:
frags[cls]["cal"].extend(
load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
for cls in frags:
rng.shuffle(frags[cls]["cal"])
return frags
def toks(s: str) -> int:
return len(tokenizer.encode(s, add_special_tokens=False).ids)
def emit_calib() -> tuple[int, list[dict]]:
targets = {"code": .45, "prose": .30, "tools": .25}
rows: list[dict] = []
for cls, frac in targets.items():
want = int(a.calib_tokens * frac)
pool = frags[cls]["cal"]
def pack_class(cls: str, pool: list[str], want: int, tok,
chunk_tokens: int) -> list[dict]:
"""Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
if not pool:
return []
rows: list[dict] = []
# Count every fragment once: the loop cycles the pool up to POOL_CYCLES
# times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
# build's wall time.
counts = [qc.n_tokens(tok, f) for f in pool]
got = 0
i = 0
buf: list[str] = []
nbuf = 0
while got < want:
j = i % len(pool)
frag = pool[j]
i += 1
n = counts[j]
if nbuf + n > chunk_tokens and buf:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
buf, nbuf = [], 0
if got >= want:
break
buf.append(frag)
nbuf += n
if i > len(pool) * POOL_CYCLES:
break # pool exhausted with repeats; stop
if buf and got < want:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
print(f"calib {cls}: {got} tokens, {i} fragments")
return rows
def emit_calib(a: argparse.Namespace, frags, tok,
rng: random.Random) -> tuple[int, list[dict]]:
"""calib.jsonl rows: the class mix, shuffled, prompts only."""
rows: list[dict] = []
for cls, frac in CLASS_MIX.items():
rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
tok, a.chunk_tokens))
rng.shuffle(rows)
return len(rows), [{"prompt": r["prompt"]} for r in rows]
def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
env: Environment, tname: str) -> tuple[int, list[dict], int]:
"""kld.jsonl rows: multi-turn conversations, each with a scored tail."""
template = env.get_template(tname)
rows: list[dict] = []
total = scored_total = 0
classes = ["code", "prose", "tools"]
cursor = {c: 0 for c in classes}
guard = 0
while total < a.kld_tokens and guard < KLD_GUARD:
guard += 1
msgs, srcs = [], []
for t in range(rng.choice([2, 3, 4])):
cls = classes[(len(rows) + t) % 3]
pool = frags[cls]["kld"]
if not pool:
continue
got, i, buf, nbuf = 0, 0, [], 0
while got < want:
frag = pool[i % len(pool)]
i += 1
n = toks(frag)
if nbuf + n > a.chunk_tokens and buf:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
buf, nbuf = [], 0
if got >= want:
break
buf.append(frag)
nbuf += n
if i > len(pool) * 40:
break # pool exhausted with repeats; stop
if buf and got < want:
rows.append({"prompt": "\n".join(buf), "cls": cls})
got += nbuf
print(f"calib {cls}: {got} tokens, {i} fragments")
rng.shuffle(rows)
return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
for r in rows]
frag = pool[cursor[cls] % len(pool)]
cursor[cls] += 1
msgs.append({"role": "user",
"content": "Summarize and explain this "
f"engineering material:\n{frag}"})
msgs.append({"role": "assistant",
"content": frag[:4000]})
srcs.append(cls)
rendered = template.render(messages=msgs, add_generation_prompt=False)
n = qc.n_tokens(tok, rendered)
if n < a.kld_min_prompt:
continue
rows.append({"prompt": rendered,
"score_from": n - a.score_tail,
"source": "+".join(sorted(set(srcs)))})
total += n
scored_total += a.score_tail
return len(rows), rows, scored_total
def emit_kld() -> tuple[int, list[dict]]:
rows, total = [], 0
classes = ["code", "prose", "tools"]
i = {c: 0 for c in classes}
guard = 0
while total < a.kld_tokens and guard < 5000:
guard += 1
msgs, srcs = [], []
for t in range(rng.choice([2, 3, 4])):
cls = classes[(len(rows) + t) % 3]
pool = frags[cls]["kld"]
if not pool:
continue
frag = pool[i[cls] % len(pool)]
i[cls] += 2
msgs.append({"role": "user",
"content": "Summarize and explain this "
f"engineering material:\n{frag}"})
msgs.append({"role": "assistant",
"content": frag[:4000]})
srcs.append(cls)
rendered = env.get_template(tname).render(
messages=msgs, add_generation_prompt=False)
n = toks(rendered)
if n < a.kld_min_prompt:
continue
rows.append({"prompt": rendered,
"score_from": n - a.score_tail,
"source": "+".join(sorted(set(srcs)))})
total += n
return len(rows), rows
os.makedirs(a.out_dir, exist_ok=True)
nc, rows = emit_calib()
with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
nk, krows = emit_kld()
with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
for r in krows:
f.write(json.dumps(r) + "\n")
scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
- r["score_from"] for r in krows)
manifest = {
def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
scored: int) -> dict:
return {
"seed": a.seed, "template": tname,
"tokenizer": os.path.basename(a.tokenizer_json),
"calib_prompts": nc, "kld_prompts": nk,
"kld_scored_positions": scored,
"kld_score_tail": a.score_tail,
"disjoint_file_pools": "local KLD material only; HF code feeds calib",
"calib_mix": {"code": .45, "prose": .30, "tools": .25},
"calib_mix": dict(CLASS_MIX),
"hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
}
with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
json.dump(manifest, f, indent=1)
print(json.dumps(manifest, indent=1))
def main() -> None:
a = parse_args()
rng = random.Random(a.seed)
tok = qc.load_tokenizer(a.tokenizer_json)
env, tname = load_template(a.template)
frags = build_frags(a, tok, rng)
os.makedirs(a.out_dir, exist_ok=True)
nc, calib_rows = emit_calib(a, frags, tok, rng)
qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
man = manifest(a, tname, nc, nk, scored)
qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
print(json.dumps(man, indent=1))
if __name__ == "__main__":
+454
View File
@@ -0,0 +1,454 @@
#!/usr/bin/env python3
"""media-apply.py — the only thing that moves movie-library files.
Reads the plan written by scripts/media-plan.py (rows: `action src dst kind`,
plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
an interrupted run has destroyed nothing. Every run is resumable and every run
is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
BEFORE the rename it authorises. That ordering is the whole design: a crash
leaves either an intent whose move happened or an intent whose move never ran,
and both are recognised on the next run. A file that moved with no journal line
cannot have been moved by this tool, and saying so is better than guessing.
Four subcommands:
apply --plan PLAN [--root ROOT] [--journal J] [--apply]
no --apply => report only and write NOTHING, not even folders
rollback --journal J [--root ROOT] [--apply]
back to the paths the census saw; folders this run created are
removed only when empty, never anything else
verify --plan PLAN --journal J [--root ROOT]
every planned file is at its target with the journalled byte count
sweep --journal J [--root ROOT] [--apply]
removes the folders THIS RUN emptied — the journal has to show a
file that moved out of one, so a folder someone made by hand can
never be removed by this tool, and rollback puts them back
Refusals (exit 2, before the first rename, with nothing touched): a target
already occupied by a foreign file, a source that has gone, a duplicate target,
a path outside --root, a journal that was already rolled back, a move that would
cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
--force relaxes only the guards that are opinion (journal already rolled back,
paths outside --root); it never allows overwriting a file.
--metadata-view names the overlay the media servers read their library through
(default /var/lib/media-meta/view). The sidecar follow-up printed after an apply
or rollback is conditioned on that path being a mountpoint: where no view is
mounted the servers bind the array itself, the sidecars moved with the media,
and a reap order would point at a directory that is not there.
The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
inside it: a log inside the library is a file Jellyfin will index.
"""
import argparse
import datetime
import json
import os
import sys
DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
class Refuse(Exception):
"""A gate the operator has to see; every refusal leaves the library alone."""
def journal_default(root):
return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
class Journal:
"""Append-only JSONL of intents. Writes are fsynced before the tool acts."""
def __init__(self, path, writing):
self.path, self.writing = path, writing
self.records = []
if os.path.exists(path):
for line in open(path):
line = line.strip()
if line:
self.records.append(json.loads(line))
self._fh = None
def already(self, dst, src):
"""An intent for this exact (src, dst) was written by an earlier run."""
return any(r.get("op") == "move" and r.get("dst") == dst
and r.get("src") == src for r in self.records)
@property
def rolled_back(self):
return any(r.get("op") == "rollback" for r in self.records)
def record(self, **fields):
self.records.append(fields)
if not self.writing:
return
if self._fh is None:
self._fh = open(self.path, "a")
self._fh.write(json.dumps(fields) + "\n")
self._fh.flush()
os.fsync(self._fh.fileno())
def close(self):
if self._fh:
self._fh.close()
def read_plan(path, root, force):
"""Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
mkdirs, keeps, moves = [], [], []
with open(path) as fh:
header = fh.readline()
if not header.startswith("action\t"):
raise Refuse(f"{path} is not a plan (first column is "
f"{header.split(chr(9))[0]!r}, expected 'action') —"
" regenerate it with scripts/media-plan.py")
for line in fh:
f = line.rstrip("\n").split("\t")
if f[0] == "mkdir":
mkdirs.append(f[2])
elif f[0] == "keep":
keeps.append(f[1])
elif f[0] == "move":
moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
if not force:
for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
if p and not (p == root or p.startswith(root + os.sep)):
raise Refuse(f"{p} is outside --root {root}"
" (a plan built for another library?)")
return mkdirs, keeps, moves
# The media servers may read the library through the SSD overlay view
# (scripts/media-view.sh). A relayout moves bytes on the raw array, so sidecars
# the servers wrote through that view stay at the old coordinates — naming the
# follow-up here is the point of use for that fact, but only where the view
# exists: where it is not mounted the servers bind the array itself, the
# sidecars moved with the media, and a reap order points at a directory that
# is not there. See the Media Metadata View page for the host's state.
METADATA_VIEW = "/var/lib/media-meta/view"
METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
" these moves — next: scripts/media-view.sh reap")
def metadata_note(args):
"""Print the sidecar follow-up only when the overlay is really mounted."""
if os.path.ismount(args.metadata_view):
print(METADATA_HINT)
def do_apply(args, root, journal, mkdirs, keeps, moves):
"""Preflight everything, then folders, then renames. Never a partial start."""
problems = []
for p in keeps:
if not os.path.exists(p):
problems.append(f"missing: keep row source is gone: {p}")
seen = {}
for src, dst, _kind in moves:
if src == dst:
problems.append(f"no-op: move row whose target is its source: {src}")
if dst in seen:
problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
seen[dst] = src
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue # this intent already ran
if not os.path.exists(src):
if os.path.exists(dst):
problems.append(f"missing: {src} is gone and {dst} carries no journal"
" line — it moved outside this tool, refusing to guess")
else:
problems.append(f"missing: {src} is gone and {dst} is not there either")
continue
if os.path.exists(dst):
problems.append(f"occupied: target already holds another file: {dst}")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved")
for p in sorted(problems)[:20]:
print(" " + p)
if len(problems) > 20:
print(f" ... and {len(problems) - 20} more")
return 2
if not args.apply:
print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
" — nothing written (pass --apply to run)")
for src, dst, _k in moves[:5]:
print(f" would move {src} -> {dst}")
return 0
made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
dirs_new = 0
for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
if d in made or os.path.isdir(d):
continue
journal.record(op="mkdir", dst=d)
os.makedirs(d, exist_ok=True)
dirs_new += 1
# same-filesystem gate: rename cannot cross a device, and a silent copy of a
# library this size is not an option anyone would choose.
cross = []
for src, dst, _k in moves:
if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
continue
parent = os.path.dirname(dst)
if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
cross.append(dst)
if cross:
print(f"refusing: {len(cross)} moves would cross a filesystem"
f" (first: {cross[0]}) — this tool only renames")
return 2
moved = done = 0
for src, dst, _kind in moves:
if journal.already(dst, src):
if os.path.exists(dst) and not os.path.exists(src):
done += 1
continue
if os.path.exists(src) and not os.path.exists(dst):
os.rename(src, dst) # the intent ran, the move did not
moved += 1
continue
st = os.stat(src)
journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
os.rename(src, dst)
moved += 1
print(f"applied: {moved} moved, {done} already done in an earlier run,"
f" {dirs_new} folders created, {len(keeps)} files stayed put")
print(f"journal: {journal.path} — undo with: media-apply.py rollback"
f" --journal {journal.path} --root {root} --apply")
metadata_note(args)
return 0
def do_verify(root, journal, moves, keeps):
"""Every planned file sits at its target carrying the journalled bytes."""
journalled = {(r["src"], r["dst"]): r for r in journal.records
if r.get("op") == "move"}
todo = moves or [(r["src"], r["dst"], "") for r in journal.records
if r.get("op") == "move"]
bad, missing, bytes_total = [], [], 0
for src, dst, _kind in todo:
rec = journalled.get((src, dst))
if not os.path.exists(dst):
missing.append(dst)
continue
size = os.path.getsize(dst)
bytes_total += size
if rec and rec.get("bytes") not in (None, size):
bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
if os.path.exists(src) and src != dst:
bad.append(f"{src}: still in place as well as at {dst}")
missing += [p for p in keeps if not os.path.exists(p)]
if bad or missing:
print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
for line in (missing + bad)[:20]:
print(" " + line)
return 2
empty = [r["dst"] for r in journal.records
if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
and not os.listdir(r["dst"])]
print(f"verified: {len(todo)} files at their targets"
f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
if empty:
print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
return 0
def sweep_targets(root, journal):
"""The folders this run emptied, deepest first — never one it did not touch.
Eligibility is evidence, not appearance. A folder is a target only when the
journal shows a file moved out of it or out of something beneath it, it holds
nothing but empty directories now, it is not a folder this run created, and no
journalled destination lives under it. A symlink anywhere inside disqualifies
the whole subtree: this tool never follows a link, and deleting the folder
that holds one would delete someone's mount point.
Proof carries downward: once a folder qualifies, every empty directory inside
it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
a move source, so without that rule it holds its emptied parent on disk
forever — the parent can never be removed while the child stands.
"""
moves = [r for r in journal.records if r.get("op") == "move"]
made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
# An rmdir INTENT that failed left its folder standing, and the next sweep
# must be able to finish it: only a folder that is really gone is out of
# scope. Counting every intent as done made 146 live survivors permanent.
gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
and not os.path.exists(r["path"])}
dsts = [r["dst"] for r in moves]
marked = set()
for r in moves:
d = os.path.dirname(r["src"])
while d.startswith(root + os.sep):
marked.add(d)
d = os.path.dirname(d)
def removable(d):
if d in made or d in gone or os.path.islink(d):
return False
if any(p == d or p.startswith(d + os.sep) for p in dsts):
return False
try:
entries = os.listdir(d)
except OSError:
return False
for e in entries:
c = os.path.join(d, e)
if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
return False
return True
roots = [d for d in marked if removable(d)]
targets = set(roots)
for d in roots:
for dirpath, dirnames, _ in os.walk(d):
for c in dirnames:
p = os.path.join(dirpath, c)
if removable(p):
targets.add(p)
return sorted(targets, key=lambda q: (-q.count(os.sep), q))
def do_sweep(args, root, journal):
"""Remove the empty husks an apply left standing at the library root."""
moves = [r for r in journal.records if r.get("op") == "move"]
if not moves:
raise Refuse("the journal holds no moves — a sweep follows an apply, it"
" does not guess at emptiness")
outside = sorted({p for r in moves for p in (r["src"], r["dst"])
if not (p == root or p.startswith(root + os.sep))})
if outside and not args.force:
raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
f" (first: {outside[0]}) — this journal belongs to another"
" library")
targets = sweep_targets(root, journal)
if not targets:
print("swept: 0 folders — nothing this run emptied is still standing empty")
return 0
if not args.apply:
print(f"dry run: {len(targets)} folders this run emptied would be removed"
" (pass --apply to run)")
for d in targets[:5]:
print(f" would remove {d}")
return 0
removed = 0
for d in targets:
journal.record(op="rmdir", path=d)
try:
os.rmdir(d)
except OSError as e:
print(f"left in place: {d} ({e})")
continue
removed += 1
print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
print(f"journal: {journal.path} — rollback recreates them with their files")
return 0
def do_rollback(args, root, journal):
"""Undo every journalled move, newest first, after a full preflight."""
moves = [r for r in journal.records if r.get("op") == "move"]
made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
problems = []
for r in reversed(moves):
src, dst = r["src"], r["dst"]
if not args.force and not (src.startswith(root + os.sep)
or dst.startswith(root + os.sep)):
problems.append(f"outside --root: {dst}")
continue
if os.path.exists(dst) and os.path.exists(src):
problems.append(f"occupied: both paths exist, refusing to choose:"
f" {src} | {dst}")
elif os.path.exists(dst):
pass
elif os.path.exists(src):
continue # already undone
else:
problems.append(f"missing: {dst} is gone and {src} never came back")
if problems:
print(f"refusing: {len(problems)} problem(s), nothing moved back")
for p in problems[:20]:
print(" " + p)
return 2
if not args.apply:
print(f"dry run: would move {len(moves)} files back to their census paths"
f" and recreate {len(set(swept))} swept folders (pass --apply)")
return 0
undone = 0
recreated = 0
for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
if not os.path.isdir(d):
os.makedirs(d, exist_ok=True)
recreated += 1
for r in reversed(moves):
if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
os.rename(r["dst"], r["src"])
undone += 1
dirs_gone = 0
for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
if d == root or not os.path.isdir(d) or os.listdir(d):
continue
try:
os.rmdir(d)
dirs_gone += 1
except OSError as e:
print(f"left in place: {d} ({e})")
journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
t=datetime.datetime.now(datetime.timezone.utc).isoformat())
print(f"rolled back: {undone} files back where the census found them,"
f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
metadata_note(args)
return 0
def main():
ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
ap.add_argument("--plan")
ap.add_argument("--root", default=DEFAULT_ROOT)
ap.add_argument("--journal")
ap.add_argument("--apply", action="store_true", help="actually change the library")
ap.add_argument("--force", action="store_true", help="relax the opinion guards")
ap.add_argument("--metadata-view", default=METADATA_VIEW,
help="overlay the media servers read through; the sidecar"
" note prints only when this path is a mountpoint")
args = ap.parse_args()
root = os.path.abspath(args.root)
if not os.path.isdir(root):
raise Refuse(f"no library at {root}")
journal_path = args.journal or journal_default(root)
journal = Journal(journal_path, args.apply)
try:
if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
raise Refuse("this journal has already been rolled back"
" (--force to replay it)")
if args.cmd in ("apply", "verify") and not args.plan:
raise Refuse(f"{args.cmd} needs --plan")
mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
if args.plan else ([], [], []))
if args.cmd == "apply":
return do_apply(args, root, journal, mkdirs, keeps, moves)
if args.cmd == "verify":
return do_verify(root, journal, moves, keeps)
if args.cmd == "sweep":
return do_sweep(args, root, journal)
return do_rollback(args, root, journal)
finally:
journal.close()
if __name__ == "__main__":
try:
sys.exit(main())
except Refuse as r:
print(f"refusing: {r}")
sys.exit(2)
+425
View File
@@ -0,0 +1,425 @@
#!/usr/bin/env python3
r"""media-audiobooks.py — a plan that re-files the audiobook library into one folder per book.
media-audiobooks.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Plex reads this library and Jellyfin does not: its music scanner has no audiobook
concept, and the tree is indexed entirely by Plex, which matched 58,975 of the
62,879 files here into its Artist/Album/Track shape. So the authority is
`scripts/media-census.py audiobooks` — the BOOK (album row), its NARRATOR (that
book's single artist row) and its CHAPTERS (track rows) — and never the folder a
release pack made. The plan is:
Audiobooks/<Narrator>/<Book> (<year>)/<the file's own name>.<ext>
... the same folder, " (2)" on the name, when two chapters
of one book carry the same name
What that buys is the one property both media servers' book scanners assume: a
folder holds exactly one book. Today 212 top-level folders are a mix of
author-named shelves and collection packs, and the books inside the packs are
split across `… [Part 1]`, `… [Part 2]` folders three or four deep.
Three laws here are the opposite of what a music shelf does, and each is forced
by what the rows actually hold:
* A chapter is NEVER renumbered. 7,556 of the 10,485 books contain at least one
chapter whose index Plex does not know, so a number written into a name would
be invented — and it would buy nothing, because both servers order playback by
the embedded track tag, not by the file name. A wrong number in a name is worse
than none: it is the number a human later trusts. A file already carrying its
own number keeps it, untouched.
* The shelf is the NARRATOR, not the author. Every book has exactly one artist row
and it is the narrator; the author survives only inside 10% of the album titles
(`Kryptos by Graham McNeill`) and in no column Plex stores, so deriving the
author would mean parsing titles for one book in ten and inventing it for the
rest. One rule that always resolves beats a shelf that is half author and half
narrator; the author stays visible in the book folder's own name, which is the
title Plex shows.
* Nothing is guessed at. A chapter row with no book or no narrator parks — a
tag-scan placeholder (`[Unknown Album]`) names neither, since unrelated books
share it and filing on one merges them, and so does a chapter number the ripper
mistagged as the artist (`0.4`). An audio file Plex never matched parks, a
video container parks, and a loose file in a folder holding two books parks.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What a chapter file can be. .m4b is the audiobook container; the rest are the
# audio a pack ships when it was ripped as music.
AUDIO_EXT = {".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav", ".flac",
".wma", ".aiff", ".aif", ".m4v", ".aac", ".mka"}
# Never an audiobook, whatever the library it sits in says: Plex reads these as
# video, and renaming one into an audio extension would claim it is a chapter.
VIDEO_EXT = {".mkv", ".mp4", ".webm", ".avi", ".mov", ".mpg", ".mpeg", ".ts",
".m2ts", ".vob", ".ifo", ".wmv", ".iso", ".img", ".bdmv", ".ogm",
".divx"}
# Tag-scan placeholders. These strings are not identities: 1,106 chapters from
# 14 unrelated top-level folders carry the album `[Unknown Album]`, so filing on
# them would merge books that have nothing to do with each other — the exact
# shape this relayout exists to make impossible. Such a chapter parks with the
# path it has today and the fix belongs in its tags. `Various Artists` is NOT a
# placeholder: that is what a multi-cast read carries, and the one book here
# with 1,189 chapters under it is genuinely one book.
PLACEHOLDER_BOOK = {"[unknown album]", "unknown album", "no title", "untitled",
"[untitled]", "unknown title", "n/a", "-"}
PLACEHOLDER_NARRATOR = {"[unknown artist]", "unknown artist", "unknown author",
"<unknown>", "[unknown]", "unknown"}
def identity(value, placeholders, needs_letter=False):
"""The tag value Plex matched, when it names something; '' when it does not.
A trailing parenthesized group is stripped for the test only, because rippers
glue timestamps to the placeholder (`Unknown album (03/03/2015 21:39:55)`).
`needs_letter` is for a NARRATOR alone: a shelf is a person's name, and 272
chapters here carry a chapter number in the artist tag (`0.4`, `31`). The same
test must never judge a BOOK title, which may honestly be `1984`.
"""
v = (value or "").strip()
bare = re.sub(r"\s*\([^()]*\)\s*$", "", v).strip().casefold()
if bare in placeholders or (needs_letter and not any(c.isalpha() for c in v)):
return ""
return v
# The characters Plex documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the chapter row Plex matched
PLACED = {} # chapter path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
AUDIO_BY_DIR = defaultdict(set) # source dir -> chapter files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding chapters
FOLDERS_SEEN = {} # (dir, depth) -> book folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its chapters may sit
BOOKS = {} # (narrator, book, year) -> the book group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the rows mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing unprintable,
no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 60k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ------------------------------------------------------------------------ books
def group_books():
"""Every matched chapter into one group per (narrator, book, year).
The key is the identity Plex asserts in its own database, not the folder the
file sits in: one book arrives as four part folders inside a collection pack,
and two copies of one book in one pack are one book to Plex, whose album item
IS that narrator, title and year. The year is part of the key so a 1965
recording and a 2020 recording that share a title stay two books.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above Part 1/Part 2 sees both
narrator = identity(r["narrator"], PLACEHOLDER_NARRATOR, needs_letter=True)
book = identity(r["book"], PLACEHOLDER_BOOK)
if not book:
park(path, "a chapter row that names no book — Plex's album title is a "
"placeholder like [Unknown Album], so filing on it would "
"merge unrelated books onto one shelf")
continue
if not narrator:
park(path, "a chapter row that names no narrator — the artist row is a "
"placeholder like [Unknown Artist] or a chapter number a "
"ripper mistagged as the artist, so no shelf owns it")
continue
key = (narrator.casefold(), book.casefold(), (r["year"] or "").strip())
g = BOOKS.setdefault(key, {"narrators": [], "books": [], "year": key[2],
"members": [], "indexed": 0})
g["members"].append(path)
g["narrators"].append(narrator)
g["books"].append(book)
if str(r["track"] or "").isdigit():
g["indexed"] += 1
def book_folder(g):
"""The shelf Plex's own rows name: <narrator>/<book> (<year>). None when a
name is made of characters no path can hold — inventing a shelf is filing it
wrong, so those chapters park instead."""
narrator = safe(dominant(g["narrators"]))
book = safe(dominant(g["books"]))
if not narrator or not book:
return None
return os.path.join(ARGS.root, narrator,
f"{book} ({g['year']})" if g["year"] else book)
def place_books():
"""One folder per book; every chapter keeps the name it has today."""
for key in sorted(BOOKS):
g = BOOKS[key]
folder = book_folder(g)
if folder is None:
for path in sorted(g["members"]):
park(path, "a book or narrator Plex spells with characters no "
"folder name can hold")
continue
for path in sorted(g["members"]):
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
# The name the ripper wrote, unchanged: the track tag inside the file
# is what orders playback, and 72% of the books here have chapters no
# row numbers, so any number added here would be invented.
PLACED[path] = assign(path, folder, base, ext, "chapter",
"two chapters of one book carried the same name")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The book folders the chapters in `d`, or within `depth` directories below
it, were placed in. Memoised, because every loose file in a big collection
folder asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The book folder that owns this directory — the first place up the tree
where chapters sit, in the directory itself or within SUBTREE_DEPTH levels
below it, which is how a pack whose parts are `… [Part 1]`/`… [Part 2]` owns
the cover art lying above them. None when no chapter is near enough to say,
AMBIGUOUS when the chapters there belong to more than one book."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_other():
"""Everything that is not a chapter Plex matched: covers, booklets, EPUBs,
logs, checksums, playlists — and the clutter that is none of those."""
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d = os.path.dirname(path)
base = os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the audiobook library: Plex reads "
"it as video, and renaming it would claim it is a chapter")
elif ext in AUDIO_EXT:
park(path, "an audio file no chapter row names — it is not in the "
"library Plex matched, so nothing says which book it is")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a file whose folder holds more than one book, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one book folder would have carried two files of the same"
" sidecar name")
else:
park(path, "not an audiobook file, and no book's folder lies with"
" it to carry it")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Audiobook relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
unindexed = sorted(f"{safe(dominant(g['narrators']))}/{safe(dominant(g['books']))}"
for g in BOOKS.values()
if 0 < g["indexed"] < len(g["members"]))
if unindexed:
fh.write(f"{len(unindexed):>6} books have chapters Plex does not number. Their\n")
fh.write(" files keep the names they carry: playback order comes from the\n"
" track tag inside each file, and a number written here would be\n"
" invented for the chapters with no index. Fix it in the tags, not\n"
" in the names — Plex re-reads tags on a scan and never names files.\n\n")
fh.write(" Playlists move with their book and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"))
ap.add_argument("--enum", default="/tmp/audiobooks-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/plex-audiobooks.tsv",
help="the chapter rows scripts/media-census.py audiobooks wrote")
ap.add_argument("--out", default="/tmp/audiobooks-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/audiobooks-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
group_books()
place_books()
place_other()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a chapter row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+526
View File
@@ -0,0 +1,526 @@
#!/usr/bin/env python3
r"""Read-only census of a media library — the inputs a relayout plan is built from.
media-census.py enumerate [--root DIR] [--out TSV]
media-census.py probe [--enum TSV] [--out TSV] [--jobs N]
media-census.py plex [--root DIR] [--db PATH] [--out TSV]
media-census.py jellyfin [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
media-census.py audiobooks [--root DIR] [--db PATH] [--out TSV]
Run it on the machine that mounts the library. Nothing here writes inside the
library and nothing is deleted; the only outputs are the TSVs below, and the
scripts/media-plan.py and scripts/media-shows.py read them instead of touching media.
enum bytes <TAB> ext <TAB> path every file under the root
probed bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
<TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
one row per video-container file; `audio` is "codec:lang+codec:lang",
`subs` the subtitle-stream count, `spare` the audio streams carrying
neither a default nor a forced tag
plex file <TAB> leaf_title <TAB> … (see PLEX_COLS)
the titles Plex already matched — the authority over any filename guess
jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
<TAB> tmdb <TAB> title <TAB> tvdb (see JF_COLS)
one row per episode Jellyfin has matched and numbered, its path mapped
from the container's view back to the host; `year`, `tmdb` and `tvdb`
are the SERIES' values, `title` is the episode title the row carries,
`extra` is Jellyfin's own extra type when the row is bonus material
rather than an episode
music file <TAB> artist <TAB> album <TAB> year <TAB> disc <TAB> track <TAB> title
<TAB> mb <TAB> performer (see MUSIC_COLS)
one row per track Jellyfin has tagged: the tag read is what the row IS,
so `artist` is the album artist (the track artist when that is empty) and
`album`/`year` are the album item's provider values when the folder is an
album item and the tags' otherwise; `mb` is the album's MusicBrainz
release group, `performer` the track artist the tags carry
audiobooks file <TAB> narrator <TAB> book <TAB> year <TAB> track <TAB> chapter
(see AB_COLS)
one row per chapter Plex matched under the audiobook root: `book` is the
album row Plex built from the files' own tags, `narrator` that book's
artist row — Plex keeps exactly one artist per book, which is why it, and
not an author parsed out of a title, names the shelf. `track` is empty
wherever Plex has no chapter number, which is most of this library.
`probe` is the slow step (every container is opened for its header, ~10 min for
4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
live service must not starve it. `enumerate` is a stat pass, so re-censusing
before a plan is cheap — and a plan is only ever as good as these files.
"""
import argparse
import csv
import json
import os
import re
import sqlite3
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
# a mounted volume, not a stream, and the planner treats it as a movie file instead.
PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
# Trash and thumbnail caches are not library content, and this repo's trash policy
# means .AGENTTRASH can sit inside a movie folder at any time.
SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
"parent_title", "parent_orig", "parent_year", "parent_edition",
"container", "vcodec", "acodec", "width", "height"]
PLEX_SQL = """
select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
coalesce(mi.edition_title,''), coalesce(mi.year,0),
coalesce(par.title,''), coalesce(par.original_title,''),
coalesce(par.year,0), coalesce(par.edition_title,''),
m.container, m.video_codec, m.audio_codec, m.width, m.height
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
left join metadata_items par on par.id = mi.parent_id
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
"""
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
JF_COLS = ["file", "series", "year", "season", "episode", "extra",
"tmdb", "title", "tvdb"]
# Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
"MediaBrowser.Controller.Entities.TV.Season",
"MediaBrowser.Controller.Entities.TV.Series")
JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType,
IsVirtualItem, Name from BaseItems where Type = ? and Path like ?""")
# The music tree is these two entity classes. The track row carries the tags
# Jellyfin read out of the file; the album row carries what the provider matched,
# which is the better folder name of the two.
JF_AUDIO = "MediaBrowser.Controller.Entities.Audio.Audio"
JF_ALBUM = "MediaBrowser.Controller.Entities.Audio.MusicAlbum"
MUSIC_COLS = ["file", "artist", "album", "year", "disc", "track", "title", "mb",
"performer"]
MUSIC_SQL = ("""select b.Path, b.AlbumArtists, b.Artists, b.Album, b.ProductionYear,
b.ParentIndexNumber, b.IndexNumber, b.Name, b.Id,
a.Name, a.ProductionYear, a.Id
from BaseItems b
left join BaseItems a on a.Id = b.ParentId and a.Type = ?
where b.Type = ? and b.Path like ?""")
# The album's identity, in preference order. A release group is the album; a bare
# release id is one reissue of it.
MUSIC_PROVIDERS = ("MusicBrainzReleaseGroup", "MusicBrainzAlbum")
# Plex's audiobook tree is the same Artist/Album/Track shape as its music tree:
# the leaf (10) is a CHAPTER, its parent (9) is the BOOK, and that book's artist
# (8) is the NARRATOR. The track's own `index` is exported because the exceptions
# report counts how many books have chapters Plex does not number — the reason the
# relayout refuses to invent numbers into file names.
PLEX_ARTIST, PLEX_ALBUM, PLEX_TRACK = 8, 9, 10
AB_COLS = ["file", "narrator", "book", "year", "track", "chapter"]
AB_SQL = f"""
select p.file, coalesce(ar.title,''), coalesce(al.title,''), coalesce(al.year,0),
coalesce(mi."index",-1), coalesce(mi.title,'')
from media_parts p
join media_items m on m.id = p.media_item_id
join metadata_items mi on mi.id = m.metadata_item_id
join metadata_items al on al.id = mi.parent_id and al.metadata_type = {PLEX_ALBUM}
left join metadata_items ar on ar.id = al.parent_id
and ar.metadata_type = {PLEX_ARTIST}
where p.file like ? and p.deleted_at is null and mi.deleted_at is null
and al.deleted_at is null
"""
def open_plex_ro(db):
"""Plex's library, read the way Plex writes it.
`immutable=1` tells sqlite the file never changes, so it skips the
write-ahead log: every row a scan has committed but not yet checkpointed
is invisible, and reading a live database that way can fail outright as
"database disk image is malformed". Plain ro reads the WAL through its
-shm; immutable is only the fallback for a database whose -shm cannot be
read. The same law scripts/media-verify.py states.
"""
last = None
for opt in ("mode=ro", "mode=ro&immutable=1"):
try:
return sqlite3.connect(f"file:{db}?{opt}", uri=True)
except sqlite3.Error as e:
last = e
sys.exit(f"cannot open the Plex database read-only ({db}): {last}\n"
"Plex runs rootless here: run this as the container's owner, never "
"with sudo — rootless storage is per-user, so root sees nothing.")
def enumerate_lib(root, out):
"""One row per real file under root: size, extension, path."""
rows = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames
if not SKIP_DIR.search(os.path.join(dirpath, d))]
for fn in filenames:
p = os.path.join(dirpath, fn)
if SKIP_DIR.search(p):
continue
try:
st = os.lstat(p)
except OSError:
continue
if not os.path.isfile(p) or os.path.islink(p):
continue # links and sockets are not library content
rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
with open(out, "w") as fh:
for size, ext, p in rows:
fh.write(f"{size}\t{ext}\t{p}\n")
vid = [r for r in rows if r[1] in PROBE_EXT]
print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
def probe_one(path):
"""ffprobe header read -> the probed TSV row (never the whole file)."""
try:
os.nice(19) # a census must never outrank the media server
except OSError:
pass
try:
size = os.stat(path).st_size
except OSError:
return None
cmd = ["ffprobe", "-v", "error", "-print_format", "json",
"-show_format", "-show_streams", path]
try:
done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
data = json.loads(done.stdout or "{}")
except (subprocess.TimeoutExpired, subprocess.SubprocessError,
json.JSONDecodeError, OSError):
data = {}
st = data.get("streams", [])
fmt = data.get("format", {})
v = next((s for s in st if s.get("codec_type") == "video"), None)
aud = [s for s in st if s.get("codec_type") == "audio"]
subs = [s for s in st if s.get("codec_type") == "subtitle"]
fps = "-"
if v and v.get("r_frame_rate"):
try:
n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
fps = f"{round(n / d * 100) / 100:g}" if d else "0"
except ValueError:
pass
def num(x):
try:
return round(float(x))
except (TypeError, ValueError):
return 0
return [size, path, num(fmt.get("duration")),
(v or {}).get("codec_name", "-"),
f"{v['width']}x{v['height']}" if v else "-",
fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
"+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
len(subs),
sum(1 for a in aud if not re.search(
r"default|forced", json.dumps(a.get("tags", {})), re.I)),
fmt.get("format_name", "-") or "-"]
def probe(enum, out, jobs):
"""Probe every video-container row of the enum TSV, in parallel."""
paths = []
with open(enum) as fh:
for line in fh:
f = line.rstrip("\n").split("\t", 2)
if len(f) == 3 and f[1] in PROBE_EXT:
paths.append(f[2])
print(f"probing {len(paths)} files with {jobs} workers", flush=True)
done = unreadable = 0
with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
for row in pool.map(probe_one, paths):
if row is None:
continue
if row[3] == "-":
unreadable += 1 # header unreadable: the row still gets written
done += 1
fh.write("\t".join(str(x) for x in row) + "\n")
if done % 250 == 0:
print(f" {done}/{len(paths)}", flush=True)
print(f"probed={done} unreadable={unreadable} -> {out}")
def plex(root, db, out):
"""Plex's matched titles for files under root, from its DB, read-only."""
con = open_plex_ro(db)
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(PLEX_COLS)
n = 0
for r in con.execute(PLEX_SQL, (root + "/%",)):
w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
n += 1
print(f"plex-matched={n} -> {out}")
def jellyfin(root, db, container, out):
"""The episodes Jellyfin has already numbered, for the library at root.
Jellyfin stores each path as the CONTAINER sees it (its quadlet binds the
share at /data), so `container` is the in-container prefix that stands for
root and every exported path is mapped back to the host before it is written
— the planner reads host paths, because it plans moves on the array.
The numbering is what matters, not the name: a release group's file name says
nothing about which episode a file is, and Jellyfin has already answered that
question against a provider. Season/episode and the episode's own title come
from the episode row; the series name, its year and its TMDb and Tvdb ids come
from the Series row the episode climbs to, because those are the only place
Jellyfin keeps them.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
"It is a WAL database: copy it with its -wal and -shm beside it and"
" point --db at the copy. Never sudo it — the data dir belongs to"
" the container's owner, and rootless storage is per-user.")
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
need = {"Id", "ParentId", "Type", "Path", "IndexNumber", "ParentIndexNumber",
"ExtraType", "IsVirtualItem", "ProductionYear", "Data"}
if not need <= have:
sys.exit(f"{db} is not a Jellyfin library database (BaseItems has no"
f" {','.join(sorted(need - have))})")
node = {}
for i, pid, typ, name, year, data in con.execute(
"select Id, ParentId, Type, Name, ProductionYear, Data from BaseItems"
" where Type in (?,?,?)", JF_TV):
node[i] = (pid, typ, name, year, data)
# Jellyfin 12 keeps provider ids in their own table; older builds kept them in
# the row's Data blob. Both shapes are read, because the tvdb id the series
# naming depends on is wherever this build happened to write it.
prov = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in ('Tmdb', 'Tvdb')"):
prov[(iid, pid)] = val
def provider_of(item_id, want):
got = prov.get((item_id, want))
if got:
return str(got)
try:
blob = json.loads(node[item_id][4] or "{}").get("ProviderIds") or {}
except (KeyError, ValueError, TypeError):
return ""
return str(blob.get(want, "") or "")
def series_of(item_id):
"""Climb episode -> season -> series; empty when the chain is broken."""
for _ in range(4):
if item_id not in node:
return "", "", "", ""
pid, typ = node[item_id][0], node[item_id][1]
if typ.endswith(".Series"):
return (node[item_id][2] or "", node[item_id][3] or "",
provider_of(item_id, "Tmdb"), provider_of(item_id, "Tvdb"))
item_id = pid
return "", "", "", ""
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
if not container:
container = "/data/" + os.path.basename(os.path.normpath(root))
prefix = container.rstrip("/") + "/"
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(JF_COLS)
for iid, path, idx, pidx, extra, virt, epname in con.execute(
JF_SQL, (JF_TV[0], prefix + "%")):
if virt or not path or not path.startswith(prefix):
skipped += 1 # a virtual item is a variant, not a file
continue
sname, syear, tmdb, tvdb = series_of(iid)
w.writerow([root + "/" + path[len(prefix):],
sname.replace("\t", " "), syear,
"" if pidx is None else pidx,
"" if idx is None else idx,
(extra or "").replace("\t", " "), tmdb,
(epname or "").replace("\t", " "), tvdb])
written += 1
print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
def music(root, db, container, out):
"""The tracks Jellyfin has already tagged, for the library at root.
Jellyfin reads music metadata out of the file itself, so an Audio row IS the
tag read, done once, by the vendor's own parser: its album-artist, album,
year, disc and track columns are what the embedded tags say and its Name is
the track title. That makes this export the whole authority for a music
relayout — the path a ripper chose is not evidence of anything, and the
folder Jellyfin put a track under is often a release pack, not an album.
Two gaps this export answers from the row rather than the tree. Many tracks
sit under a folder that is not an album item at all (a disc folder, or a
mixed one), so the album name and year fall back to the tags when the album
row is absent. And an album's MusicBrainz ids are not in the row's own Data
— the current schema leaves that empty for music — they are rows of
BaseItemProviders keyed by the item's Id.
"""
try:
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
except sqlite3.Error as e:
sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
"It is a WAL database: copy it with its -wal and -shm beside it and"
" point --db at the copy. Never sudo it — the data dir belongs to"
" the container's owner, and rootless storage is per-user.")
have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
need = {"Id", "ParentId", "Type", "Path", "Name", "IndexNumber",
"ParentIndexNumber", "ProductionYear", "Album", "AlbumArtists",
"Artists"}
if not need <= have:
sys.exit(f"{db} is not a Jellyfin library database with music columns"
f" (BaseItems has no {','.join(sorted(need - have))})")
# Provider ids, best identity first: the release group is the album, a bare
# release id is one reissue of it and never overrides a group.
mb = {}
if "BaseItemProviders" in {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}:
rank = {p: i for i, p in enumerate(MUSIC_PROVIDERS)}
best = {}
for iid, pid, val in con.execute(
"select ItemId, ProviderId, ProviderValue from BaseItemProviders"
" where ProviderId in (?, ?)", MUSIC_PROVIDERS):
if iid not in best or rank[pid] < best[iid][0]:
best[iid] = (rank[pid], val)
mb = {k: v[1] for k, v in best.items()}
# The fleet's quadlets bind the share at /data, so the container's view of
# this library is /data/<the library's own folder name> unless told otherwise.
if not container:
container = "/data/" + os.path.basename(os.path.normpath(root))
prefix = container.rstrip("/") + "/"
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(MUSIC_COLS)
for (path, aal, arts, tag_album, tag_year, disc, track, name, iid,
alb_name, alb_year, alb_id) in con.execute(
MUSIC_SQL, (JF_ALBUM, JF_AUDIO, prefix + "%")):
if not path or not path.startswith(prefix):
skipped += 1 # a row for a file outside this library
continue
# The album artist owns the folder; the track artist is the fallback
# for the compilations whose album-artist tag was left empty.
w.writerow([root + "/" + path[len(prefix):],
((aal or "").strip() or (arts or "").strip()),
(alb_name or tag_album or "").replace("\t", " "),
alb_year or tag_year or "",
"" if disc is None else disc,
"" if track is None else track,
(name or "").replace("\t", " "),
mb.get(alb_id or "") or mb.get(iid or "") or "",
(arts or "").replace("\t", " ")])
written += 1
print(f"tracks={written} outside-root-or-blank={skipped} -> {out}")
def audiobooks(root, db, out):
"""The chapters Plex matched inside the audiobook library, read-only.
Plex stores HOST paths here (unlike Jellyfin, which stores the container's
view), so no prefix mapping is needed: the part's `file` is the path the
operator has today. Rows for files outside `root` are skipped, because one
Plex library DB carries every library on the box.
The chapter's `index` is exported as an empty field when Plex has none. That
is not a gap to fill in here: three quarters of the books on this box contain
at least one un-indexed chapter, so the relayout that reads this file must
not renumber anything from it.
"""
con = open_plex_ro(db)
tables = {r[0] for r in con.execute(
"select name from sqlite_master where type='table'")}
missing = {"media_parts", "media_items", "metadata_items"} - tables
if missing:
sys.exit(f"{db} is not a Plex library database (no "
f"{', '.join(sorted(missing))})")
written = skipped = 0
with open(out, "w", newline="") as fh:
w = csv.writer(fh, delimiter="\t", lineterminator="\n")
w.writerow(AB_COLS)
for file, narrator, book, year, track, chapter in con.execute(
AB_SQL, (root.rstrip("/") + "/%",)):
if not file or not file.startswith(root.rstrip("/") + "/"):
skipped += 1 # a row for a file outside this library
continue
w.writerow([file.replace("\t", " "), narrator.replace("\t", " "),
book.replace("\t", " "), "" if not year else year,
"" if track is None or int(track) < 0 else track,
chapter.replace("\t", " ")])
written += 1
print(f"chapters={written} outside-root={skipped} -> {out}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
"help": "library root"}
e = sub.add_parser("enumerate", help="stat every file under the root")
e.add_argument("--root", **root)
e.add_argument("--out", default="/tmp/media-enum.tsv")
p = sub.add_parser("probe", help="ffprobe every video container")
p.add_argument("--enum", default="/tmp/media-enum.tsv")
p.add_argument("--out", default="/tmp/media-probed.tsv")
p.add_argument("--jobs", type=int, default=6)
x = sub.add_parser("plex", help="export Plex's matched titles for the root")
x.add_argument("--root", **root)
x.add_argument("--db", default=PLEX_DB)
x.add_argument("--out", default="/tmp/plex-movies.tsv")
j = sub.add_parser("jellyfin",
help="export the episodes Jellyfin has numbered for the root")
j.add_argument("--root", default=os.environ.get("MEDIA_ROOT",
"/mnt/entropy/TVShows"),
help="TV library root on the host")
j.add_argument("--db", default=JF_DB)
j.add_argument("--container", default=None,
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
m = sub.add_parser("music",
help="export the tracks Jellyfin has tagged for the root")
m.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"),
help="music library root on the host")
m.add_argument("--db", default=JF_DB)
m.add_argument("--container", default=None,
help="the in-container prefix that stands for --root"
" (default /data/<root's folder name>)")
m.add_argument("--out", default="/tmp/jellyfin-music.tsv")
b = sub.add_parser("audiobooks",
help="export the chapters Plex matched for the audiobook root")
b.add_argument("--root", default=os.environ.get("MEDIA_AUDIOBOOKS_ROOT",
"/mnt/entropy/Audiobooks"),
help="audiobook library root on the host")
b.add_argument("--db", default=PLEX_DB)
b.add_argument("--out", default="/tmp/plex-audiobooks.tsv")
a = ap.parse_args()
if a.cmd == "enumerate":
enumerate_lib(a.root, a.out)
elif a.cmd == "probe":
probe(a.enum, a.out, a.jobs)
elif a.cmd == "plex":
plex(a.root, a.db, a.out)
elif a.cmd == "music":
music(a.root, a.db, a.container, a.out)
elif a.cmd == "audiobooks":
audiobooks(a.root, a.db, a.out)
else:
jellyfin(a.root, a.db, a.container, a.out)
if __name__ == "__main__":
main()
+445
View File
@@ -0,0 +1,445 @@
#!/usr/bin/env python3
r"""media-music.py — a plan that re-files the music library into Jellyfin's own shape.
media-music.py --root DIR --enum TSV --rows TSV [--out TSV] [--exceptions FILE]
Jellyfin's music contract (https://jellyfin.org/docs/general/server/media/music/)
is not the movie one. Jellyfin reads the metadata out of the file, so a folder is
an ALBUM and nothing else, every disc of an album belongs in that one folder, the
tags decide the track order, and a file name only matters when it carries a
character known to break playback (`< > : " / \ | ? *`) or when a lyric file has
to match the basename of the track it belongs to. The plan is therefore:
Music/<album artist>/<Album> (<year>)/NN - <Title>.<ext>
... the same folder, D.NN - <Title> when the album has discs
Music/Various Artists/... when its tracks name
more than one artist
The authority is the tag row Jellyfin already read — `scripts/media-census.py
music` — never the release-pack folder a ripper made. A file with no row is not
guessed at: it parks under `<root>/_unsorted` with the relative path it has today.
The rules below exist because of what a downloaded discography pack actually
contains:
* A cuesheet beside exactly one audio file means the album is one image with a
track list inside it. That file keeps its own name: numbering it `01` would
claim it is one track, and Jellyfin shows one track either way.
* Sidecars — images, cuesheets, EAC logs, checksums, booklets, playlists —
follow the album the audio in their own folder belongs to. A folder holding
two albums gives a sidecar no single owner, so it parks instead of being
guessed into one.
* A video container (.mkv, .vob, .ifo, …) inside a music library is not music.
Renaming it to .mka/.m4a would tell Jellyfin to index video as audio, so it
parks for a human decision.
* Playlists move with their album and their CONTENTS are never rewritten: the
paths inside an .m3u go stale. Editing a file the operator owns to keep a
convenience file tidy is not this tool's job.
Nothing is deleted and nothing is overwritten. The output is a list of rename(2)
intents for `scripts/media-apply.py`, whose preflight refuses any target that is
already claimed and whose journal is the rollback.
"""
import argparse
import csv
import os
import re
import sys
from collections import Counter, defaultdict
# What Jellyfin will index as music. .mp4/.mkv/.webm are deliberately NOT here:
# the vendor doc says an audio-only stream in those containers is never read as
# music, and this tool renames for a reason it can prove, not to guess at one.
AUDIO_EXT = {".flac", ".mp3", ".m4a", ".m4b", ".ogg", ".oga", ".opus", ".wav",
".wma", ".aiff", ".aif", ".ape", ".dsf", ".dff", ".wv", ".mka",
".mpc", ".aac"}
VIDEO_EXT = {".mkv", ".vob", ".ifo", ".bup", ".mpg", ".mpeg", ".avi", ".mov",
".ts", ".m2ts", ".wmv", ".webm", ".mp4", ".m4v", ".iso", ".img",
".bdmv", ".divx", ".ogm"}
# A lyric has to carry the basename of its track, so it is named after the audio
# file and not after itself. .txt counts only when its stem matches an audio stem:
# a loose .txt in an album folder is a note.
LYRIC_EXT = {".lrc", ".elrc", ".txt"}
IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tiff"}
# The album-artist strings that mean "compilation", so such an album is not filed
# under one artist who does not own it.
VARIOUS = {"various", "various artists", "unknown", "unknown artist", "unknow"}
# The characters Jellyfin documents as breaking playback. They are dropped, not
# substituted: a replacement character is a guess at what the title meant.
BAD_CHARS = set('<>:"/\\|?*')
TRASH_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir"
r"|lost\+found)(/|$)")
ARGS = None
HELD = {} # path -> (bytes, ext)
ROWS = {} # path -> the tag row
PLACED = {} # audio path -> the destination the plan gave it
CLAIMED = set() # every destination path the plan has handed out
MOVES, KEEPS = [], []
PARKED = defaultdict(list) # reason -> [path]
NOTES = defaultdict(list) # section -> [line]
CUE_DIRS = set() # source dirs holding a .cue
AUDIO_BY_DIR = defaultdict(set) # source dir -> audio files a row names
CHILD_OF = defaultdict(set) # dir -> the dirs directly below it holding audio
FOLDERS_SEEN = {} # (dir, depth) -> album folders placed below it
SUBTREE_DEPTH = 2 # how far below a folder its audio may sit
ALBUMS = {} # (artist, album, year) -> the album group
AMBIGUOUS = object()
def dominant(values):
"""The spelling the tags mostly agree on: the most frequent value, ties broken
by sort so the answer never depends on the order rows arrived in."""
counts = Counter(values)
return min(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0]
def rel(path):
"""The path as seen from the library root."""
return os.path.relpath(path, ARGS.root)
def safe(name):
"""A name every client can carry: no reserved character, nothing
unprintable, no trailing dot or space, and short enough to rename."""
out = "".join(c for c in (name or "")
if c not in BAD_CHARS and (c.isprintable() or c == " "))
out = re.sub(r"\s+", " ", out).strip().rstrip(". ")
while out and len(out.encode("utf-8", "ignore")) > 200:
out = out[:-4].rstrip()
return out
def assign(src, folder, name, ext, kind, note):
"""Book one file into `folder` under `name`, renamed if that exact destination
is already taken — by another file in the plan, or by a file standing there
today. Two files claiming one path is the fault that costs a re-run of a 90k
plan, and so is a plan that only works if the renames happen in the right
order: the executor refuses any target that is occupied, so an occupied path
is never a destination, even when its occupant is scheduled to leave.
A file already standing where the contract puts it is a keep, not a move."""
stem = safe(name) or "file"
dst = os.path.join(folder, stem + ext)
if dst == src:
CLAIMED.add(dst.lower()) # it is standing on its own target
KEEPS.append((src, dst, kind))
return dst
# A file already in this folder under this name — or under the numbered
# variant the collision rule gives it — keeps the name it has. Renumbering it
# every run would turn the plan into a shuffle instead of a fixed point.
base = os.path.splitext(os.path.basename(src))[0]
if os.path.dirname(src) == folder and (base == stem
or base.startswith(stem + " (")):
CLAIMED.add(src.lower())
KEEPS.append((src, src, kind))
return src
n = 1
while dst.lower() in CLAIMED:
n += 1
dst = os.path.join(folder, f"{stem} ({n}){ext}")
if n > 1:
NOTES[note].append(rel(dst))
CLAIMED.add(dst.lower())
MOVES.append((src, dst, kind))
return dst
def park(path, reason):
"""Hold a file where it is, under the park folder, keeping its relative path."""
PARKED[reason].append(path)
dst = os.path.join(ARGS.park_dir, rel(path))
if dst != path:
CLAIMED.add(dst.lower())
MOVES.append((path, dst, "park"))
def under_park(path):
"""True for a file already parked: that pile is for a human to review, and
re-parking it every run would only push it deeper."""
return path.startswith(ARGS.park_dir + os.sep)
# ---------------------------------------------------------------------- albums
def group_albums():
"""Every tagged track into one group per (album-artist tag, album, year).
The key is the identity the tags assert, not the folder the file sits in: one
album arrives as three discs in three folders, and two copies of an album in
one pack are one album to Jellyfin, whose album item is that name, artist and
year. The artist in the key is the ALBUM-artist tag and never the performer,
because a compilation is one album whose tracks name many performers — keying
on the performer would file one album once per artist.
"""
for path, r in sorted(ROWS.items()):
if path not in HELD or under_park(path):
continue
d = os.path.dirname(path)
AUDIO_BY_DIR[d].add(path)
CHILD_OF[os.path.dirname(d)].add(d) # so a cover above CD1/CD2 sees both
tag = (r["artist"] or "").strip()
album = (r["album"] or "").strip()
performer = (r["performer"] or "").strip()
if not album:
park(path, "a track no album names")
continue
if not tag and not performer:
park(path, "a track no artist names")
continue
key = (tag.casefold(), album.casefold(), (r["year"] or "").strip())
g = ALBUMS.setdefault(key, {"tags": [], "albums": [], "year": key[2],
"members": [], "performers": set(),
"mb": set()})
g["members"].append(path)
g["tags"].append(tag)
g["albums"].append(album)
g["performers"].add(performer or tag)
if r.get("mb"):
g["mb"].add(r["mb"])
def folder_artist(g):
"""Whose shelf the album sits on. The album-artist tag owns it when it names
one; when the tags name only performers, a single performer names the album
and several make it a compilation, which belongs to no one of them. The key is
case-folded, so the spelling that reaches the folder is the one the tracks
mostly carry — see dominant()."""
tags = [t for t in g["tags"] if t]
if tags:
name = dominant(tags)
return "Various Artists" if name.casefold() in VARIOUS else name
if len({p.casefold() for p in g["performers"]}) == 1:
return dominant(g["performers"])
return "Various Artists"
def album_folder(g):
name = safe(dominant(g["albums"])) or "Untitled"
return os.path.join(ARGS.root, safe(folder_artist(g)) or "Unknown Artist",
f"{name} ({g['year']})" if g["year"] else name)
def is_cue_image(path):
"""A folder holding this one audio file and a cuesheet is an album image."""
d = os.path.dirname(path)
return len(AUDIO_BY_DIR.get(d, ())) == 1 and d in CUE_DIRS
def number_of(value, width):
return f"{int(value):0{width}d}" if str(value or "").isdigit() else ""
def place_albums():
for key in sorted(ALBUMS):
g = ALBUMS[key]
folder = album_folder(g)
discs = {g0["disc"] for g0 in (ROWS[p] for p in g["members"]) if g0["disc"]}
tracks = [int(ROWS[p]["track"]) for p in g["members"]
if str(ROWS[p]["track"] or "").isdigit()]
# More than one disc, or a disc numbered past the first: the track number
# alone cannot name a file uniquely, so the disc goes in front of it.
multi = len(discs) > 1 or any(str(d).isdigit() and int(d) > 1 for d in discs)
width = 3 if multi or (max(tracks, default=0) > 99) else 2
for path in sorted(g["members"]):
r = ROWS[path]
ext = HELD[path][1]
base = os.path.splitext(os.path.basename(path))[0]
if is_cue_image(path):
name = safe(base) or "image"
else:
title = safe(r["title"] or base) or "untitled"
num = number_of(r["track"], 2 if multi else width)
if multi and num:
num = f"{r['disc'] or '1'}.{num}"
name = f"{num} - {title}" if num else title
PLACED[path] = assign(path, folder, name, ext, "track",
"two tracks claimed one name inside one album")
# --------------------------------------------------------------------- sidecars
def folders_below(d, depth):
"""The album folders the audio in `d`, or within `depth` directories below it,
was placed in. Memoised, because every loose file in a big discography folder
asks the same question about it."""
if (d, depth) not in FOLDERS_SEEN:
seen = {os.path.dirname(PLACED[a]) for a in AUDIO_BY_DIR.get(d, ())
if a in PLACED}
if depth:
for child in CHILD_OF.get(d, ()):
seen |= folders_below(child, depth - 1)
FOLDERS_SEEN[(d, depth)] = seen
return FOLDERS_SEEN[(d, depth)]
def owner_folder(d):
"""The album folder that owns this directory — the first place up the tree
where audio sits, in the directory itself or within SUBTREE_DEPTH levels below
it, which is how a box with its discs in CD1/CD2 owns the cover lying above
them. None when no audio is near enough to say, AMBIGUOUS when the audio there
belongs to more than one album."""
while d.startswith(ARGS.root + os.sep) and d != ARGS.root:
folders = folders_below(d, SUBTREE_DEPTH)
if folders:
return next(iter(folders)) if len(folders) == 1 else AMBIGUOUS
d = os.path.dirname(d)
return None
def place_sidecars():
"""Everything that is not a tagged track: lyrics, images, cuesheets, logs,
booklets, playlists — and the clutter that is none of those."""
# The lyric rule needs the audio's stem as it is named TODAY, before the plan
# renames it, because that is what the sidecar's own name was copied from.
stem = {(d, os.path.splitext(os.path.basename(a))[0]): a
for d, audios in AUDIO_BY_DIR.items() for a in audios}
for path, (_size, ext) in sorted(HELD.items()):
if path in ROWS or under_park(path) or TRASH_DIR.search(path):
continue
d, base = os.path.dirname(path), os.path.splitext(os.path.basename(path))[0]
if ext in VIDEO_EXT:
park(path, "a video container inside the music library: Jellyfin will "
"not read it as music, and renaming it would claim it is")
elif ext in AUDIO_EXT:
park(path, "an audio file no provider row names — its tags are not in "
"the library Jellyfin has matched")
elif ext in (LYRIC_EXT | IMAGE_EXT) and (d, base) in stem:
# A lyric, or a picture named after its track: it follows that track
# and takes the name the track now has, or Jellyfin cannot find it.
audio = stem[(d, base)]
dst = PLACED.get(audio)
if dst:
folder, name = os.path.dirname(dst), os.path.splitext(
os.path.basename(dst))[0]
assign(path, folder, name, ext, "lyric",
"a sidecar whose name collided with another in its album")
else:
park(path, "a sidecar of a track the plan could not place")
else:
owner = owner_folder(d)
if owner is AMBIGUOUS:
park(path, "a sidecar whose folder holds more than one album, so "
"nothing says which one it belongs to")
elif owner:
assign(path, owner, base, ext,
"playlist" if ext in (".m3u", ".m3u8") else "sidecar",
"one album folder would have carried two files of the same "
"sidecar name")
else:
park(path, "not music, or music with nothing beside it to belong to")
# ---------------------------------------------------------------------- outputs
def write_plan():
"""mkdir rows for every destination folder and its whole chain, then the moves.
Parents first: the executor creates directories in the order they are listed."""
chain = set()
for _s, dst, _k in MOVES:
d = os.path.dirname(dst)
while d and d != ARGS.root and d != os.path.dirname(d):
chain.add(d)
d = os.path.dirname(d)
dirs = sorted(chain, key=lambda p: (p.count("/"), p))
with open(ARGS.out, "w") as fh:
fh.write("action\tsrc\tdst\tkind\n")
for d in dirs:
fh.write(f"mkdir\t\t{d}\tdir\n")
for src, dst, kind in sorted(MOVES, key=lambda m: (m[1], m[0])):
fh.write(f"move\t{src}\t{dst}\t{kind}\n")
for src, dst, kind in sorted(KEEPS, key=lambda m: (m[1], m[0])):
fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
parked = [m for m in MOVES if m[2] == "park"]
sized = sum(HELD[s][0] for s, _d, _k in MOVES if s in HELD)
print(f"plan: {len(MOVES) - len(parked)} moves, {len(parked)} parked"
f" ({sized / 2**40:.2f} TiB touched), {len(KEEPS)} already placed,"
f" {len(dirs)} folders -> {ARGS.out}")
def write_exceptions():
with open(ARGS.exceptions, "w") as fh:
fh.write(f"Music relayout exceptions for {ARGS.root}\n\n")
fh.write(f"Everything below is parked under {ARGS.park_dir}, keeping the\n"
"relative path it has today. Nothing is deleted and nothing is\n"
"overwritten: a parked file is one rename from wherever the plan is\n"
"pointed next.\n\n")
for reason in sorted(PARKED, key=lambda r: -len(PARKED[r])):
rs = PARKED[reason]
fh.write(f"{len(rs):>6} files {reason}\n")
for p in sorted(rs, key=rel)[:400]:
fh.write(f" {rel(p)}\n")
if len(rs) > 400:
fh.write(f" … and {len(rs) - 400} more\n")
fh.write("\n")
for section in sorted(NOTES, key=lambda s: -len(NOTES[s])):
lines = sorted(set(NOTES[section]))
fh.write(f"{len(lines):>6} {section}\n")
for line in lines[:200]:
fh.write(f" {line}\n")
if len(lines) > 200:
fh.write(f" … and {len(lines) - 200} more\n")
fh.write("\n")
no_year = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if not g["year"])
if no_year:
fh.write(f"{len(no_year):>6} albums carry no year in any tag, so their folders\n"
" are named without one; Jellyfin still matches them.\n")
fh.write(" " + ", ".join(no_year[:300]) + "\n\n")
clash = sorted(f"{folder_artist(g)}/{dominant(g['albums'])}"
for g in ALBUMS.values() if len(g["mb"]) > 1)
if clash:
fh.write(f"{len(clash):>6} albums put one folder name on more than one\n"
" MusicBrainz release group — different releases, one name.\n"
" They share the folder Jellyfin's own album item names.\n")
fh.write(" " + ", ".join(clash[:200]) + "\n\n")
fh.write(" Playlists move with their album and their contents are never\n"
" rewritten: the paths inside an .m3u go stale.\n")
print(f"exceptions -> {ARGS.exceptions}")
def main():
global ARGS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_MUSIC_ROOT",
"/mnt/entropy/Music"))
ap.add_argument("--enum", default="/tmp/music-enum.tsv",
help="the enumerate TSV (bytes, ext, path)")
ap.add_argument("--rows", default="/tmp/jellyfin-music.tsv",
help="the tag rows scripts/media-census.py music wrote")
ap.add_argument("--out", default="/tmp/music-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/music-exceptions.txt")
ap.add_argument("--park", default="_unsorted",
help="the folder inside the root that holds what cannot be filed")
ARGS = ap.parse_args()
ARGS.root = os.path.normpath(ARGS.root)
ARGS.park_dir = os.path.join(ARGS.root, ARGS.park)
read_inputs()
# Every path a file occupies today is claimed before a single destination is
# handed out: the executor refuses an occupied target, so a plan that aimed at
# one would only fail after the renames that were supposed to clear it.
CLAIMED.update(p.lower() for p in HELD)
for path, (_size, ext) in HELD.items():
if ext == ".cue":
CUE_DIRS.add(os.path.dirname(path))
group_albums()
place_albums()
place_sidecars()
write_plan()
write_exceptions()
def read_inputs():
with open(ARGS.enum) as fh:
for line in fh:
parts = line.rstrip("\n").split("\t")
if len(parts) >= 3:
HELD[parts[2]] = (int(parts[0]), parts[1].lower())
with open(ARGS.rows, newline="") as fh:
for row in csv.DictReader(fh, delimiter="\t"):
if row.get("file"):
ROWS[row["file"]] = row
gone = [p for p in ROWS if p not in HELD]
if gone:
NOTES["a tag row whose file the census no longer sees"].extend(gone)
if __name__ == "__main__":
main()
+881
View File
@@ -0,0 +1,881 @@
#!/usr/bin/env python3
r"""Plan a Jellyfin-format relayout of a movie library. DRY RUN ONLY: nothing moves.
media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
[--out TSV] [--exceptions TXT]
Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
metadata-cache export whose clean titles outrank the filename parser; the title
authority for everything Plex never matched is scripts/media-titles.py beside this
file. No media file is opened, so planning the whole library costs seconds.
PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
parents exist before children. EXCEPTIONS is the human review list: every file the
plan could not name a film for, every collision it refused, the READMEs the
executor must write into the piles it created.
Target contract (jellyfin.org/docs/general/server/media/movies):
* one folder per movie, named Title (year)
* the video inside is named exactly like the folder
* extra copies of one film are versions: "<base> - <label>.ext"; the base must
match the folder name character-for-character or Jellyfin splits them apart
* extras live in canonical subfolders (trailers, featurettes, deleted scenes,
interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
* sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
* <>:"/\|?* break the matcher, so they never survive here
* VIDEO_TS/BDMV folders support neither multiple versions nor parts
"""
import argparse
import collections
import csv
import importlib.util
import os
import re
# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
# this file is the title authority. Nothing here opens a media file.
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="library root the plan moves files inside")
ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
help="Plex metadata export (clean titles); absent = parser only")
ap.add_argument("--out", default="/tmp/jf-plan.tsv")
ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
return ap.parse_args()
ARGS = parse_args()
ROOT = ARGS.root
ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
r"|(additional|extra) languages|languages|resources")
AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
r"|(additional|extra) languages|commentar(y|ies)"
r"|audio commentary|audio tracks?")
AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
".opus", ".ogg", ".wma"}
NFO_EXT = {".nfo"}
IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
"banner": "banner", "logo": "logo", "clearlogo": "logo",
"thumb": "thumb", "landscape": "thumb", "art": "backdrop",
"clearart": "backdrop", "jacket": "poster", "cdart": "poster",
"discart": "poster", "disc": "poster"}
RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
# library holds six of those films), so leaving it in a name turns ONE movie folder
# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
# folders, none of them a movie.
BAD_CHARS = re.compile(r'[<>:"/|?*]')
PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
"spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
"da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
"bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
"the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
"eng", "english", "italian", "dual", "audio", "forced", "default"}
# Canonical title where the library only ever carries a localised name.
OVERRIDES = {"28 giorni dopo": "28 Days Later",
"28 settimane dopo": "28 Weeks Later"}
rows = []
for line in open(ENUM):
size, ext, path = line.rstrip("\n").split("\t")
rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
ext_by = {r["path"]: r["ext"] for r in rows}
size_by = {r["path"]: r["bytes"] for r in rows}
def rel(p):
return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
def parts_of(p):
return [q for q in rel(p).split("/") if q]
def is_qb(p):
return ".!q" in p.lower()
def disc_root(p, fold=True):
"""Absolute root of the disc tree containing p (the MOVIE folder that holds
BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
ps = parts_of(p)
kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
for i, q in enumerate(ps):
if q.upper() in kinds:
return ROOT + "/" + "/".join(ps[:i])
return None
def row_name(r):
"""The name a disc-tree entry should be identified by: the engine stores the
disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
return r.get("name_src") or r["path"]
def sanitize(name):
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return (name[:180] or "untitled")
def ok_year(y):
"""True for a year a real film could carry. Plex stores 0 for a file it indexed
but never MATCHED, and "0" is a digit string: trusting it named 67 movie
folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
WORD_SPLIT = re.compile(r"[^0-9a-z]+")
def words_of(text):
"""The matching words of a name — or of a whole path, which is how callers
pass a file together with the folders it sits in.
Split on everything that is not a letter or a digit: eng.norm does not touch
"/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
"(1960)/extras" and "(1954)" that no other name could ever agree with, and a
film could not be found by the folder that holds it — which is how one box
set's featurettes ended up in another film's folder. Both sides of every
comparison below go through here. A bare number counts only when it could be
a release year, so release junk ("1080", "2160", "2", "7") never poses as
agreement between two names.
"""
out = set()
for w in WORD_SPLIT.split(eng.norm(text)):
if not w or w in eng.STOP:
continue
if w.isdigit() and not 1870 <= int(w) <= 2100:
continue
out.add(w)
return out
# ------------------------------------------------------------------ Plex titles
# Plex has matched a large part of the library's media files to real titles
# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
# holds a MATCH, its title wins; the filename parser carries everything else
# (extras, disc trees, TV, and whatever Plex never matched).
def plex_titles():
if not os.path.exists(PLEX):
return {}
out = {}
with open(PLEX, newline="") as fh:
for r in csv.DictReader(fh, delimiter="\t"):
t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
# Only a match is an authority, and a match carries a year: an
# unmatched row echoes the filename, which the parser reads better
# than a copied release name. The junk filter stays for the matches
# whose title still carries the release.
if t and ok_year(y) and not re.search(
r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
out[r["file"]] = (t, y)
return out
PLEX_T = plex_titles()
# ------------------------------------------------------------- classification
erows = eng.load(PROBE)
for r in erows:
for k, v in OVERRIDES.items():
if k in r["title"]:
r["title"] = v
r["key"] = eng.key_of(r["title"])
r["base"] = eng.base_words(r["title"], r["key"])
marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
extras = {r["path"] for r in erows
if r["path"] not in marks
and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
matched=r["path"] in PLEX_T)}
eps = collections.defaultdict(list)
for r in erows:
if r["path"] in marks | extras:
continue
ep = eng.ep_of(r["path"])
if ep:
eps[ep[0]].append(r) # keyed by the SERIES, never the episode title
tv_keys = {k for k, v in eps.items() if len(v) >= 5}
tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
# A folder INSIDE a movie's folder is bonus territory even when its own name is
# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
# misspelled Extras folder, so the engine's folder rule never fired and the
# featurette became a movie folder of its own — whose words then out-scored the
# real Saboteur and drew that box set's other extras into it. The rule needs no
# metadata: the folder holding the longest film owns the shorter video filed
# under it, provided that shorter file's own name speaks the film's name — a
# whole other film filed there stays a film, exactly as the engine's folder rule
# exempts it. A different year in the file's own name IS that other film
# speaking, so it vetoes the rule.
def longest_of(rows):
"""The longest video in each folder: that folder's feature."""
out = {}
for r in rows:
if r["dur"] > 0:
d = os.path.dirname(r["path"])
if d not in out or r["dur"] > out[d]["dur"]:
out[d] = r
return out
def not_disc(r):
"""A disc tree's innards belong to the tree, never to the folder's film."""
return not any(c in DISC_DIRS for c in r["path"].split("/"))
FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
def under_a_feature(path, dur):
"""True when a folder above this file holds a longer film that this file's
own name names."""
d = os.path.dirname(os.path.dirname(path))
while len(d) > len(ROOT):
f = FEATURE_OF.get(d)
if f:
if f["dur"] < dur:
return False
theirs = (words_of(os.path.basename(f["path"]))
| words_of(os.path.basename(d)))
if not words_of(os.path.basename(path)) & theirs:
return False
y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
return not (y1 and y2 and y1.group() != y2.group())
d = os.path.dirname(d)
return False
extras |= {r["path"] for r in erows
if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
and not_disc(r) and under_a_feature(r["path"], r["dur"])}
film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
buckets = collections.defaultdict(list)
for r in film_rows:
buckets[eng.key_of(r["title"])].append(r)
def res_label(r):
m = RES_RE.search(r["res"] or "")
if m:
n, kind = m.group(1).lower(), m.group(2).lower()
return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
try:
w = int((r["res"] or "0x0").split("x")[0])
except (ValueError, IndexError):
w = 0
return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
def version_label(r):
bits = [res_label(r)]
vc = (r["vcodec"] or "").lower()
if vc in ("av1", "libaom", "libsvtav1"):
bits.append("AV1")
elif vc in ("vc1", "wmv3"):
bits.append("VC1")
if r["bytes"] > 40 * 2**30:
bits.append("REMUX")
if "imax" in r["path"].lower():
bits.append("IMAX")
return sanitize(" ".join(bits))
def group_name(items):
titles = collections.Counter(r["title"] for r in items)
ascii_t = [t for t in titles if t.isascii()]
title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
years = [r["year"] for r in items if r["year"] != "?"] or \
[r["soft"] for r in items if r["soft"]]
year = collections.Counter(years).most_common(1)[0][0] if years else None
return sanitize(f"{title} ({year})" if year else title)
groups = []
for key, items in buckets.items():
for c in eng.dur_clusters(items):
g = {"rows": c, "base": group_name(c)}
known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
if known:
# one film, one name: a Plex match for any copy names every copy
t, yr = collections.Counter(known).most_common(1)[0][0]
g["plex"] = t
g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
groups.append(g)
clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
for g in groups:
if g["base"] in clash:
# two genuinely different films share title+year: separate folders, and
# say so in the name, because Jellyfin can only match one item per folder
g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
g["flag"] = "same title and year as another film"
moves, mkdirs, problems, readmes = [], set(), [], []
# --------------------------------------------------------------- unnamed groups
# A movie folder has to READ as a title. The engine's runtime veto says any file
# over 45 minutes that is not named like an extra is a film whatever it is called
# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
# with no identity whose numeric keys went on to capture every subtitle named
# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
# real film), so the gate is: a year, or a word that is not release noise. With
# neither, the rows are bonus material until a human names them.
def named(g):
if re.search(r"\(\d{4}\)$", g["base"]):
return True
return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
unnamed = [g for g in groups if not named(g)]
groups = [g for g in groups if named(g)]
for g in unnamed:
for r in g["rows"]:
extras.add(r["path"])
problems.append(
f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
f"bonus pile, not a folder — give the file a real name and re-plan")
group_of = {}
for g in groups:
g["dir"] = f"{ROOT}/{g['base']}"
for r in g["rows"]:
group_of[r["path"]] = g
dr = disc_root(r["path"])
if dr:
group_of[dr] = g
by_path = {r["path"]: r for r in erows}
# -------------------------------------------------------------- bookkeeping
claimed = {}
existing = set(rows_map := {r["path"] for r in rows})
at_target, booked_src = set(), set()
def book(src, dst, kind):
if src == dst:
at_target.add(src)
booked_src.add(src)
return
if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
why = "rebook" if src in booked_src else "collision"
problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
if src in booked_src: # one source may only ever move once
problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
return
if dst.lower() in claimed:
problems.append(f"COLLISION {rel(dst)}\n from {rel(src)}\n and {rel(claimed[dst.lower()])}")
return
claimed[dst.lower()] = src
booked_src.add(src)
comp = dst.split("/")
for i in range(1, len(comp)):
mkdirs.add("/".join(comp[:i]))
moves.append((src, dst, kind))
def uniq_dir(d, stem, ext, hint=""):
"""A destination that no other source already took."""
cand = f"{d}/{stem}{ext}"
if cand.lower() not in claimed:
return cand
n = 2
while True:
cand = f"{d}/{stem} {n}{ext}"
if cand.lower() not in claimed:
return cand
n += 1
def media_dir(p):
dr = disc_root(p)
return os.path.dirname(dr) if dr else os.path.dirname(p)
# ------------------------------------------------------------------ films
# film / disc moves: the largest copy carries the folder's own name, the rest
# become Jellyfin versions; a disc tree moves as a whole into the movie folder
def vext(p):
return ext_by.get(p) or os.path.splitext(p)[1].lower()
disc_children = collections.defaultdict(list)
for q in rows:
dr = disc_root(q["path"])
if dr:
disc_children[dr].append(q["path"])
for g in groups:
primary_taken = False
disc_kinds = set()
for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
dr = disc_root(r["path"])
if dr:
inner = os.path.basename(dr).upper()
if inner in disc_kinds:
problems.append(
f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
f"share the folder — discs have no multiple versions")
continue
disc_kinds.add(inner)
for src in disc_children[dr]:
tail = os.path.relpath(src, dr)
fold = disc_root(src, fold=False)
if fold != dr and os.path.dirname(src) != dr:
tail = os.path.basename(dr) + "/" + tail
book(src, f"{g['dir']}/{tail}", "disc-file")
continue
label = "" if not primary_taken else f" - {version_label(r)}"
dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
n = 2
while dst.lower() in claimed:
dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
n += 1
r["dst"] = dst
primary_taken = True
book(r["path"], dst, "film")
g["primary"] = primary_taken
# ------------------------------------------------------------------ extras
EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
("interviews", r"interview|cast and crew|session piece|commentary"),
("featurettes", r"featurette|behind the scenes|making of|documentary"
r"|heritage|archival|video archive|music video"),
("samples", r"sample"),
("shorts", r"\bshort\b|short film"),
("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
r"|audition|test footage|watermark|workprint|poster"
r"|picture in picture|singalong|karaoke|original score")]
THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
r"|\bost\b|music|song", re.I)
BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
def extras_kind(path):
low = path.lower()
if THEME_RE.search(low):
return "theme-music"
if BACKDROP_RE.search(low):
return "backdrops"
for name, rx in EXTRA_KIND:
if re.search(rx, low):
return name
return "extras"
def attachable_title(path):
t, _y = eng.title_of(path)
return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
for g in groups:
g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
top_groups = collections.defaultdict(list)
for g in groups:
for t in g["tops"]:
top_groups[t].append(g)
group_by_key = collections.defaultdict(list)
for g in groups:
for r in g["rows"]:
group_by_key[eng.key_of(r["title"])].append(g)
group_by_dir = {}
for g in groups:
for r in g["rows"]:
group_by_dir.setdefault(media_dir(r["path"]), g)
dr = disc_root(r["path"])
if dr:
group_by_dir.setdefault(os.path.dirname(dr), g)
def group_words(g):
"""The words that identify a film group: its titles and the names of the
folders its files live in."""
if "words" not in g:
w = set()
if g.get("plex"):
w |= words_of(g["plex"])
for r in g["rows"]:
w |= words_of(r["title"])
w |= words_of(os.path.basename(media_dir(r["path"])))
g["words"] = w
return g["words"]
def score_words(text, groups):
"""Ranks film groups by how many of their words a path carries."""
words = words_of(text)
out = [(len(words & group_words(g)), g) for g in groups]
out.sort(key=lambda x: -x[0])
return out
def group_for_orphan(path):
"""Which movie folder an extra/sidecar belongs to. A collection folder holding
several films must be resolved by WORDS, never by position: taking the first
film of the folder attached every loose subtitle in a box set to whichever
movie happened to sort first."""
stem = os.path.splitext(os.path.basename(path))[0]
key = attachable_title(path)
if key and group_by_key.get(key):
cands = group_by_key[key]
top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
for r in g["rows"] if len(parts_of(r["path"])) > 1)]
pool = same_top or cands
return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
pool = top_groups.get(gtop, [])
if len(pool) > 1: # a loose collection folder (box set, bonus
n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
return g if n >= 2 else None # album) is resolved by words, not position
if len(pool) == 1:
return pool[0]
d = os.path.dirname(path)
while d and d != ROOT:
if d in group_by_dir:
return group_by_dir[d]
d = os.path.dirname(d)
return None
tv_tops = {parts_of(p)[0] for p in tv_set}
def film_folder_for(path):
"""A movie folder for a lone feature that has no group: derived from its own
title when a year proves it, else None (goes to the review pile)."""
t, y = eng.title_of(path)
if not t or y == "?":
return None
d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
mkdirs.add(d)
return d
for p in sorted(extras):
if disc_root(p):
continue
if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
continue # a special inside a TV set: staged with it
g = group_for_orphan(p)
ext = ext_by.get(p, os.path.splitext(p)[1].lower())
stem = os.path.splitext(os.path.basename(p))[0]
if g is None:
solo = film_folder_for(p)
if solo:
dst = f"{solo}/{sanitize(stem)}{ext}"
problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
f"{os.path.basename(solo)}")
else:
dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
problems.append(f"NO-FILM extra {rel(p)}")
else:
dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
ext, os.path.dirname(p)), "extra")
# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
# they belong to the film of the folder they sit in, kept as a named subtree.
for q in rows:
pth = q["path"]
dr = disc_root(pth)
if not dr or pth in by_path or pth in booked_src:
continue
g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
if g is None:
continue
inner = os.path.basename(dr)
problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
f"(a second master: compare before keeping both)")
for src in disc_children[dr]:
if src not in booked_src:
book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
# ----------------------------------------------------------------- sidecars
media_by_dir = collections.defaultdict(list)
for r in erows:
if r["path"] in marks:
continue
dst = r.get("dst")
if dst:
media_by_dir[media_dir(r["path"])].append(r)
def sidecar_target(path, e):
d = os.path.dirname(path)
stem = os.path.splitext(os.path.basename(path))[0]
if d == ROOT:
# A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
# set of root-level MOVIE files, and scoring a stray name against it
# attached every root-level leftover to whichever root-level film was
# largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
# way. A root-level stray is nobody's sidecar: it goes to the review pile.
return None, None
cands = media_by_dir.get(d, [])
if not cands:
nd = os.path.dirname(d) # a subs/ or Extras/ folder beside the media
while nd and nd != ROOT and not media_by_dir.get(nd):
nd = os.path.dirname(nd)
if media_by_dir.get(nd):
if nd != d: # a NAMED subfolder (Subs/, Extras/, the
g = group_for_orphan(path) # box set's own root) carries evidence
dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
if g is None or g["dir"] not in dirs:
# ...but only a STRONG word match outranks the folder the file
# sits in. A weak one is an accident: "3_Eng.srt" scored 1
# against a junk group named "3" and escaped its own movie's
# Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
# and genuinely belongs to the other film of the set.
n = score_words(stem, [g])[0][0] if g else 0
if n >= 2:
return (g, None)
cands = media_by_dir[nd]
if not cands:
g = group_for_orphan(path)
return (g, None) if g else (None, None)
if len(cands) > 1:
w = [t for t in eng.norm(stem).split() if t not in NOISE]
cands = sorted(cands, key=lambda r: (
-len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
-r["bytes"]))
return (None, cands[0])
for q in rows:
p, e = q["path"], q["ext"]
if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
continue
if os.path.basename(p).startswith("._"):
# macOS AppleDouble resource forks: a few hundred bytes carrying a name
# COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
# dead files through four movie folders (and they are video extensions in
# the AppleDouble case, so they also skipped every other filter). They
# claim nothing: one pile, and they stay in it — nothing is deleted here.
book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
"", os.path.dirname(p)), "junk")
problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
continue
if e == ".iso": # a DVD image: Jellyfin plays it as a movie file
g = group_for_orphan(p)
stem = os.path.splitext(os.path.basename(p))[0]
if g is not None:
book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
else f"{g['dir']}/{g['base']}{e}", "iso")
else:
solo = film_folder_for(p)
if solo:
book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
f"folder -> {os.path.basename(solo)}")
else:
book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
problems.append(f"NO-FILM disc image {rel(p)}")
continue
if e in VIDEO_EXT:
# A video the engine never probed (unreadable header) and not an AppleDouble
# remnant: the plan does not guess at it, so it stays where it is.
continue
stem = os.path.splitext(os.path.basename(p))[0]
g, media = sidecar_target(p, e)
if media is not None:
base = os.path.splitext(os.path.basename(media["dst"]))[0]
d = os.path.dirname(media["dst"])
elif g is not None:
base, d = g["base"], g["dir"]
else:
# Nothing names this file's film. NEVER scatter it: an archive set or an
# album tree must move as the unit it is, so the whole loose folder goes
# to the review pile and a film folder's own leftovers stay with it.
ps = parts_of(p)
if len(ps) < 2:
dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
elif len(top_groups.get(ps[0], [])):
dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
else:
dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
book(dst=dst, src=p, kind="orphan")
problems.append(f"NO-OWNER sidecar {rel(p)}")
continue
if e in IMG_EXT:
low = eng.norm(stem)
word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
kind = "poster" if word else next(
(IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
if re.search(r"\btheme\b", low):
dst = f"{d}/theme{e}"
else:
dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
e, os.path.dirname(p)), "image")
elif e in NFO_EXT:
book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
elif e in SUB_EXT | AUDIO_EXT:
toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
langs = [t.lower() for t in toks if t.lower() in LANGS]
flags = [t.lower() for t in toks if t.lower() in FLAGS]
keep = [sanitize(t) for t in toks
if t.lower() not in langs + flags + list(NOISE)
and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
if e in SUB_EXT:
# subtitles: <movie>/srt[/language]
d = f"{d}/srt"
pd = os.path.basename(os.path.dirname(p))
if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}" # a real language folder survives
elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
d = f"{d}/theme-music" # bonus audio: one canonical folder
pd = os.path.basename(os.path.dirname(p))
if not MUSIC_TOP_RE.search(pd): # an album folder keeps its name
d = f"{d}/{sanitize(pd)}"
else:
pd = os.path.basename(os.path.dirname(p))
if not AUDIO_DIR_OK.match(pd):
d = f"{d}/{sanitize(pd)}"
stem2 = ".".join(bits) or sanitize(os.path.basename(p))
book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
else:
book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
os.path.dirname(p)), "misc")
readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
"one-folder-per-movie layout; `_release-group watermarks/`, "
"`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
for key in tv_keys:
readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
"these are TV episodes found in the MOVIES library; move the folder "
"to a TV library source, else Jellyfin groups them wrongly as one "
"movie.")
# --------------------------------------------------------------------- TV
for key in tv_keys:
items = eps[key]
year = collections.Counter(r["year"] for r in items if r["year"] != "?")
yr = year.most_common(1)[0][0] if year else None
base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
tops = {parts_of(r["path"])[0] for r in items}
members = sorted([r["path"] for r in items]
+ [q for q in extras | marks if len(parts_of(q)) > 1
and parts_of(q)[0] in tops],
key=lambda q: q)
for src in members:
rel_parts = rel(src).split("/")
tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
book(src, f"{ROOT}/{base}/{tail}", "tv")
for p in sorted(marks):
stem, ext = os.path.splitext(os.path.basename(p))
book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
os.path.dirname(p)), "watermark")
for q in rows:
if is_qb(q["path"]):
problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
book(q["path"], q["path"], "qb-partial") # stays: resume data
# --------------------------------------------------------------- debug hook
if os.environ.get("JF_ISO"):
x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs DVDrip/High Anxiety [1977].iso"
print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
"| g", group_for_orphan(x) and group_for_orphan(x)["base"])
raise SystemExit
if os.environ.get("JF_DEBUG"):
import json
unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
print("at_target", len(at_target))
print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
for up in unp[:20]:
d = os.path.dirname(up)
print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
"| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
# -------------------------------------------------------------- validation
srcs = [m[0] for m in moves]
dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
for s in dup_src:
problems.append(f"DOUBLE-CLAIM source {rel(s)}")
all_src = set(srcs)
untouched = {r["path"] for r in rows} - all_src
bad_names, into_alive, lost = [], [], []
for src, dst, kind in moves:
for comp in dst[len(ROOT) + 1:].split("/"):
if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
or len(comp) > 200):
bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
if dst in untouched: # something staying put is there
into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
if dst.lower() in {s.lower() for s in untouched}:
into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
if dst.startswith(src + "/") and src in all_src:
lost.append(f"{rel(src)} moves into its own subtree")
# every accounted-for source must exist exactly once in the plan
covered = {r["path"] for r in rows}
not_planned = sorted(covered - all_src - at_target)
with open(PLAN, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
if d.startswith(ROOT + "/") and d != ROOT:
out.write(f"mkdir\t\t{d}\tdir\n")
for p in sorted(booked_src - {m[0] for m in moves}):
out.write(f"keep\t{p}\t{p}\n")
for src, dst, kind in sorted(moves, key=lambda m: m[0]):
out.write(f"move\t{src}\t{dst}\t{kind}\n")
kinds = collections.Counter(k for _s, _d, k in moves)
films_n = sum(1 for _s, _d, k in moves if k == "film")
folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
with open(EXCEPTS, "w") as out:
out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
out.write(f"kinds={dict(kinds)}\n")
out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
out.write("\n".join(sorted(set(readmes))) + "\n")
out.write("\n=== PROBLEMS ===\n")
out.write("\n".join(sorted(set(problems))) + "\n")
out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+ "\n".join(sorted(set(into_alive))) + "\n")
out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+ "\n".join(rel(p) for p in not_planned[:60]) + "\n")
print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
f"mkdirs={len(mkdirs)} film-files={films_n}")
print(f"kinds={dict(kinds)}")
print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
print(f"left-unmoved={len(not_planned)}")
print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
f"disc-trees={len(disc_children)}")
+35
View File
@@ -0,0 +1,35 @@
# The operator's tvdb id map: `folder <TAB> tvdb id <TAB> note`.
#
# These are the pack folders whose show tvdb's NAME SEARCH cannot reach: the
# record exists, but tvdb titles it in the show's own language and the English
# name rides in aliases the search never reads. Without a row here the folder
# keeps the release folder's words and no episode title; with one, both matchers
# read `[tvdbid-NNNN]` and the files rename to tvdb's own name and titles.
#
# Every id was read from tvdb's own record (`/series/<id>/extended`) on
# 2026-10-31, and the note states what that record says plus the neighbouring
# record a name search would have picked instead — the row is checkable without
# re-searching, because one wrong id renames a whole pack to the wrong show. The
# plan refuses a line whose id is not a number and lists every folder the map
# touched in the exceptions file.
#
# Folders the name search DOES reach (the EXACT and PUNCT classes of the proposal
# table) stay off this file on purpose: a map row would hide a live lookup behind
# a number nobody re-checks.
[Anime Time] Pokemon Complete Series (Season 01-25+Movies+Specials+Shorts+OST) [DVD+BD][1080p][HEVC 10bit x265][AAC] 76703 tvdb: ポケットモンスター (1997), the umbrella record for every era this pack spans; 1268 files
[Anime Time] One Piece (0001-1071+Movies+Specials) [BD+CR] [Dual Audio] [1080p][HEVC 10bit x265][AAC][Multi Sub] 81797 tvdb: ワンピース (1999), alias ONE PIECE; the bare alias "One Piece" belongs to 392276, the 2023 live action; 1072 files
[Judas] Naruto (Complete Series) [BD 1080p][HEVC x265 10bit][Multi-Audio][Multi-Subs] 78857 tvdb: NARUTO-ナルト- (2002), alias "Naruto" carries the full-width dashes; 220 files
Star Trek Deep Space 9 Deluxe DVD Boxset + Extras DVDRip TSV 72073 tvdb: Star Trek: Deep Space Nine (1993), alias "Star Trek Deep Space 9"; the box set's own words match nothing; 279 files
Top Gear UK 1-17 74608 tvdb: Top Gear (2002), no "UK" alias — the US version is 199051; 131 files
Black Clover 001 331753 tvdb: ブラッククローバー (2017), which holds NO alias at all; 378762 is Mugyutto! Black Clover (2019), a different show; 108 files
Shin Chan - Complete Eng Dub Series 2160p Ai Upscaled -Mesc 437356 tvdb: Shin Chan (2006) — the English dub record this pack is; 79654 is the 1992 Japanese original; 78 files
Fairy Tail (2009-2014) (Seasons 1-8 + OVAs) (BD 1080p)(HEVC x265 10bit)(Dual-Audio)(Eng-Subs)-Judas[TGx] 114801 tvdb: フェアリーテイル (2009), alias FAIRY TAIL; 410031 is FAIRY TAIL 100 NEN QUEST (2024), the sequel; 59 files
(CBB) Tokyo Ghoul - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-10bit-x265) 281249 tvdb: 東京喰種トーキョーグール (2014) — :re and √A are seasons of THIS record, confirmed by search; 58 files
[Anime Time] Demon Slayer - Kimetsu no Yaiba (S01+S02+S03+S04+Mugen Train Arc+Movie) [BD] [Dual Audio][1080p][HEVC 10bit x265][AAC][Eng Sub] 348545 tvdb: 鬼滅の刃 (2019), alias "Demon Slayer: Kimetsu no Yaiba" — the folder's hyphen splits it differently; 56 files
(CBB) Tengen Toppa Gurren Lagann - The Complete Series (BD 1080p)(Dual-Audio)(HEVC-x265-10bit) 80096 tvdb: 天元突破グレンラガン (2007), alias Tengen Toppa Gurren Lagann; 51 files
Fullmetal Alchemist - Brotherhood 85249 tvdb: 鋼の錬金術師 FULLMETAL ALCHEMIST (2009), alias "Full Metal Alchemist Brotherhood" — spaced, unlike this folder; 64 files
[NewbSubs] Attack on Titan~Shingeki no Kyojin Series (1080p Blu-ray x265 Dual Audio) 267440 tvdb: 進撃の巨人 (2013), aliases carry "Attack on Titan" and "Shingeki no Kyojin", joined here by a tilde; 27 files
Kimetsu no Yaiba 348545 the same record as the Anime Time pack above: two release folders, ONE show; 12 files
Rick and Morty The Anime 420279 tvdb: リック・アンド・モーティ:ジ・アニメ (2024), no English alias on the record; 10 files
[Anime Time] Dorohedoro (Season 1+OVA+Extra) [Dual Audio][BD][1080p][HEVC 10bit x265][AAC][Eng Sub] 370761 tvdb: ドロヘドロ (2020), alias Dorohedoro; 9 files
Ken Il Guerriero HOKUTO NO KEN 79156 tvdb: 北斗の拳 (1984), alias "Hokuto no Ken"; the Italian title matches nothing and 465216 is the 2026 remake; 5 files
Can't render this file because it contains an unexpected character in line 20 and column 182.
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env python3
r"""Ask tvdb what it calls the shows a plan renamed WITHOUT a tvdb id.
media-shows-tvdb.py [--plan TSV] [--root DIR] [--park DIR] [--map TSV]
[--out TSV] [--credentials FILE]
`media-shows.py` renames a series to tvdb's own words only when an id backs the
name; everything else keeps the metadata provider's words and no episode title,
and the plan's mkdir rows are where those folders show up. This script turns that
list into a decision sheet: one row per destination series folder that carries no
`[tvdbid-NNNN]`, with what tvdb's search answers for it and a verdict that says
HOW MUCH was guessed to get there.
It changes nothing. The sheet is for the operator; the file that changes
behaviour is `scripts/media-shows-tvdb-ids.tsv`, whose rows he approves. That is
the loop: plan -> this sheet -> approved map rows -> re-plan.
Verdicts, weakest last:
EXACT tvdb's own name equals the folder's words as they stand.
PUNCT it equals them only after release junk is stripped (pack numbers,
codecs, group names, `Complete Series`); the difference is spelling,
not identity.
NEAR tvdb answered something whose name is NOT the query. The best hit is
printed so a human can judge it, and it is never applied by anything.
NONE nothing left to ask with, or no answer at all.
Acceptance is deliberately unforgiving — a hit counts only when tvdb's name
equals the query with every non-alphanumeric dropped from both sides, and the
series' year still has to fall inside any year span the folder names. The
cleaner is the only thing that got smarter over the passes this tool replaces;
the acceptance test never moved, because a wrong id renames a whole pack to the
wrong show. A query that needs the cleaner is reported as PUNCT for exactly that
reason: the machine stripped words, so a human reads the row.
Read-only against tvdb. The credential file is the planner's own
(`api_key=`, optional `pin=`), read here and never printed.
"""
import argparse
import collections
import importlib.util
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
formatter_class=argparse.RawDescriptionHelpFormatter)
ARGS.add_argument("--plan", default="/tmp/tv-plan.tsv",
help="the plan media-shows.py just wrote (its mkdir rows are "
"the folders this sheet asks about)")
ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
help="the TV library the plan renames")
ARGS.add_argument("--park", default="",
help="the review pile inside it (its folders are not shows)")
ARGS.add_argument("--map",
default=os.path.join(os.path.dirname(os.path.abspath(__file__)),
"media-shows-tvdb-ids.tsv"),
help="the operator's approved map; its folders are reported as "
"MAPPED and never asked again")
ARGS.add_argument("--credentials",
default=os.path.expanduser("~/.config/secrets/tvdb"),
help="file with an api_key= line (and a pin= for a user-issued "
"key); mode 600, and never printed")
ARGS.add_argument("--out", default="/tmp/tvdb-proposal.tsv",
help="the decision sheet")
ARGS = ARGS.parse_args()
ROOT = os.path.abspath(ARGS.root)
PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
_spec = importlib.util.spec_from_file_location(
"eng", os.path.join(os.path.dirname(os.path.abspath(__file__)),
"media-titles.py"))
eng = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(eng)
# Release metadata, not the show: everything a scene folder name says about the
# ENCODING, the packaging or the languages, which tvdb has never heard of.
JUNK = re.compile(
r"complete|series|dual|multi|audio|sub|subs|multisub|dualaudio|"
r"\d{3,4}p|1080p|720p|2160p|480p|4k|hd|hdtv|cam|hdcam|telesync|x264|x265|hevc|avc|"
r"\d+bit|bit|bluray|blu-?ray|bdrip|web-?rip|web-?dl|webrip|hdtvrip|"
r"dvdrip|dvdr|dvdscr|tsr|tsv|mkv|avi|mp4|aac|ac3|dts|eac3|flac|ddp?5[.1]1|"
r"bd|dvd|rip|boxset|box|season|seasons|episodes?|specials?|shorts|ost|"
r"ova|ovas|oad|repack|proper|internal|mi?xed|budgetbits|deluxe|remux|"
r"remastered|uncut|collector|edition|extras?|volume|volumes|part|"
r"disc|disk|cd|ntsc|pal|region|fixed|patched|proof|english|esp|"
r"espagnol|french|vostfr|subtitul\w*|spanish|german|italian|ita|eng|fre|spa|lat|"
r"upscaled?|ai|dd|movie|movies",
re.I)
GROUP = re.compile(r"KONTRAST|TGx|Judas|Silence|FreetheFilms|NTb|DEADPOOL|Xploit|"
r"SM737|TSA|3OISL3R|PLAYWEB|EPiC|FLUX|HONEY|Zaxton|Cca3rW|"
r"monikeres|frank|hdh|psi|rtfpemaker|cbm|web|asser|astro|"
r"yfn|uindex|www|org|com|net|mesc|ataraxiaprime|seph\d*|"
r"[a-z]+subs?|edge", re.I)
BRACKET = re.compile(r"[\[({]([^\])}]*)[\])}]")
NUM_RANGE = re.compile(r"(?<!\w)\d{1,4}\s*(?:-|–|to|through)\s*\d{1,4}(?!\w)", re.I)
INSIDE = re.compile(r"all|complete|full|dual|multi|audio|sub|subs|ova|ovas|"
r"movies?|episodes?|extras?|bonus|pack|remux|region\s*\d*|"
r"bd|dvd|cd|ntsc|pal|hr-?dvd", re.I)
def drop_bracket(m):
"""A bracket says nothing when its words are junk; keep it when they aren't."""
toks = [t for t in re.split(r"[\s_.,+~&]+", m.group(1)) if t]
if not toks or all(INSIDE.fullmatch(t) or JUNK.fullmatch(t)
or GROUP.fullmatch(t)
or re.fullmatch(r"\d+[a-z]?", t, re.I)
or re.fullmatch(r"(19|20)\d\d", t) for t in toks):
return " "
return " " + m.group(1) + " "
def clean(name):
"""The folder's words with everything about the RELEASE taken out, so what is
left is what a person would type into tvdb. `Star Trek Deep Space 9 Deluxe
DVD Boxset + Extras DVDRip TSV` becomes `Star Trek Deep Space 9`; `SNL 16-20`
becomes `SNL`."""
name = re.sub(r"^\s*www\b.*?\b(org|com|net)\b\s*[-–]\s*", " ", name, flags=re.I)
name = re.sub(r"\bS\d{1,2}([ .Ee]\d+)?[-–Ee\d.]*", " ", name)
name = re.sub(r"\b\d{1,2}[xX]\d{1,3}\b", " ", name)
name = NUM_RANGE.sub(" ", name)
name = re.sub(r"(19|20)\d\d(?:\s*[-/–]\s*(?:19|20)\d\d)?", " ", name)
name = re.sub(r"\bby\s+[A-Za-z0-9_.-]+$", " ", name)
name = BRACKET.sub(drop_bracket, name)
name = re.sub(r"[\[()\]:,;!?*]", " ", name)
toks = re.split(r"[\s_.\-+~&]+", name)
out = []
for i, t in enumerate(toks):
if not t or JUNK.fullmatch(t) or GROUP.fullmatch(t):
continue
if re.fullmatch(r"\d+", t):
after = sum(1 for x in toks[i + 1:] if x
and not JUNK.fullmatch(x) and not GROUP.fullmatch(x)
and not re.fullmatch(r"\d+", x))
# A pack index (`001`) and a lead-in number over a real title
# (`01 Charles Barkley`) are noise; `30 Rock`, `24`, `Deep Space 9`
# are the title itself.
if len(t) >= 3 or (not out and after >= 2):
continue
out.append(t)
while out and out[-1].lower() in ("and", "with", "the", "a", "an", "to", "of"):
out.pop()
return " ".join(out).strip()
def years(name):
return [int(y) for y in re.findall(r"\b(19\d\d|20\d\d)\b", name)]
def bare(s):
return re.sub(r"[^a-z0-9]", "", s.lower())
CRED = {}
try:
for _line in open(ARGS.credentials):
if "=" in _line:
k, v = _line.strip().split("=", 1)
CRED[k.strip()] = v.strip()
except OSError:
print(f"tvdb: no credentials at {ARGS.credentials} — the sheet would be all "
"NEAR and NONE, which is not a decision sheet", file=sys.stderr)
sys.exit(2)
if not CRED.get("api_key"):
print(f"tvdb: no api_key in {ARGS.credentials}", file=sys.stderr)
sys.exit(2)
TOKEN = {"jwt": ""}
def call(path):
"""One tvdb v4 GET, logging in again once when the JWT expires mid-run."""
for attempt in (1, 2):
if not TOKEN["jwt"]:
body = {"apiKey": CRED["api_key"]}
if CRED.get("pin"):
body["token"] = CRED["pin"]
req = urllib.request.Request("https://api4.thetvdb.com/v4/login",
data=json.dumps(body).encode(),
headers={"Content-Type":
"application/json"})
TOKEN["jwt"] = json.load(
urllib.request.urlopen(req, timeout=30))["data"]["token"]
try:
req = urllib.request.Request("https://api4.thetvdb.com/v4" + path,
headers={"Authorization":
"Bearer " + TOKEN["jwt"]})
return json.load(urllib.request.urlopen(req, timeout=30))["data"]
except urllib.error.HTTPError as e:
if e.code == 401 and attempt == 1:
TOKEN["jwt"] = ""
continue
raise
MAPPED = {}
if ARGS.map and os.path.exists(ARGS.map):
for _line in open(ARGS.map):
_line = _line.rstrip("\n")
if not _line.strip() or _line.startswith("#"):
continue
cols = _line.split("\t")
if len(cols) > 1 and cols[1].strip().isdigit():
MAPPED[cols[0].strip()] = cols[1].strip()
# The folders the plan created inside the library, with the file count each one
# will hold. The review pile is not a show: its files belong to no identity, and
# naming the pile would put one junk series on the sheet for all of them.
shows = collections.Counter()
for line in open(ARGS.plan).read().splitlines()[1:]:
parts = line.split("\t")
if len(parts) < 4 or parts[0] != "mkdir":
continue
dst = os.path.abspath(parts[2])
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
continue
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
if top.startswith((".", "_")):
continue
if "[tvdbid-" in top:
continue # an id already backs it
shows.setdefault(top, 0)
for line in open(ARGS.plan).read().splitlines()[1:]:
parts = line.split("\t")
if len(parts) < 4 or parts[0] != "move":
continue
dst = os.path.abspath(parts[2])
if not dst.startswith(ROOT + os.sep) or dst.startswith(PARK + os.sep):
continue
top = os.path.relpath(dst, ROOT).split(os.sep)[0]
if "[tvdbid-" in top:
continue
shows[top] += 1
tally = collections.Counter()
rows = []
for folder in sorted(shows, key=lambda f: (-shows[f], f)):
files, tag, note = shows[folder], "NONE", ""
if folder in MAPPED:
rows.append([files, "MAPPED", folder, "", "", "", MAPPED[folder],
"the operator's map already states this id"])
tally["MAPPED"] += 1
continue
query, untouched = clean(folder), clean(folder) == folder
ys = years(folder)
if not query:
rows.append([files, "NONE", folder, "", "", "", "",
"nothing is left of the folder's name once release junk"
" is gone"])
else:
try:
hits = call("/search?query=" + urllib.parse.quote(query)
+ "&type=series") or []
except Exception as e: # noqa: BLE001
hits = []
note = f"tvdb did not answer: {e}"
want = bare(query)
pick = None
for h in hits:
if bare(h.get("name") or "") != want:
continue
hy = str(h.get("year") or "")
if ys and hy.isdigit() and not (min(ys) <= int(hy) <= max(ys)):
continue
pick = h
break
if pick:
tag = "EXACT" if untouched else "PUNCT"
rows.append([files, tag, folder, query, pick.get("name", ""),
str(pick.get("year", "") or ""),
str(pick.get("tvdb_id", "") or ""), ""])
elif hits:
best = hits[0]
rows.append([files, "NEAR", folder, query, best.get("name", ""),
str(best.get("year", "") or ""),
str(best.get("tvdb_id", "") or ""),
f"{len(hits)} answer(s); tvdb's own name is not the "
"query, so nothing would take it"])
else:
rows.append([files, "NONE", folder, query, "", "", "",
note or "tvdb has no series by that name"])
tally[rows[-1][1]] += 1
time.sleep(0.12)
# Two folders mapping to one record is not two shows; it is one show the release
# folders split. Say it on every row involved, or the operator approves the same
# id twice and reads the second as a mistake.
by_id = collections.Counter(r[6] for r in rows if r[6] and r[1] != "MAPPED")
for r in rows:
if r[6] and r[1] != "MAPPED" and by_id[r[6]] > 1:
r[7] = (f"MERGE: {by_id[r[6]]} folders map to tvdb {r[6]} — one show, "
"the releases split it; " + r[7]).strip("; ")
with open(ARGS.out, "w") as fh:
fh.write("files\tverdict\tfolder\tquery\ttvdb_name\ttvdb_year\ttvdb_id\tnote\n")
for r in rows:
fh.write("\t".join(str(c) for c in r) + "\n")
print(f"sheet: {len(rows)} folders"
+ "".join(f" {t}={n}" for t, n in sorted(tally.items()))
+ f" -> {ARGS.out}")
+1235
View File
File diff suppressed because it is too large. Load diff
+579
View File
@@ -0,0 +1,579 @@
#!/usr/bin/env python3
r"""The fleet's release-filename title parser: the grouping authority the relayout
plan uses for every file the media server never matched.
report: media-titles.py report [--probed TSV] [--out TSV]
the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
ffprobe census scripts/media-census.py writes.
parse: media-titles.py parse NAME [NAME…]
what title_of() makes of a name — the way to check a naming accident
before it merges two films into one folder.
scripts/media-plan.py imports this file; over there the paths are arguments.
Method — three independent signals, so a naming accident cannot merge two films:
* franchise KEY : first two significant title tokens. Stable across parts of a
collection ("resident evil", "jump street") but NOT film-specific — it only
selects candidates.
* DURATION : the identity check. Same master encodes differ by at most a
couple of seconds; different films / cuts do not collide.
* SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
are byte-twins (the hash pass confirms); same duration but different sizes are
two different releases of one film (deletable redundancy).
TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
would otherwise collapse a whole series into one "duplicate" group.
"""
import argparse
import collections, os, re, sys, unicodedata
# Library root: the folder directly under this one names the film, and every
# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
# their own; nothing here reads media, only the census TSVs about it.
ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
# Release-tag vocabulary: never part of a title, so the title stops here.
JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
JUNK_RE = re.compile(
r"^("
r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
r")$", re.I)
RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
YEAR_RE = re.compile(r"^(19|20)\d{2}$")
SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
WORD_RE = re.compile(r"[A-Za-z0-9']+$")
EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
"del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
"vs", "avec", "con"}
def norm(s):
s = unicodedata.normalize("NFKD", s)
return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
def fold_sep(s):
"""norm(), plus: every character that is left over is not a letter, digit or
apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
of the word. Release groups file titles with lookalikes for punctuation the
filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
those (no decomposition mapping), and the scrubber inside title_of DELETES
what it does not recognise, so five different V/H/S films squashed into the one
title "vhs2" and would have shared one folder. Deleting is the wrong operation:
they separate. Accents are gone by then — norm() removed the combining marks —
so "Amélie" stays "amelie" and never splits into "ame lie"."""
return " ".join("".join(c if c.isalnum() or c == "'" else " "
for c in norm(s)).split())
def title_of(path):
"""(title, year) from the basename.
Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
(2) Find the boundary: the first release-noise token (resolution / codec /
source / audio / language / group vocabulary) ends the title, so trailing
tags can never leak in. (3) Collect up to that boundary: leading index
numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
are skipped while nothing is collected; once a real word appears a sentinel
CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
"12. Monkeys. 1995") is a numeric title and is prepended to what follows.
"""
# fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
# docstring): the tokens below are then plain alnum/apostrophe words.
name = fold_sep(os.path.splitext(os.path.basename(path))[0])
raw = [x for x in SEP_RE.split(name) if x]
years = [t for t in raw if YEAR_RE.match(t)]
two_year = len(years) > 1 # "2001…1968": one of them is the title
toks, year = [], None
for t in raw:
if YEAR_RE.match(t):
year = t # last year wins
toks.append("\x00")
else:
toks.append(t)
stop, seen_word = len(toks), False
for i, t in enumerate(toks):
if t == "\x00":
continue
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
stop = i
break
if c.isalpha():
seen_word = True
# A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
# it is three digits or more ("007", "000", "1408") or the name carries no year
# at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
# the rare real title that looks like one ("0 Things I Hate About You (1999)")
# loses its zero to the parser — Plex's matched title overrides it anyway.
lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
keep, num_title = [], None
for t in toks[:stop]:
if t == "\x00":
if keep:
break # year closes the title
if two_year and t is not None and years[0] != year:
# two DIFFERENT years ("2001…1968"): the leading one is the title;
# "1940 - Fantasia (1940)" repeats one year, so it is not a title
num_title = num_title or years[0]
continue # leading year: not the title
if track_idx and IDX_RE.match(t) and not keep and t == lead:
continue # "03." / "1c." ordering
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if not keep and not c.isalpha():
num_title = num_title or (c or None)
continue # numeric title candidate
keep.append(c)
if len(keep) >= 8:
break
keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
title = " ".join(keep)
if not keep:
# nothing collected: a pure-numeric leading token is the title even when it
# sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
for t in raw:
c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
return c, year or "?" # never "2160p" / "1080": they are not pure digits
if (num_title and title and not keep[0].isdigit()
and (two_year or not track_idx)):
# a leading number that only echoes the release year ("1917.2019.2160p…") is
# not a title; one that stands alone ("300.2006…", "1900") is
title = f"{num_title} {title}"
title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
return title, year or "?"
# Bonus-material vocabulary: names that say nothing about the film, so an extras
# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
# Clips) — and the running-time guard in is_extra() carries the rest.
SEP_CLS = r"[\s._-]+"
def _bonus_rx(word):
return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
EXTRA_WORDS = [
"trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
"tv spot", "advert", "commercial", "behind the scenes", "making of",
"deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
"blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
"menus", "gallery", "still gallery", "stills", "poster", "posters",
"montage", "preview", "previews", "epilogue", "visual effects",
"music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
"test footage", "screen test", "audition", "cast and crew", "isolated score",
"picture in picture", "original score", "session piece", "sample",
"watermark",
]
# Words that describe bonus material which can run LONG (a feature-length
# commentary-track cut, a documentary about the film). Trusted only below
# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
"archival", "documentary"]
EXTRA_NAME_RE = re.compile(
"|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
EXTRA_LONG_NAME_RE = re.compile(
"|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
# search = it only mentions extras ("... + Extras" box sets hold feature films).
EXTRA_FOLDER_RE = re.compile(
r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
r"archiv(?:al|es)|video[\s._-]archive)", re.I)
VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
# Anything longer than this is a feature film whatever it is called: a 118-minute
# file named "The Menu" or "Trailer" is a movie, not bonus material.
SHORT_MAX = 45 * 60
# Long-shaped bonus vocabulary is only believed below this: a 40-minute
# commentary cut is an alternate version of the film, not a featurette.
LONG_BONUS_MAX = 25 * 60
def name_words(x):
"""Bare words of a file or folder name: no release metadata, no year."""
x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
return [w for w in SEP_RE.split(fold_sep(x))
if w and not YEAR_RE.match(w) and not RES_RE.match(w)
and not JUNK_RE.match(w) and w not in JUNK]
def mentions(short, long_):
"""True when one word list appears in the other, anywhere: a release names
the film at the front ("Alien (1979) - Trailer") or in the middle
("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
a, b = sorted([short, long_], key=len)
if not a:
return False
return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
def folder_is_bonus(folder):
"""True when a folder name carries bonus vocabulary as WHOLE words: the
vocabulary is matched against a padded word string, so "Extras" decides and
a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
"""True for bonus material, which must never be counted as a film: a file
called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
extras otherwise pair with each other across the whole library.
Evidence, strongest first:
* RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
unless it sits in an explicit bonus folder and metadata never matched it.
Bonus words that describe long-form material (commentary, documentary,
interview) are believed only
under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
* The file's OWN name carries bonus vocabulary -> bonus.
* ANY folder of the path below the film's own root folder carries bonus
vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
"Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
is exempt because it names the film, and a film may be titled with a
bonus word; and a file whose own name IS that folder's film name is the
feature, not the bonus ("Ghost in the Shell Movies/Extras/
Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
there stays a film).
Bonus vocabulary is matched as whole words only, so the film
"Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
the folder rule, and "…Extras" as a fragment of a word is never evidence.
"""
parts = [q for q in path.split("/") if q]
if not parts:
return False
name = (os.path.splitext(parts[-1])[0]
if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
# A bonus folder anywhere BELOW the film's own folder decides for its
# contents; the film's own folder is the film's name, whatever words it
# contains, so it is exempt. It is the first component AFTER ROOT's own
# components — those are the mount path, not folders of the library.
head = len([q for q in root.split("/") if q]) + 1
bonus_folder = False
for folder in parts[head:-1]:
if folder_is_bonus(folder):
# The folder's words are often the film's own name ("Angel's Egg
# Extras/"), and then they are not bonus evidence. Only WORDS can name
# a film: "extras" is release junk that name_words drops, so a folder
# like "01. Extras" reduces to its index — and an index matching an
# index ("NCOP 01") is a coincidence, not a title.
nw = [w for w in name_words(name) if not w.isdigit()]
fw = [w for w in name_words(folder) if not w.isdigit()]
bonus_folder = not fw or not mentions(nw, fw)
break
if dur and dur > SHORT_MAX:
# Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
# and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
# a movie's Featurettes/ is a disc extra, and nothing in the library claims
# it as a film. Long-form bonus words still describe a film, so a
# feature-length commentary cut or documentary keeps its runtime veto.
if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
return True
return False
if EXTRA_NAME_RE.search(name):
return True
if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
return True
return bonus_folder
# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
# "RARBG.com.mp4") into many different films, so its name carries no film
# identity at all and it would otherwise pair with itself across the library.
WATERMARK_MAX = 20 * 2**20
WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
def is_watermark(path, size):
"""True for a release group's own watermark video, not a film or an extra."""
return (size < WATERMARK_MAX
and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
def key_of(title):
"""Franchise key: first two alphabetic tokens (numeric-only title = itself).
Deliberately NOT film-specific — it only selects candidates for the duration
check, so a two-token collision (two films starting "star trek") cannot by
itself create a duplicate group."""
toks = [t for t in title.split() if t]
words = [t for t in toks if not t.isdigit()]
if not words:
return " ".join(toks)
sig = [t for t in words if t not in STOP] or words
return " ".join(sig[:2])
def ep_of(path):
"""(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
The series is the name BEFORE the marker. The box-set convention writes
"Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
follows the marker is not the show's name: keying on the whole title gave
every episode of one series its own group, so a 124-episode box set never
reached the size that calls something a series and each episode was filed
as its own film. A file named only by its marker has nothing before it, so
its whole title stands in."""
name = os.path.splitext(os.path.basename(path))[0]
m = EP_RE.search(name)
if not m:
return None
head = name[:m.start()].strip(" ._\u2013-")
series = title_of(head)[0] if head else None
return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
def base_words(title, key):
"""Title tokens beyond the franchise key — the FILM identity part.
"evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
kt = set(key.split())
return frozenset(t for t in title.split() if t not in kt and t not in STOP
and not t.isdigit() and not YEAR_RE.match(t))
def folder_year(path):
"""Soft year evidence: a 19xx/20xx token in any parent directory name
(collection folders are written both "(2002)" and "[2009]"; a range such as
"(1998-2000)" deliberately yields nothing)."""
for d in reversed(os.path.dirname(path).split("/")):
ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
if len(ys) == 1:
return ys[0]
return None
def same_film(a, b):
"""Two files are copies of ONE film only when the film-identity tokens agree
(one side may carry none) AND no year evidence contradicts them. The key says
the franchise matches; this says the actual film matches — it is what keeps
'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
Powers apart when their runtimes happen to agree."""
ba, bb = a["base"], b["base"]
if ba and bb and ba != bb:
return False
ya = a["year"] if a["year"] != "?" else a["soft"]
yb = b["year"] if b["year"] != "?" else b["soft"]
if ya and yb and ya != yb:
return False
return True
def dur_close(a, b):
if a <= 0 or b <= 0:
return True # unknown duration: let titles decide
return abs(a - b) <= max(2.0, 0.003 * max(a, b))
# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
# is folded into ONE entry, described by its longest main-feature clip (measured
# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
BDMV_MAIN = {
"Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
"Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
}
def load(probed):
"""Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
BDMV tree collapses to one entry carrying the tree's total size."""
trees = {}
rows = []
for line in open(probed):
f = line.rstrip("\n").split("\t")
if len(f) < 12:
continue
b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
base = os.path.basename(p)
if base.startswith("._") or base == ".DS_Store":
continue # macOS sidecar, not media
m = re.search(r"/(BDMV|VIDEO_TS)/", p)
root = p[:m.start()] if m else None
if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
if not root:
continue # disc bookkeeping: folds into its tree, never a film
if root:
t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
t["bytes"] += int(b)
t["audio"].update(a for a in aud.split("+") if a)
t["subs"] += int(subs or 0)
continue
try:
dur = float(dur)
except ValueError:
dur = 0.0
rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
for root, t in trees.items():
folder = os.path.basename(root)
m = BDMV_MAIN.get(folder, {})
rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
name_src=folder, dur=m.get("dur", 0.0),
vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
fmt="BDMV tree"))
for r in rows:
r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
r["key"] = key_of(r["title"])
r["base"] = base_words(r["title"], r["key"])
r["soft"] = folder_year(r["path"])
r["ep"] = ep_of(r["path"])
return rows
def dur_clusters(items):
"""Group files that are the SAME FILM and the SAME MASTER duration. Candidate
buckets come from the franchise key; same_film then rejects the different
installments whose runtimes happen to agree."""
clusters = []
for r in sorted(items, key=lambda r: -r["dur"]):
for c in clusters:
if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
c.append(r)
break
else:
clusters.append([r])
return clusters
def tier(groups):
"""Split same-film/same-duration clusters: DUP = different releases of one
film (real, deletable redundancy); REVIEW = identical size AND duration
(byte-twins — the hash pass is the authority) or clashing year evidence."""
dup, review = [], []
for g in groups:
g.sort(key=lambda r: -r["bytes"])
years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
years.discard(None)
spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
if len(years) > 1 or spread < 0.005:
review.append(g)
else:
dup.append(g)
return dup, review
def redun(g):
return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
def line(r):
return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
def dump(tag, groups, out, note=""):
for gi, g in enumerate(groups, 1):
g.sort(key=lambda r: -r["bytes"])
print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
for rank, r in enumerate(g):
print(f" {line(r)} [{r['year']}] "
f"{r['path'].replace(ROOT + '/', '')}")
out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
f"{r['path']}\n")
def report(probed, out_path):
rows = load(probed)
for r in rows:
r["mark"] = is_watermark(r["path"], r["bytes"])
r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
marks = [r for r in rows if r["mark"]]
movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
extras = [r for r in rows if r["extra"]]
print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
f"TV episodes + {len(extras)} bonus/trailers "
f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
f"+ {len(marks)} release-group watermarks")
out = open(out_path, "w")
buckets = collections.defaultdict(list)
for r in movies:
buckets[r["key"]].append(r)
cand = {k: v for k, v in buckets.items() if len(v) > 1}
clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
dup, review = tier(clusters)
dup.sort(key=lambda g: -redun(g))
review.sort(key=lambda g: -redun(g))
dump("DUP", dup, out)
dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
ebuf = collections.defaultdict(list)
for r in episodes:
ebuf[(r["ep"][0], r["ep"][1])].append(r)
edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
for c in dur_clusters(v) if len(c) > 1])
edup.sort(key=lambda g: -redun(g))
dump("EP-DUP", edup, out)
shows = collections.Counter()
for r in episodes:
shows[r["ep"][0]] += 1
multi = {s: n for s, n in shows.items() if n > 4}
print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
print(f" {n:4d} episodes {s}")
out.close()
print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
print(f"REVIEW groups={len(review)} "
f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
print(f"EP-DUP groups={len(edup)} "
f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
def parse(names):
for n in names:
t, y = title_of(n)
print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = ap.add_subparsers(dest="cmd", required=True)
rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
rep.add_argument("--probed", default="/tmp/media-probed.tsv")
rep.add_argument("--out", default="/tmp/media-similar.tsv")
par = sub.add_parser("parse", help="show the parsed title/year of a name")
par.add_argument("names", nargs="+")
a = ap.parse_args()
if a.cmd == "report":
report(a.probed, a.out)
else:
parse(a.names)
if __name__ == "__main__":
main()
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""media-tv.py — move TV episodes out of the movie library into a TV library.
`scripts/media-plan.py` refuses to file TV: it parks episode files in a
`_TV needs a TV library - <series>` pile for a human. This is the other half of
that refusal. It walks the movie library, groups the files whose own name
carries an episode marker by SERIES, and writes a plan that places each series
in the shape both Jellyfin and Plex read without help:
<library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
It writes a plan and nothing else. Every rename belongs to
`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
the executor's own path guard still has teeth, and give it its own journal —
one journal is one undo, and a TV move must not be welded to a movie relayout:
python3 scripts/media-tv.py --root /mnt/entropy/Movies \
--library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
--root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
python3 scripts/media-apply.py sweep --root /mnt/entropy \
--journal /mnt/entropy/media-relayout-tv.jsonl --apply
Guards, and why each one is here:
* The same five-file threshold as the planner. An episode-shaped name on one
file is not a series — `The Godfather Part 2` is a film — so a smaller group
is reported and left exactly where it is.
* An episode whose name carries no season number is reported and left, never
filed into an invented `Season 01`.
* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
episode; an episode without its subtitles is a half-move.
* A group that would claim one destination twice is not resolved here. The
executor refuses the whole plan before its first rename, which is the right
outcome: nobody but Frank decides which copy of an episode is the keeper.
* An episode the destination library already holds at the same byte count is
left where it is. The TV library has held a complete Monk box set for years
while the movie library carried a second copy of all 124 episodes; a mover
that did not look where it was filing would have put both inside one library.
* A symlink is never followed and never moved.
"""
import argparse
import collections
import importlib.util
import os
import re
def parse_args():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
help="the movie library to empty of TV")
ap.add_argument("--library", default="/mnt/entropy/TVShows",
help="the TV library to fill (a different share or folder)")
ap.add_argument("--out", default=None,
help="plan TSV for scripts/media-apply.py"
" (default: media-tv-plan.tsv beside --root)")
ap.add_argument("--min", type=int, default=5,
help="episode files that make something a series")
a = ap.parse_args()
if not a.out:
a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
"media-tv-plan.tsv")
return a
ARGS = parse_args()
ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
spec = importlib.util.spec_from_file_location("eng", ENGINE)
eng = importlib.util.module_from_spec(spec)
spec.loader.exec_module(eng)
BAD_CHARS = re.compile(r'[<>:"/|?*]')
# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
# Sidecars that follow an episode; macOS forks and resume data do not.
SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
# A box set that stamps the show's year into every file name; the folder's year
# must not be added on top of it.
YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
"by", "de", "la", "vs"}
def clean(name):
"""A name the filesystem and both matchers can read (same law as the plan)."""
name = BAD_CHARS.sub("-", name).replace("\\", "-")
name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
return name[:180] or "untitled"
def episode_name(path):
"""The file's own name, kept: the publisher's spelling beats a re-derivation.
A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
already carries spaces is left exactly as it was written."""
stem, ext = os.path.splitext(os.path.basename(path))
if " " not in stem and stem.count(".") >= 2:
stem = stem.replace(".", " ")
return clean(stem) + ext.lower()
def head_of(path):
"""The publisher's series name: the text BEFORE the episode marker.
`Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
after the marker is not the show's name, and case is kept because it is the
library's own spelling, not a normalised key."""
name = os.path.splitext(os.path.basename(path))[0]
m = eng.EP_RE.search(name)
head = name[:m.start()] if m else name
return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
def title_series(head):
"""The series folder in the case a TV library is written in.
Only a name with no capital at all is retitled: a box set that was
lower-cased on the way in would otherwise file `monk (2002)` among
`Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
exactly as it was written."""
if any(c.isupper() for c in head):
return head # the publisher wrote it with case; keep it
words = head.split()
return " ".join(w if i and w.lower() in SMALL else w.capitalize()
for i, w in enumerate(words))
def season_episode(token):
"""(season, episode) an episode token states, or None when it states none."""
m = SEASON_RE.match(token)
if not m:
return None
return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
def sidecars(path):
"""The files beside PATH that carry its exact stem — its subtitles, its nfo."""
d, stem = os.path.split(path)
stem = os.path.splitext(stem)[0]
out = []
try:
entries = sorted(os.listdir(d))
except OSError:
return out
for e in entries:
p = os.path.join(d, e)
if p == path or os.path.islink(p) or not os.path.isfile(p):
continue
s, _ext = os.path.splitext(e)
if s == stem and not SKIP_SIDECAR.match(e):
out.append(p)
return out
def episodes_under(root):
"""(path, series, token) for every episode-shaped video file under root."""
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames
if not d.startswith(".") and d != "@eaDir")
for f in filenames:
p = os.path.join(dirpath, f)
if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
continue
e = eng.ep_of(p)
if e:
yield p, e[0], e[1]
def walk(root, library):
"""Episode files under ROOT keyed by series, ignoring the destination."""
groups = collections.defaultdict(list)
lib = os.path.abspath(library)
for p, series, token in episodes_under(root):
if os.path.abspath(p).startswith(lib + os.sep):
continue
groups[series].append((p, token))
return groups
def holdings(library):
"""What the destination already carries: (series, season, episode) -> sizes.
The one expensive read here, and the one that matters. The movie library
holds a complete Monk box set that the TV library has held for years: 124
episodes, the same episode list, identical byte counts. Moving it would
have filed a second copy of a whole show into the library that already had
it. Byte count is the test on purpose — two encodes of one episode are two
files and both are wanted; the same bytes are the same file."""
out = collections.defaultdict(set)
for p, series, token in episodes_under(library):
se = season_episode(token)
if se is None:
continue
try:
out[(series, *se)].add(os.path.getsize(p))
except OSError:
continue
return out
def main():
root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
groups = walk(root, library)
held_here = holdings(library)
mkdirs = {library}
rows, lines, skipped = set(), [], []
for series, items in sorted(groups.items()):
if len(items) < ARGS.min:
skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
f"episode-shaped file{'s' if len(items) != 1 else ''},"
f" below {ARGS.min}: not a series")
continue
years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
if y)
year = years.most_common(1)[0][0] if years else None
heads = collections.Counter(head_of(p) for p, _t in items)
head = heads.most_common(1)[0][0]
if year and YEAR_TAIL.search(head):
year = None
name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
series_dir = os.path.join(library, name)
moved = held = 0
for path, token in items:
se = season_episode(token)
if se is None:
skipped.append(f"skipped: {path} — no season number in its name,"
" nothing here invents one")
continue
try:
size = os.path.getsize(path)
except OSError:
continue
if size in held_here.get((series, *se), ()):
held += 1
continue
dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
mkdirs.update([series_dir, dst_dir])
base = episode_name(path)
rows.add((path, os.path.join(dst_dir, base)))
moved += 1
for side in sidecars(path):
rows.add((side, os.path.join(
dst_dir, clean(os.path.splitext(base)[0]
+ os.path.splitext(side)[1].lower()))))
if moved:
lines.append(f"{name}: {moved} episodes"
f"{'' if year else ' (no year in any folder name)'}"
f" -> {series_dir}")
if held:
lines.append(f"{name}: {held}"
f" {'episode' if held == 1 else 'episodes'} already in"
f" {library} at the same byte count — left there")
# An empty plan creates nothing, not even the library folder: a run that
# found no series must not be able to leave a stray directory behind.
dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
with open(ARGS.out, "w") as out:
out.write("action\tsrc\tdst\tkind\n")
for d in dirs:
out.write(f"mkdir\t\t{d}\tdir\n")
for src, dst in sorted(rows):
out.write(f"move\t{src}\t{dst}\ttv\n")
for line in lines + skipped:
print(line)
print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
if rows:
print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
f" --root {os.path.dirname(root)}"
f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
" (add --apply to run it)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env python3
"""Prove a relayout landed, using the media server's own tables.
A scan answers `204` whether or not it found anything, and a rename that the
scanner has not followed looks exactly like a rename that never happened from
the outside. The only honest proof is the run's journal replayed against the
database the server actually serves from: every `move` row must have a row at
its destination, and NO row may still name its source — a source row is a scan
that has not caught up, and it is the one signature of a library pointing at
files that no longer exist.
Two doors, both read-only, both without a lock while the server runs (there is
no `sqlite3` binary on these hosts). Open `mode=ro` PLAIN: both servers keep
their libraries in WAL mode, and `immutable=1` — which claims the file never
changes — makes sqlite skip the write-ahead log entirely, so every row the
server has committed but not yet checkpointed reads as MISSING, and a read
beside a running scanner can even fail as "database disk image is malformed".
`immutable=1` is only the fallback, for a database whose `-shm` cannot be read.
jellyfin `BaseItems.Path`. The container binds the share at `/data`, so the
database holds `/data/<library>/…` where the journal holds
`/mnt/entropy/<library>/…`. Translate, or every row reads as missing.
plex `media_parts.file`, joined up to `metadata_items`. Plex is bound at
the host path, so its rows match the journal verbatim. Filter
`deleted_at is null` on BOTH tables — a deleted scan leaves its parts
behind, and one of those rows is not proof anything landed.
Neither server holds a row for everything the journal renames: a `.log`, `.cue`
or cover image has no row to land in, so its absence is not a fault. The report
therefore breaks the missing rows down by extension, and `--ignore-ext` names the
extensions this server does not index, which are then counted separately and kept
out of the verdict. What is left is split by what the filesystem says, because
the database alone cannot: a file still at its `src` is `REAL` (this pass's own
damage), a file at neither path is `gone` (a later pass or the server renamed it),
and a file sitting at its `dst` with no row naming it is simply something the
server does not index. A row that still names `src` is none of these: it is the
server's own stale pointer — a phantom — and it costs a re-scan, never a restore.
Exit codes: 0 every move landed; 1 something is stale or missing (the details are
printed); 2 the journal or the database is not usable at all.
"""
import argparse
import json
import os
import sqlite3
import sys
from collections import Counter
JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
"/Plug-in Support/Databases/com.plexapp.plugins.library.db")
# Per kind: the table whose columns are required, those columns, and the query
# that returns every path the server believes it has.
KINDS = {
"jellyfin": {
"table": "BaseItems",
"columns": {"Path"},
"sql": "select Path from BaseItems",
"default_db": JF_DB,
},
"plex": {
"table": "media_parts",
"columns": {"file", "media_item_id", "deleted_at"},
"sql": ("select p.file from media_parts p"
" join media_items m on m.id = p.media_item_id"
" join metadata_items mi on mi.id = m.metadata_item_id"
" where p.deleted_at is null and mi.deleted_at is null"),
"default_db": PLEX_DB,
},
}
def die(msg):
print(f"media-verify: {msg}", file=sys.stderr)
sys.exit(2)
def read_moves(path):
"""The (src, dst) pairs of a journal, in the order the run performed them."""
moves = []
try:
fh = open(path, encoding="utf-8")
except OSError as e:
die(f"cannot read the journal {path}: {e.strerror}")
with fh:
for n, line in enumerate(fh, 1):
line = line.strip()
if not line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
die(f"{path} line {n} is not a journal record")
if rec.get("op") == "move":
if not rec.get("src") or not rec.get("dst"):
die(f"{path} line {n} is a move without both src and dst")
moves.append((rec["src"], rec["dst"]))
return moves
def server_paths(db, kind):
"""Every path the server holds, or a refusal that names what is missing."""
spec = KINDS[kind]
con = None
err = None
# Plain ro reads the WAL the server commits through; immutable is the last
# resort, because it cannot see a row the server has not checkpointed.
for opt in ("mode=ro", "mode=ro&immutable=1"):
try:
con = sqlite3.connect(f"file:{db}?{opt}", uri=True)
break
except sqlite3.Error as e:
err = e
if con is None:
die(f"cannot open {db}: {err}")
with con:
have = {r[1] for r in con.execute(f"pragma table_info({spec['table']})")}
if not spec["columns"] <= have:
die(f"{db} is not a {kind} library database ({spec['table']} has no "
f"{','.join(sorted(spec['columns'] - have))})")
try:
return {r[0] for r in con.execute(spec["sql"]) if r[0]}
except sqlite3.Error as e:
die(f"{db} refused the {kind} query: {e}")
def to_host(path, container, share):
"""The database's path in the journal's terms; identity when it is not a
container-side path."""
if not container:
return path
prefix = container.rstrip("/") + "/"
if path.startswith(prefix):
return share.rstrip("/") + "/" + path[len(prefix):]
return path
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--journal", required=True,
help="the run's journal (scripts/media-apply.py writes it)")
ap.add_argument("--db-kind", choices=sorted(KINDS), required=True)
ap.add_argument("--db", default=None, help="the server's database on this host")
ap.add_argument("--share", default="/mnt/entropy",
help="the share every journal path lives under")
ap.add_argument("--container", default=None,
help="what the share is called inside the container; empty"
" compares verbatim (default /data for jellyfin, nothing"
" for plex)")
ap.add_argument("--show", type=int, default=20,
help="how many offenders to name before saying 'and N more'")
ap.add_argument("--ignore-ext", default="",
help="comma-separated extensions this server does not index,"
" kept out of the verdict (e.g. log,cue,nfo,srt)")
args = ap.parse_args()
db = args.db or KINDS[args.db_kind]["default_db"]
container = ("/data" if args.db_kind == "jellyfin" else "") \
if args.container is None else args.container
ignore = set()
for ext in args.ignore_ext.split(","):
ext = ext.strip().lower()
if ext:
ignore.add(ext if ext.startswith(".") else "." + ext)
moves = read_moves(args.journal)
stored = {to_host(p, container, args.share) for p in server_paths(db, args.db_kind)}
landed, stale, missing, ignored = 0, [], [], 0
for src, dst in moves:
if os.path.splitext(dst)[1].lower() in ignore:
ignored += 1
continue
at_dst, at_src = dst in stored, src in stored
if at_dst and not at_src:
landed += 1
continue
if at_src:
stale.append(src)
if not at_dst:
missing.append((src, dst))
# The database cannot tell a lost file from one it never indexes: both read
# as "no row at the destination". Stat both paths, so the report separates
# the classes the runbook names — `REAL` is the only one that is this pass's
# own damage, and `gone` is a later pass or the server's own rename.
on_disk, real, gone = [], [], []
for src, dst in missing:
if os.path.exists(dst):
on_disk.append(dst)
elif os.path.exists(src):
real.append(src)
else:
gone.append(dst)
print(f"{args.db_kind}: {len(moves)} moves checked")
print(f"landed: {landed}")
print(f"still at src: {len(stale)}")
print(f"missing from the DB: {len(missing)}")
print(f" on disk at dst, no row names it: {len(on_disk)}")
print(f" REAL — file still at src: {len(real)}")
print(f" gone — at neither path: {len(gone)}")
if ignored:
print(f"not indexed (extension ignored): {ignored}")
if missing:
exts = Counter(os.path.splitext(d)[1].lower() or "(no extension)"
for _, d in missing)
print("missing by extension: "
+ ", ".join(f"{e} {n}" for e, n in sorted(exts.items())))
for label, items in (("real", real), ("gone", gone), ("missing", on_disk),
("stale", stale)):
for path in items[:args.show]:
print(f" {label}: {path}")
if len(items) > args.show:
print(f" {label}: … and {len(items) - args.show} more")
return 1 if (stale or missing) else 0
if __name__ == "__main__":
sys.exit(main())
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env bash
# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
#
# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
# ONLY when it sits next to the media file. The media lives on the RAID6 array at
# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
# container, and slow to seek for a thousand tiny XML reads. So neither media
# server binds the array any more: they bind an OverlayFS merge of it with a
# writable layer on the NVMe.
#
# lowerdir /mnt/entropy RAID6 (ROTA=1) — source of truth, never written
# upperdir /var/lib/media-meta/upper NVMe — every sidecar and image byte
# workdir /var/lib/media-meta/work NVMe — kernel bookkeeping
# Where /var/lib/media-meta/view the merged tree jellyfin and plex bind
#
# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
#
# Consequences worth knowing:
# * A sidecar write is a few KB onto the SSD; the media bytes never move and the
# array is still never written by anything automated.
# * Deleting through the view writes a whiteout into the upper layer instead of
# deleting from the array. Nothing should delete through the view: deletions
# go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
# * inotify events on the lower layer do not cross the merge, so a file that
# arrives on the raw array is invisible to a file-system watcher in the view:
# trigger a library refresh (see the Media Relayout page).
#
# Subcommands
# plan [--upper-base D] [--lower D] print the resolved shape, touch nothing
# verify prove the live mount IS that shape
# install [--upper-base D] create the layers, install+enable+start the unit
# reap [--apply] [--upper-base D] [--lower D]
# list upper-layer entries whose media is gone
# status one line per fact a human wants
#
# install must run as root (it writes /etc/systemd/system and mounts); the others
# are read-only and safe unprivileged.
set -euo pipefail
LOWER=/mnt/entropy
UPPER_BASE=/var/lib/media-meta
VIEW=$UPPER_BASE/view
# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
# repo copy keeps a readable name; install writes it under the escaped one.
UNIT='var-lib-media\x2dmeta-view.mount'
UNIT_DIR=/etc/systemd/system
# The rootless containers write as this host uid (the deploy user). The upper
# layer has to accept those writes, so it is owned by them.
MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
die() {
printf 'media-view: %s\n' "$*" >&2
exit 1
}
usage() {
printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
exit 2
}
cmd="${1:-}"
[ -n "$cmd" ] || usage
shift || true
upper_base="$UPPER_BASE"
lower="$LOWER"
apply=0
while [ $# -gt 0 ]; do
case "$1" in
--upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
--lower) [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
--apply) apply=1; shift ;;
*) usage ;;
esac
done
upper="$upper_base/upper"
work="$upper_base/work"
# The device backing a path, and whether the kernel calls it rotational.
rotational_of() {
local src rota
src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
[ -n "$src" ] || die "cannot find a block device behind $1"
rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
[ -n "$rota" ] || die "cannot read the rotational flag of $src"
printf '%s %s\n' "$src" "$rota"
}
require_upper_base() {
[ -d "$upper_base" ] \
|| die "refused: upper base $upper_base does not exist (put it on the SSD first)"
}
do_plan() {
printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
"$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
}
do_verify() {
local fail=0 fst opts src rota
if ! mountpoint -q "$VIEW"; then
printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
printf ' library that scans itself empty, not a metadata view.\n' >&2
return 1
fi
fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
if [ "$fst" != overlay ]; then
printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
fail=1
fi
opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
local want
for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
case ",$opts," in
*",${want},"*) ;;
*) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
fail=1 ;;
esac
done
read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
if [ "$rota" != 0 ]; then
printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
printf ' belongs on the SSD, not on the array.\n' >&2
fail=1
fi
if [ ! -w "$UPPER_BASE/upper" ]; then
printf 'refused: %s is not writable\n' "$upper" >&2
fail=1
fi
[ "$fail" -eq 0 ] || return 1
printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
}
do_install() {
[ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
require_upper_base
mountpoint -q "$LOWER" \
|| die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
local src rota
read -r src rota <<<"$(rotational_of "$upper_base")"
[ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
[ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
# systemd will not create a mount point; a missing directory fails the start.
install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
install -d -m 0700 "$upper_base/.AGENTTRASH"
install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
systemctl daemon-reload
systemctl enable "$UNIT" >/dev/null 2>&1 || true
systemctl start "$UNIT"
do_verify
}
# An upper-layer DIRECTORY with no counterpart below is metadata for a media
# folder that has been moved or renamed on the RAW path — a ghost the media
# servers would still browse. Files are never candidates on their own: a sidecar
# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
# is for. A copy-up whose lower file was deleted by hand is the one case this
# leaves in place; the raw path is where deletions belong, so it stays visible.
do_reap() {
[ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
count=0
ghosts=0
reap_walk ""
if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
printf 'reap: no ghosts under %s\n' "$upper"
return 0
fi
if [ "$ghosts" -gt 0 ]; then
printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
"$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
printf ' Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
printf ' Deletions belong in .AGENTTRASH on the raw path.\n' >&2
fi
if [ "$count" -eq 0 ]; then return 0; fi
if [ "$apply" -eq 1 ]; then
printf 'reap: moved %d entr%s to %s\n' "$count" \
"$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
else
printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
"$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
fi
}
# Descend only where the lower layer still has the directory: a directory that is
# missing below is itself the ghost, and its whole subtree goes with it.
reap_walk() {
local rel="$1" entries abs r name
entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
[ -n "$entries" ] || return 0
while IFS= read -r abs; do
r="${abs#"$upper"/}"
name="${abs##*/}"
if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
ghosts=$((ghosts + 1))
continue
fi
if [ -e "$lower/$r" ]; then
[ -d "$abs" ] && reap_walk "$r"
continue
fi
[ -d "$abs" ] || continue
count=$((count + 1))
printf '%s\n' "$r"
if [ "$apply" -eq 1 ]; then
mkdir -p "$stamp/$(dirname "$r")"
mv "$abs" "$stamp/$r"
fi
done <<<"$entries"
}
do_status() {
if mountpoint -q "$VIEW"; then
printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
else
printf 'view: %s NOT MOUNTED\n' "$VIEW"
fi
if [ -d "$upper" ]; then
printf 'upper: %s %s files %s\n' "$upper" \
"$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
else
printf 'upper: %s missing\n' "$upper"
fi
systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
}
case "$cmd" in
plan) do_plan ;;
verify) do_verify ;;
install) do_install ;;
reap) do_reap ;;
status) do_status ;;
*) usage ;;
esac
+27 -17
View File
@@ -1,5 +1,6 @@
# open-notebook.env — configuration for **open-notebook**, the fleet's podcast
# generator on Dankclaw (UI host port **8502**, API host port **5056**). Read by
# generator on Dankclaw (user path https://notebook.franksdank.cloud; UI host
# port **8502**, API host port **5056 loopback-only**). Read by
# BOTH Quadlet units (dankclaw/open-notebook/quadlet/) as the FIRST
# EnvironmentFile= line, which is why every path and host name below is the
# value the CONTAINER must see.
@@ -47,13 +48,17 @@
#
# THE SCRIPTWRITER. ESPERANTO_LLM_TIMEOUT is the app's own default (180 s), kept
# visible beside the TTS number because both belong to one podcast run: the
# outline call and the transcript call are the slow text work, and frank-zealot
# answers them well inside 180. Provider choice is made in the UI, not here —
# see [[Open Notebook]] for the one-time credential provisioning — and it is
# frank-zealot on purpose: script writing asks for a JSON-schema response, which
# the OpenAI-compatible llama.cpp endpoint is known to serve, while halogen's
# structured-output behaviour on frank-dragoon is UNVERIFIED. halogen remains
# the unproven alternative, not the fallback.
# outline call and the transcript call are the slow text work, and the endpoint
# provisioned today answers them well inside 180. The provider itself is chosen
# in the database, not here — [[Open Notebook]] owns that provisioning and the
# address census behind it. Short version: frank-zealot was the plan (script
# writing asks for a JSON-schema response, which llama.cpp serves and halogen's
# behaviour on frank-dragoon is UNVERIFIED), but no Dankclaw LLM is dialable from
# this container's netns, and the single address that reaches one —
# `host.containers.internal`, 169.254.1.2 — the app refuses to store at all,
# because link-local is a blocked address class. The writer is therefore
# frank-archon (`http://192.168.0.235:7770/v1`), the only fleet LLM that both
# answers from inside this netns and passes that check.
#
# THE DATABASE COORDINATES. SURREAL_URL addresses the sibling container BY NAME
# on systemd-open-notebook (container DNS is scoped to the owning network, so
@@ -73,15 +78,20 @@ SURREAL_PASS=
OPEN_NOTEBOOK_ENCRYPTION_KEY=
OPEN_NOTEBOOK_PASSWORD=
# WHERE THE BROWSER FINDS THE API. The UI auto-detects `<host-you-opened>:5055`,
# and 5055 on this host belongs to seerr, so the API leaves the container on 5056
# and this line has to say so. Both origins are listed in CORS_ORIGINS because
# the API is called cross-origin from the UI's own port: the LAN origin is the
# normal one, the localhost one covers a browser opened on Dankclaw itself. The
# app logs a warning at start while CORS_ORIGINS is unset and defaults to `*`,
# which is a wildcard in front of an endpoint that serves every episode's audio.
API_URL=http://192.168.0.143:5056
CORS_ORIGINS=http://192.168.0.143:8502,http://localhost:8502
# WHERE THE BROWSER FINDS THE API. The guarded vhost
# (notebook.franksdank.cloud → Caddy → 8502) is the only user path, and the
# image's own Next.js layer rewrites `/api/*` to the FastAPI running beside it,
# so ONE origin serves UI and API and API_URL has to name that vhost. This is
# not cosmetic: unset, the app auto-detects `<host-you-opened>:5055` — the port
# upstream ships, and the one seerr holds on this host — and hands the browser
# an address nothing answers; pointed at the LAN `:5056` instead, every call
# leaves the guarded origin and an authenticated browser session does not ride
# it. Same-origin traffic needs no CORS at all, but CORS_ORIGINS stays set to
# the one origin because the app warns at start while it is unset and defaults
# to `*` — a wildcard in front of an endpoint that serves every episode's
# audio. No trailing slash: the UI concatenates `${API_URL}/api/...`.
API_URL=https://notebook.franksdank.cloud
CORS_ORIGINS=https://notebook.franksdank.cloud
# Concurrency: see "THE CPU-SHUTDOWN GUARD" above. Defaults are 5 and 5.
TTS_BATCH_SIZE=1
+16 -5
View File
@@ -147,10 +147,21 @@ def main():
deep = corpus(160_000)
wide = corpus(1_100_000)
# 3.28 chars/token is the measured ratio of this corpus through the
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
# deploy); 98.5 % of the window leaves the response room under the ceiling.
chars = int(0.985 * maxlen * 3.28)
# The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
# (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
# 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
# stale constant sends leg B past --max-model-len, where the engine answers a
# perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
# context size (200000)") and the leg reads as a capacity failure. One small
# request prices the ratio; 97 % of the window leaves the response room.
sample = prompt(wide, 4000, "CAL", "cal-marker")
cal, _, _ = post({"model": MODEL,
"messages": [{"role": "user", "content": sample}],
"max_tokens": 1, "temperature": 0.0,
"chat_template_kwargs": {"enable_thinking": False}})
ratio = len(sample) / cal["usage"]["prompt_tokens"]
print(f"[cal] corpus ratio {ratio:.2f} chars/token")
chars = int(0.97 * maxlen * ratio)
# Each marker's FIRST hyphen-separated word is unique, because show()
# scores the hit on that word — a shared prefix would score every slot HIT
# on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28)
dchars = int(0.25 * maxlen * ratio)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, "
+335
View File
@@ -0,0 +1,335 @@
#!/bin/sh
# Deterministically empty every .AGENTTRASH directory under the given roots.
#
# PURPOSE
# The fleet's deletion policy parks deleted files in .AGENTTRASH/ — in-repo
# beside the item, elsewhere in $HOME as .AGENTTRASH/YYYYMMDD-topic — and
# waits for approval to purge. This script is that purge: given a machine,
# it clears .AGENTTRASH directories and nothing else.
#
# CONTRACT
# Default dry-run: the plan prints and nothing is deleted; --execute deletes.
# Every line the script can ever print for a candidate is decided at plan time
# and stored in the plan table, so a dry run prints exactly what --execute
# will print. A .AGENTTRASH that is a root's own trash (a repo's
# .AGENTTRASH, a machine's ~/.AGENTTRASH) prints `plan empty <path>`: the
# directory is the convention and survives, its staged contents are cleared.
# A .AGENTTRASH deeper than that is content and prints `plan remove <path>`
# plus, once executed, `removed <path>`; its own nested .AGENTTRASH entries
# belong to that tree and are never planned again. Candidates are found with
# `find -xdev` per root, so a mounted tree can never be walked by accident;
# ordering is lexicographic (LC_ALL=C). `/`, symlinked, and foreign-owned
# candidates are refused and reported (exit 1), never touched.
# A summary line `removed=<candidates> deleted=<removed> refused=<n>` closes
# every run and one dated line is appended to the log.
# Exit: 0 success, 1 operational failure, 2 usage error. A second run over
# the same tree plans nothing further and exits 0.
#
# PORTABILITY
# POSIX sh, clean under `shellcheck -S style`. Externals: find (must support
# -delete), stat, rm, date, mkdir. No bashisms, no GNU-only flags, no arrays,
# no `local`. Paths containing whitespace are rejected at parse time — the
# same rule applied to roots and candidates — so the plan can be walked with
# plain word splitting and nothing can be silently mis-split.
#
# USAGE
# purge-agenttrash [--execute] [--quiet] [--log PATH] [PATH...]
# --execute delete; without it the run is a plan and deletes nothing
# --quiet suppress per-line output; errors and the summary still print
# --log PATH log destination (default: $HOME/.AGENTTRASH-cleanup.log)
# PATH... roots to search (default: $HOME)
#
# Everything else is fixed by design: the root default, the ordering, the
# guards. There is no configuration file and no environment override.
set -eu
usage() {
sed -n '3,44p' "$0" | sed 's/^# \{0,1\}//'
}
die() {
printf '%s: %s\n' 'purge-agenttrash' "$1" >&2
exit "${2:-1}"
}
# ------------------------------------------------------------------ arg parser
# Roots accumulate as one space-separated string: a root containing whitespace is
# rejected at parse time rather than silently split later. That leaves the plan
# iteratable with plain `$@`-free word splitting, the only portable construct
# that cannot corrupt a path it has already vetted.
MODE=plan
QUIET=0
LOGPATH=
ROOTS=
while [ "$#" -gt 0 ]; do
case "$1" in
--help|-h)
usage
exit 0
;;
--execute)
MODE=execute
;;
--quiet)
QUIET=1
;;
--log)
[ "$#" -ge 2 ] || die '--log needs a value' 2
LOGPATH=$2
shift
;;
--log=*)
LOGPATH=${1#--log=}
;;
-*)
die "unknown option: $1" 2
;;
*)
case $1 in
*' '*) die "root contains whitespace: $1" 2 ;;
esac
if [ -z "$ROOTS" ]; then
ROOTS=$1
else
ROOTS="$ROOTS $1"
fi
;;
esac
shift
done
[ -n "${HOME:-}" ] || die 'HOME is not set; pass roots explicitly or set HOME' 2
[ -n "$LOGPATH" ] || LOGPATH=$HOME/.AGENTTRASH-cleanup.log
# ------------------------------------------------------------ tool availability
require() {
command -v "$1" >/dev/null 2>&1 || die "required tool not found: $1" 1
}
require find
require stat
require rm
require date
require mkdir
# `find -delete` backs the in-place emptying. Probe it rather than trusting
# `find --version`: a shell function named `find` must not be able to fake it.
if ! find . -maxdepth 0 -name '.purge-probe-does-not-exist' -delete >/dev/null 2>&1; then
die 'find does not support -delete; refusing to guess at deletion' 1
fi
UID_SELF=$(id -u)
# ------------------------------------------------------------------- locking
# The lock dir also hosts this run's plan table, so both live and die together:
# the exit trap removes the whole directory and only the log survives the run.
LOCKDIR=$HOME/.agenttrash-purge.lock
PLANFILE=$LOCKDIR/plan
acquire_lock() {
if ! mkdir "$LOCKDIR" 2>/dev/null; then
held_pid=$(cat "$LOCKDIR/pid" 2>/dev/null || printf '%s\n' '')
if [ -n "$held_pid" ] && kill -0 "$held_pid" 2>/dev/null; then
die "another purge holds $LOCKDIR (pid $held_pid)" 1
fi
# Owner died mid-run: drop its pid, then retry exactly once. An
# unremovable/nonempty lock dir is not ours to reclaim.
rm -f "$LOCKDIR/pid" 2>/dev/null || true
rmdir "$LOCKDIR" 2>/dev/null || die "$LOCKDIR exists and is not ours to reclaim" 1
mkdir "$LOCKDIR" 2>/dev/null || die "cannot reacquire $LOCKDIR" 1
fi
printf '%s\n' "$$" >"$LOCKDIR/pid" || die "cannot stamp $LOCKDIR/pid" 1
: >"$PLANFILE" || die "cannot write $PLANFILE" 1
trap 'rm -rf "$LOCKDIR"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
}
# ------------------------------------------------------------------ predicates
is_dir() { # true only for a real directory, never a symlink to one
[ -d "$1" ] && [ ! -L "$1" ]
}
is_ours() { # true only when the invoking user owns the path
owner=$(stat -c '%u' "$1" 2>/dev/null) || return 1
[ "$owner" = "$UID_SELF" ]
}
refuse() { # refuse <path> <reason> — reported, never deleted
printf '%s: refused %s (%s)\n' 'purge-agenttrash' "$1" "$2" >&2
REFUSED=$((REFUSED + 1))
}
emit() { # emit <line> — silenced by --quiet
[ "$QUIET" -eq 1 ] || printf '%s\n' "$1"
}
# --------------------------------------------------------------- the plan table
# One row per candidate: `<action> <path> <line>`. The first two fields tell
# `act_on` what to do; the rest of the row is EXACTLY what the run prints for
# that candidate — in plan mode verbatim, in execute mode as the first line of
# its report. The table is a file, so no shell quoting rule (quoted-for,
# heredoc, IFS games) can corrupt an approved plan, and the two modes can
# never drift apart. A removal's second line (`removed <path>`) is derived from
# the row's action, so it is also fixed at plan time.
CANDIDATES=0
REFUSED=0
record() { # record <action> <path> <line> — append one row to the plan table
printf '%s %s %s\n' "$1" "$2" "$3" >>"$PLANFILE"
CANDIDATES=$((CANDIDATES + 1))
}
# ------------------------------------------------------------- candidate rule
# The whole contract in one place:
# `/` is refused outright (never a candidate, never walked);
# a root that is a symlink or a plain file is refused, never followed;
# a root (or any path) that IS a real .AGENTTRASH prints `plan empty <path>`:
# the convention persists, its staged contents are what gets cleared;
# otherwise the root's DIRECT .AGENTTRASH child is also `empty` — it is that
# level's own trash ($HOME/.AGENTTRASH, x1/.AGENTTRASH, repo-a/.AGENTTRASH);
# every .AGENTTRASH deeper than that is content: `remove`, printed as
# `plan remove <path>` (+ `removed <path>` when executed), unless it lives
# inside a tree already claimed (then it dies with that tree, never planned
# twice) or is foreign-owned/symlinked (then refused).
collect_root() { # collect_root <path> -> 0 ok, 1 refused
target=$1
if [ "$target" = / ]; then
refuse / 'root refusal'
return 1
fi
if [ -L "$target" ] || ! [ -d "$target" ]; then
refuse "$target" 'not a real directory'
return 1
fi
if ! is_ours "$target"; then
refuse "$target" "not owned by uid $UID_SELF"
return 1
fi
if [ "${target##*/}" = .AGENTTRASH ]; then
record empty "$target" "plan empty $target"
return 0
fi
# `own` is the root's own .AGENTTRASH once handled; hits below it belong to
# that tree and must never be planned a second time.
own=
if [ -e "$target/.AGENTTRASH" ]; then
if ! is_ours "$target/.AGENTTRASH"; then
refuse "$target/.AGENTTRASH" "not owned by uid $UID_SELF"
elif ! is_dir "$target/.AGENTTRASH"; then
refuse "$target/.AGENTTRASH" 'not a real directory'
else
record empty "$target/.AGENTTRASH" "plan empty $target/.AGENTTRASH"
own=$target/.AGENTTRASH
fi
fi
# No `-type d`: implementations disagree on whether that matches a
# symlink-to-directory (GNU find says no, others yes), so every entry named
# .AGENTTRASH is collected and classified here instead of by `find`.
if ! found=$(find "$target" -xdev -name .AGENTTRASH 2>/dev/null); then
refuse "$target" 'find failed'
return 1
fi
previous=
for entry in $found; do
[ -n "$entry" ] || continue
if [ -n "$own" ]; then
case $entry in
"$own"|"$own"/*) continue ;; # already emptied in place
esac
fi
if [ -n "$previous" ]; then
case $entry in
"$previous"/*) continue ;; # nested: dies with its parent
esac
fi
previous=$entry
if ! is_ours "$entry"; then
refuse "$entry" "not owned by uid $UID_SELF"
continue
fi
if ! is_dir "$entry"; then
refuse "$entry" 'not a real directory'
continue
fi
record remove "$entry" "plan remove $entry"
done
return 0
}
# ------------------------------------------------------------------ execution
remove_tree() { # rm -rf a real directory; symlinks inside die as entries
is_dir "$1" || return 1
rm -rf -- "$1"
}
empty_dir() { # empty a real directory in place, never following symlinks
is_dir "$1" || return 1
find "$1" -mindepth 1 -xdev -delete
}
# act_on <action> <path> <line> -> 0 when the action succeeded.
# remove: prints the stored line (`plan remove <path>`) then `removed <path>`.
# empty: prints the stored line (`plan empty <path>`) and nothing else — the
# directory survives, so the plan line IS the outcome.
act_on() { # act_on <action> <path> <line> -> 0 when the action succeeded
if [ "$1" = remove ]; then
if remove_tree "$2" 2>/dev/null; then
emit "$3"
emit "removed $2"
return 0
fi
printf '%s: failed to remove %s\n' 'purge-agenttrash' "$2" >&2
return 1
fi
if empty_dir "$2" 2>/dev/null; then
emit "$3"
return 0
fi
printf '%s: failed to empty %s\n' 'purge-agenttrash' "$2" >&2
return 1
}
# Walk the plan table once: plan mode prints every row's line verbatim, execute
# mode acts on each row and counts failures separately from plan-time refusals.
walk_plan() {
while read -r action path line; do
[ -n "$action" ] || continue
if [ "$MODE" = plan ]; then
emit "$line"
elif act_on "$action" "$path" "$line"; then
REMOVED=$((REMOVED + 1))
else
EXEC_REFUSED=$((EXEC_REFUSED + 1))
fi
done <"$PLANFILE"
}
# ----------------------------------------------------------------------- main
[ -n "$ROOTS" ] || ROOTS=$HOME
acquire_lock
for root in $ROOTS; do
collect_root "$root" || true
done
REMOVED=0
EXEC_REFUSED=0
walk_plan
STATUS=0
[ "$REFUSED" -eq 0 ] || STATUS=1
[ "$EXEC_REFUSED" -eq 0 ] || STATUS=1
printf 'removed=%s deleted=%s refused=%s\n' \
"$CANDIDATES" "$REMOVED" "$((REFUSED + EXEC_REFUSED))"
if ! printf '%s\tplan\tcandidates=%s\tremoved=%s\trefused=%s\n' \
"$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$CANDIDATES" "$REMOVED" \
"$((REFUSED + EXEC_REFUSED))" >>"$LOGPATH" 2>/dev/null; then
printf '%s: cannot write log %s\n' 'purge-agenttrash' "$LOGPATH" >&2
STATUS=1
fi
exit "$STATUS"
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
#
# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
# first, which is how a 16-hour phase gets paid for twice.
#
# Every phase is gated on the previous one, because the last time this ran unguarded
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
#
# 1. the runner must still be alive and its container must answer /health AND own the
# listening socket, or this script kills it, restores the service and exits non-zero —
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
# first, the dying process keeps its listener for a moment, and a rig container that
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
# then the pusher refused on the container name and the window died at second zero);
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
# marker names this container, and the container is running (scripts/calib-push.py
# --tap-marker), so a second lock survives a mistake in this file;
# 3. the reference container is banked into trash only when BOTH the serve and the push
# exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
#
# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
# hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
# (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
# Usage: scripts/qwen38-calib-window.sh [--dry-run]
set -uo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
DRY=0
[ "${1:-}" = "--dry-run" ] && DRY=1
say() { printf '%s %s\n' "$(date -Is)" "$*"; }
MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
RIG=qwen38-bf16
SERVICE=frank-archon
PY="${PY:-python3}"
# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
# The checks that used to live here were proxies, and each lied at least once: the space check
# asked a directory that merely LOOKED related, `/health` answered for a dying production
# process, `podman inspect` named a container that existed but had bound nothing. The shared
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
# keeps an existing tap, so the tap-empty fact is waived.
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
if [ "$DRY" = 1 ]; then
echo "dry run: calib window — the service is left running, nothing is mounted or written"
echo "would run: bash scripts/run-qwen38-bf16.sh calib (rig boots, writes $CALIB/.rig-boot)"
echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/ (only if both phases exit 0)"
exit 0
fi
say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
# ---- phase 1: the rig -------------------------------------------------------
say "phase=calib serve begin"
bash scripts/run-qwen38-bf16.sh calib & spid=$!
rig_up() {
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
# dying production process keeps its listener for a moment while the freshly created
# rig container is still loading — so `container running` + `/health` 200 both hold
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
# pusher refused on the container name and the window died at second zero. Ask the kernel
# who owns the listening socket and require it to be a rig process.
local lpid
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
[ -n "${lpid:-}" ] || return 1
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
}
waited=0
until rig_up; do
if ! kill -0 "$spid" 2>/dev/null; then
wait "$spid"; crc=$?
say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
exit 1
fi
if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
podman stop "$RIG" >/dev/null 2>&1 || true
wait "$spid"
echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
exit 1
fi
sleep 15
waited=$((waited + 15))
done
say "phase=calib health ok waited=${waited}s"
# ---- phase 2: the tap -------------------------------------------------------
say "phase=calib push begin"
"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
--tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
prc=$?
say "phase=calib push end rc=$prc"
podman stop "$RIG" >/dev/null 2>&1 || true
wait "$spid"; crc=$?
say "phase=calib serve end rc=$crc"
if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
exit 1
fi
[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
# ---- phase 3: bank the reference container ----------------------------------
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
else
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
fi
say "calib window end"
Loaded 100 of 200 files, more files were not shown because too many files have changed in this diff. Show more