An apply leaves every release folder it emptied standing where it was: the
live library run left 1675 husks at the root. sweep takes the same journal
and deletes only folders it can prove this run emptied - a journalled move
out of the folder or something under it, nothing but empty directories left,
no journalled destination beneath it, not a folder the run created. A folder
nobody moved cannot be selected at all, so a hand-made placeholder survives,
and a symlink anywhere inside disqualifies the subtree. Every rmdir is
journalled, so rollback recreates what a sweep removed.
The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.
* media-census.py the read-only inputs: a stat pass, an ffprobe header pass
(nice'd, so it is safe beside a live endpoint), and the
Plex metadata cache read that outranks any filename guess
* media-plan.py the Jellyfin-format planner: one folder per movie, the
video named exactly like the folder, extras in the
canonical subfolders, copies of one film as versions. It
opens no media file, so a whole-library plan costs seconds,
and it ends in gates — bad-names, occupied, double-claim,
left-unmoved, unplanned — that must all read zero before a
human points the executor at 24 TB
* media-apply.py the executor: rename(2) only, the intent fsynced to a
write-ahead journal BEFORE the move it authorises, so an
interrupted run is finishable and every run is undoable.
Refusals are all-or-nothing and it never overwrites a file
Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):
* a Plex year of 0 means "indexed, never matched" — trusting it produced
folders "Title (0)": items with no year, no poster, no identity
* a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
tree neither matcher reads as a movie
* a folder inside a movie's folder is bonus territory even when its name is
misspelled ("Exrtas"), unless the file's own name carries another year
* matching words split on everything but a letter or digit — the earlier
tokenizer left "(1960)/extras" glued, so a film could not be found by the
folder holding it and one box set's featurettes drifted into a neighbour
* a name with no words in it is bonus material, never a movie folder named 1
Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.
A by-digest pull leaves the image carrying every digest it was pulled under, so
reading RepoDigests[0] reported a healthy pinned image as wrong and refused a
window that was ready to run. Membership, not position. Also records the measured
step optimum: 106 tok/s at an 8,192-token step against 76 at 16,384 on a 32,768
calibration window, so the wide step has a ceiling, not a plateau.
Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid
scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.
The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.
Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
The law today's authentik bump paid for: start on an active unit is a no-op, so
the new digest installs and the old container keeps serving behind three green
gates. Restart applies a pin.
The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
Frank's rule: the served iq4r this arm competes with was calibrated on a
10M-token set, so the arm is calibrated at least as deeply. The floor was
lowered to 1.5M on the vendor's '10M beats 1M by under 1%' note; that
measures the vendor's own plate, not ours — the arm is judged against a
10M-calibrated reference, so cutting the calibration 6x to save wall clock
buys a worse position on the only comparison that matters.
One round of the re-chunked corpus is ~9.31M tokens, so a 12M floor seals
in round 1 plus part of round 2. A round that seals nothing is normal (the
floor is pooled) and round_verdict treats it as continue, never failure.
Also fixes tests/qwen38_campaign.bats' window dry run: its fixture lands on
/tmp (a 16 GiB tmpfs) where the 60 GiB seal preflight can never pass, so the
test now uses the documented Q38_CALIB_MIN_FREE_MIB override instead of
asserting environment luck.
frank-dragoon: engine 0.18.20 (image e35133b2…), the 18-model NPU catalog, the
re-cut qwen3-embedding-0.6b device programs, and the wire surface the new engine
actually answers (GET /tokenize is 405, /props is 200, /completion and /ping
stay 404). Eras gains the 2026-10-30 row.
frank-zealot: fork build 12118 (0.7.0-dev, 024474448) with the payload-load
evidence (202 pq2_0 tensors, 202 Hadamard-folded weights), plus the law a
rebuild pays for: cmake --install strips the rpath and installs bin/+lib/, so
the configure needs CMAKE_INSTALL_RPATH and LLAMA_BIN names …/bin/llama-server.
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.
LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.
The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.
Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.
0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.
The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).
Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.
The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
Nine pages under LLM/ document the upstream radiance engine at 8da043c:
- engine shape, plugin model, RAD2 container, ABI and environment
- every serve flag grouped with defaults and the law behind each knob
- the HTTP surface, request fields and refusal classes
- rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring
- the (arch id, quant) family table and the recipe of record per model
- kernel/quantiser plugin authoring, encodings, build and checks
- architecture plugin authoring: declare, step, probe, lane and media rules
- rad-info/radbench/radtune and the shell harnesses
- the container contract, host prerequisites, readiness and restart triage
Also completes the dangling Related line on the frank-archon endpoint page.
Container images re-resolved against their registries today; each pin is the
manifest-list digest the tag answers now, so no container is served from a
build that upstream has already replaced.
Version-tag moves: authentik 2026.8.3 -> 2026.8.5 (server/worker/outpost and
the ldap image, which now carries the tag instead of a bare digest so the pair
can never drift apart), seerr v3.4.1 -> v3.5.1, calibre 9.15.0 -> 9.17.0,
calibre-web 0.6.27-ls402 -> ls407, plex ls329 -> ls330, tautulli
v2.18.1-ls245 -> v2.18.2-ls250.
Digest-only moves inside the same stream: postgres 16-alpine, mariadb 11,
redis 7-alpine, valkey 9-alpine, grafana, alertmanager, gluetun, searxng,
open-webui, jellyfin, romm, chromedp/headless-shell, lightpanda/browser.
Rollback is the previous digest on the same Image= line.
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.
Four gates, each with the measurement that motivated it:
- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
@sha256:665775ed…), overridable only by naming --image deliberately. Proven by
the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
starts the service. `podman rm -f` does not kill the engine: an orphan made
production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
target configurable, so the tap lives on /mnt/entropy instead of the root fs.
Measured from the engine's own counters on the 2026-10-29 drain: prefill 71 tok/s steady state, a
180 s window showing 7 prefill steps of the capped 2,048 tokens and 3 preemptions. One 7,168-token
document is therefore ~100 s, round 1 (2,367 docs, 10.03M tokens) ~39 h, and the 12M-position
floor ~47 h of drained production — not the ~18 h the brief assumed.
The tap is the only phase whose cost is read volume per document: at the inherited 2,048-token step
a document is four steps, and each step re-streams every layer's routed expert union from the
container (43.4 GiB a rank live on disk). One wide step reads each layer's union once.
The 'one step, one slot count' audit law binds kldref and score because their KL numbers are only
comparable at an identical shape. Calib produces layer Hessians and no number any report compares,
so Q38_CALIB_MNBT / Q38_CALIB_SEQS are its own, defaulting to the inherited value — nothing moves
unless a probe asks. The KL modes ignore both.
Tests: 589/0 — the knob reaches calib's argv, does not reach kldref's, and a non-numeric value is
refused by name.
RADIANCE_CALIB_TOKENS=12000000 is a pooled floor and calib-7168.jsonl carries 10.03M tokens, so
round 1 of a corpus push seals nothing by construction. The gate PR #84 added failed exactly
there, one round too early, while the drain was already running.
Now round_verdict() separates the cases: 'shrink' (the tap lost bytes) fails the run, 'flat' says
so and continues, 'grew' continues, 'done' stops. A run that never sees the engine's completion
line still exits non-zero on max rounds, and the rig marker remains the lock that stops a push
into anything that is not the rig.
Test: round_verdict's four verdicts, behaviorally.
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.
- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
and that container is running. With the driver's existing tap-progress gate, a push can no
longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
instead of falling through the health wait, bank the 336 GiB reference container into trash
only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
section that names the live campaign instead of the retired swift15 era.
Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.
Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.
PR #81's pool half measured inert the day it deployed: the 66,560 boot printed the
same elastic 26.73 GiB / experts 20.72 / kv cache 5.64 / vram_weights 24.30 as the
49,152 boot and opened the pool with 0 B used, and the qwen38-w5nl64 rig moved
vram_kv 269.99 -> 271.25 MiB (window still 7,436 tokens: its 95.37 GiB n-gram table
claim decides, not the pool). The step half stays. Tests: 576 ok / 0 fail.
The container opened the raised pool with 0 B used and printed the same elastic 26.73 GiB / experts 20.72 GiB / kv cache 5.64 GiB / vram_weights 24.30 GiB as the 49,152 boot, so the era's superseded ledger line is destroyed and the evidence table gains the ceiling-case row beside the 45,568 one it already carried. Tests: ./tests/run.sh 576 ok / 0 fail.
frank-archon.env: HOST_POOL_MIB 49,152 -> 66,560 a rank (tp x 65 GiB against the
246 GiB the box has), and the TTM pair 162,000/204,800 -> 197,000/224,000 — the
floor the bats ledger formula computes at the new pool. The pool is the KV lever
under the budget resolver: every MiB pinned in host RAM lowers the expert floor on
the card one-for-one and the cache takes it.
scripts/run-qwen38-bf16.sh: the campaign's private FRANK_QWEN38_MNBT (1,024) is
gone — MNBT now comes from FRANK_ARCHON_MAX_BATCHED_TOKENS, production's 2,048 and
the widest step the quantised expert file tiers stream. That 1,024 was PR #80's
answer to a starved cache; the pool is the fix, a narrow step was the symptom's
disguise, and a per-mode literal measures a shape nothing serves.
tests: frank_archon_unit greps the new pool value; qwen38_reference's SHAPE fixture
carries it, and the two MNBT-override tests become the inheritance assertion (every
mode's argv quotes the value the env file owns) plus the calib assertions.
vault: the shape block moves to the new numbers and the Eras table gains the row for
the era this replaces. The 2026-10-20 boot-ledger line stays as the old era's record
until the new boot's ledger replaces it.
dank
Frank
dank