The handler called `systemctl --user start` on all three units. Start is a
no-op for an already-active unit, so installing a new digest under an active
authentik left the old container serving: today's 2026.8.3 -> 2026.8.5 bump
moved only the OUTPOST (whose handler restarts) and left server + worker on
2026.8.3 while the deploy, the health gate and the e2e suite were all green.
The generator re-renders from the .container, so a restart is the step that
applies a pin.
Rebuilt the fork from origin/prism at 024474448 with the same configure the
10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja),
installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds:
libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base
still registers PQ2_0, so type 142 and the hadamard metadata still load.
LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH
($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the
binary otherwise dies on libllama-server-impl.so.
The unit test that pinned the 10777 path now asserts the invariant its own
comment described (LLAMA_BIN names a llama-prism-* build) instead of a
directory one rebuild happens to install; the suite runs on a box with no
/home/dank, so existence is not checkable there.
Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
A version bump re-cuts the device programs, so a staged file of a different
size is stale rather than partial. `curl -C -` against it appended the new
tail to the old bytes and produced the right BYTE COUNT with the wrong
content — the sha256 gate caught it (consts.hnpb, 2026-10-30), and the
failure reads as a corrupt program rather than a fetch fault. Resume only
when the staged file is smaller than the manifest expects.
The engine pin moves to the newest vendor release (0.18.20, image built
2026-10-30). The payload lock is the gate this deploy has to pass: the v2
.hgn loads only in the flash-server line, so /health plus the boot ledger
decide, and the rollback is the 0c61bf84 digest on the same env line.
0.18.20's NPU catalog is NOT the 0.16.2 one: it names 14 device programs
(u0..u13) and an 8,495,512 B consts.hnpb where 0.16.2 had 10 and 5,086,496 B,
and the entrypoint verifies size+sha256 for every staged file before it
starts. HALOGEN_NPU_MODELS names qwen3-embedding-0.6b, so the staging set is
re-pinned to revision 8fd44c9f with the new block — bumping the engine alone
would refuse the boot on the NPU lane.
The tap did not arm on the production pin: a 17-hour calib run on 1.8.1
wrote zero bytes while /health, /stats and the push all read healthy.
On :latest (v1.14.0, revision 4a61d7296bc5) the same boot prints
'recorded passes: off, the program accumulates calibration Grams
(gram_accum)' and a 9,129-token push sealed 24,768 gram files (42.9 GB)
in the fitting shape - 4 slots at a 2,048-token step, production's own.
The vendor's calibration shape (--max-num-batched-tokens 65536,
--max-num-seqs 64, ~930 tok/s) cannot boot two 32 GiB R9700s: the
activation arena alone is 12.83 GiB of the 27.95 GiB claimable, and
every variant stops in the VRAM resolver (short 3.19 GiB .. 86 MiB,
then a starved cache).
Floor moves 12M -> 1.5M tokens: the vendor's own measurement is 'a
pooled Gram from 10M tokens beat one from 1M by under 1%', and 1.5M is
hours at this rig's rate instead of the 29 h a 6M floor cost.
The converter moves to the same digest: rad-convert's gptq:calib=<dir>
input is version-stable, so a latest-written tap converts on latest.
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.
Four gates, each with the measurement that motivated it:
- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
@sha256:665775ed…), overridable only by naming --image deliberately. Proven by
the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
starts the service. `podman rm -f` does not kill the engine: an orphan made
production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
target configurable, so the tap lives on /mnt/entropy instead of the root fs.
Measured from the engine's own counters on the 2026-10-29 drain: prefill 71 tok/s steady state, a
180 s window showing 7 prefill steps of the capped 2,048 tokens and 3 preemptions. One 7,168-token
document is therefore ~100 s, round 1 (2,367 docs, 10.03M tokens) ~39 h, and the 12M-position
floor ~47 h of drained production — not the ~18 h the brief assumed.
The tap is the only phase whose cost is read volume per document: at the inherited 2,048-token step
a document is four steps, and each step re-streams every layer's routed expert union from the
container (43.4 GiB a rank live on disk). One wide step reads each layer's union once.
The 'one step, one slot count' audit law binds kldref and score because their KL numbers are only
comparable at an identical shape. Calib produces layer Hessians and no number any report compares,
so Q38_CALIB_MNBT / Q38_CALIB_SEQS are its own, defaulting to the inherited value — nothing moves
unless a probe asks. The KL modes ignore both.
Tests: 589/0 — the knob reaches calib's argv, does not reach kldref's, and a non-numeric value is
refused by name.
RADIANCE_CALIB_TOKENS=12000000 is a pooled floor and calib-7168.jsonl carries 10.03M tokens, so
round 1 of a corpus push seals nothing by construction. The gate PR #84 added failed exactly
there, one round too early, while the drain was already running.
Now round_verdict() separates the cases: 'shrink' (the tap lost bytes) fails the run, 'flat' says
so and continues, 'grew' continues, 'done' stops. A run that never sees the engine's completion
line still exits non-zero on max rounds, and the rig marker remains the lock that stops a push
into anything that is not the rig.
Test: round_verdict's four verdicts, behaviorally.
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.
- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
and that container is running. With the driver's existing tap-progress gate, a push can no
longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
instead of falling through the health wait, bank the 336 GiB reference container into trash
only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
section that names the live campaign instead of the retired swift15 era.
Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.
Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.
PR #81's pool half measured inert the day it deployed: the 66,560 boot printed the
same elastic 26.73 GiB / experts 20.72 / kv cache 5.64 / vram_weights 24.30 as the
49,152 boot and opened the pool with 0 B used, and the qwen38-w5nl64 rig moved
vram_kv 269.99 -> 271.25 MiB (window still 7,436 tokens: its 95.37 GiB n-gram table
claim decides, not the pool). The step half stays. Tests: 576 ok / 0 fail.
frank-archon.env: HOST_POOL_MIB 49,152 -> 66,560 a rank (tp x 65 GiB against the
246 GiB the box has), and the TTM pair 162,000/204,800 -> 197,000/224,000 — the
floor the bats ledger formula computes at the new pool. The pool is the KV lever
under the budget resolver: every MiB pinned in host RAM lowers the expert floor on
the card one-for-one and the cache takes it.
scripts/run-qwen38-bf16.sh: the campaign's private FRANK_QWEN38_MNBT (1,024) is
gone — MNBT now comes from FRANK_ARCHON_MAX_BATCHED_TOKENS, production's 2,048 and
the widest step the quantised expert file tiers stream. That 1,024 was PR #80's
answer to a starved cache; the pool is the fix, a narrow step was the symptom's
disguise, and a per-mode literal measures a shape nothing serves.
tests: frank_archon_unit greps the new pool value; qwen38_reference's SHAPE fixture
carries it, and the two MNBT-override tests become the inheritance assertion (every
mode's argv quotes the value the env file owns) plus the calib assertions.
vault: the shape block moves to the new numbers and the Eras table gains the row for
the era this replaces. The 2026-10-20 boot-ledger line stays as the old era's record
until the new boot's ledger replaces it.
The calibration mode kept a private step (8 seats at MNBT 65,536) that it
called "free throughput". The step's activation arena is billed to the card,
so the wide step cost the container its own expert slab and KV: at 65,536
the arena is 10.65 GiB and the claim falls to 16.85 GiB (weights 14.24 GiB
in VRAM, 56.24 GiB stranded on disk, KV 271 MiB for 8 seats), and the boot
dies with "startup stopped: out of memory" (measured 2026-10-29, while the
kldref boot at MNBT 1,024 — arena 504 MiB, claimable 30.87 GiB — had just
recorded for 16 hours without fault).
Every mode now takes its seat count from FRANK_ARCHON_MAX_SEQS, the file that
owns it, exactly as PR #77 established for the card split, and its step from
FRANK_QWEN38_MNBT. The per-mode literals in kldref and score follow, so one
number defines the geometry for all three. The test that pinned the old
exemption is re-pinned to the invariant that replaced it.
tests/run.sh: 576/576 green.
Portable POSIX-sh purge of exactly the fleet's deletion-policy trash: a
root's own .AGENTTRASH is emptied in place (the convention survives),
deeper .AGENTTRASH trees are removed with their nested entries accounted
once, symlinked/foreign-owned candidates are refused, / is refused.
Dry-run by default; --execute performs the plan and appends one dated log
line; every printed line is fixed at plan time in the on-disk plan table,
so dry-run and execute output can never drift.
Documented in Operations/Payload and Disk Reclaim.md section 6 (one fact
one owner - no new vault page). Tests: 14 green, ./tests/run.sh 576 green.
The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.
Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.
The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
The rerun after 96c5a06 refused every corpus document again ("document 196 is
8507 tokens and the KV pool holds 7436"), and the boot table shows why the
host-pool change could not have helped: vram_budget.cpp put the container in its
`no lender` branch — elastic 25.56 GiB, experts 25.29 GiB (their plan-fixed
share), kv cache 271.25 MiB — identical at 12,288 and at 49,152 MiB of host
pool. The only supplier left there is the activation arena, and the engine says
so at vram_budget.cpp:117: "lower --max-num-batched-tokens (the arena scales
with it)". The log prices both rungs directly (a 2,048-token step costs 1.08
GiB, a 1,024-token step 504.39 MiB), so the default moves to 1,024: the cache
grows to 866 MiB / 24,300 tokens and scores every one of the 197 documents
(longest 11,970). Calibration keeps its wide 65,536; it was never tied to MNBT.
This also retracts the mechanism 96c5a06 recorded in the header ("the pool buys
KV as well as residency"). Inheriting the env's 49,152 / 96 stays right as
single-source-of-truth, but on this payload the pool is not the cache's supplier.
run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768,
copied from the w4-era swift15 rig. The engine derives the whole per-card split from
those two numbers, and the KV cache's funding is `elastic - expert_floor` where the
expert floor is a function of the pinned host pool — so the pool buys cache as well
as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the
recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV
pool holds 7436"), exiting 1.
The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from
scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and
the header records the law instead of the claim that the pool never touches logits.
tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with
it; the suite is green (558 tests).
Two tools the KLD gate depends on.
build-qwen38-bf16-rad.sh converts the checkpoint to a BF16 .rad with NO recipe
line, so every tensor stays bf16 - the reference must not carry the quantization
it is there to measure. It gates before writing 335 GiB: the checkpoint present
and fully verified by the fetcher's own verifier, the peak-disk bill (350,000
MiB: the checkpoint's 343,323 plus margin, because the .rad is built beside it),
an existing output retired to ~/.AGENTTRASH by --force instead of deleted, and
--plan-only before the real convert. --max-ctx 32768 is a declare-time check, not
a weight change: dropped, the resolver is handed the checkpoint's 262,144 window,
which at fp8 cache is 258.8 GiB on a 192 GiB machine.
run-qwen38-bf16.sh serves that container in three modes - kldref (record the
reference logits), score (record an arm's logits against that reference), calib
(open the Gram tap for the imatrix). It drains frank-archon and restores it on
every exit path, and the two logits modes ride ONE step size
(FRANK_QWEN38_MNBT, default 2048) and ONE window (FRANK_QWEN38_MAX_MODEL_LEN,
default 32768) so a reference and an arm cannot be measured at different shapes:
that is the whole validity of the KL number. Four slots over 2,048 matches
frank-archon.env's operating point (MAX_SEQS=4, MAX_BATCHED_TOKENS=2048); the
smaller host pool (12,288 vs 49,152) buys residency, which moves speed, not
logits. calib keeps 65,536 - decode-bound serving, never the gate.
kldref refuses before the drain when the corpus's longest record cannot fit the
declared window (limit = window x 3 chars, the pessimistic tokenizer bound; the
measured corpus is 38,263 chars, 2.6x inside 32,768). It runs on --dry-run too,
so the check is visible before a 20-minute drain is spent. 12 seam tests, each
red-proofed by neutralising the gate it pins.
The swift15 gate compared an arm against the BF16 reference and passed on
KL <= 0.03. That bar is now the wrong instrument: the fleet already ships the
publisher's FP8-native iq4r-moe payload, so an arm that beats BF16 but loses to
what is running is a downgrade. The gate takes three recordings - fp8 anchor,
iq4r anchor, candidate - and requires the candidate's KL strictly inside the
interval the two anchors define, at every slice, plus top-1 agreement no worse
than the served payload.
The anchors must be ordered (fp8 better than iq4r) or the run stops: an
unordered pair means the instrument or the recording is wrong, not that the arm
lost, and one number out of the report is named rather than guessed at.
--recover prints the tightened numbers the same way it prints the failing ones,
so the operator sees the bar they would have had to clear.
One template (every routed expert at five bits, E4M3 scales in 64-groups, the
top level rotated through the 128-wide Hadamard chosen by GPTQ against CALIB)
plus a driver that generates the per-arm expert row, so the other 60 lines can
never drift between arms: control (cd=3), cd=6, and act_order=1 with NO cd= -
libquant refuses the pair (lq_registry.cpp:105), so the AoA arm is its own
sweep, not a fourth column.
The header carries the campaign's order (fetch -> build reference -> one drained
window for the calibration tap and the KLD reference -> trash the bf16 container
-> imatrix -> arms -> fetch the FP8 anchor last -> gate -> trio edit), because
that sequence is point-of-use for whoever runs it next, and the disk sequencing
is forced: 754 GiB free does not hold checkpoint + reference + three arms.
Gates that bite before 5h13m of convert time is spent: the BF16 reference still
occupying the volume, a recording too thin to be a calibration (converting
uncalibrated is silently worse and only whispers on stderr), an imatrix outside
the mounted campaign dir, an arm that already exists, a protected derivation
that cannot land its target, and a destination that cannot hold the output. Each
arm runs --plan-only first so the engine's own fit check bills at plan time; the
served container's argv shape is what the arms declare.
rad-convert's GPTQ path reads importance from an imatrix GGUF; the reference
container's --kld-record pass emits per-expert and per-hidden Grams as raw
recordings. This folds them into one v3 file: a single expert-major f32 entry
per MoE layer plus the hidden-width entries, every data offset a multiple of
general.alignment as the loader requires.
Refusals, each one a silent-worse outcome the swift15 campaign hit: an expert
with no gram of its own carries the layer's pooled diagonal and the count is
reported (a silent zero is a down-quantized expert nobody priced); expert grams
with no pooled gram, a gram short by one float (names the file), the same global
expert recorded twice with different numbers, an empty recording directory, and
an existing output. --min-rows sets the noise floor below which a recording is
dropped. rows is u64 so a corpus that outgrows u32 survives the header instead
of wrapping.
The swift15 rule (top 7% by imatrix energy, bar-only) left the plane size to the
distribution. The w5nl64 arm trades a *pinned host pool* against KL, so the
operator needs to say how many GiB of bf16 experts the pool can afford and have
the set land there: --target-count walks the ranked candidates until the byte
bill matches, keeps the bar-only pass for experts that are disproportionate for
their size, and reports the pool's ceiling instead of padding when the count
cannot be reached.
TP=2 cuts the expert plane into two 256-expert rank halves, and an odd count in
either half unbalances the placement, so the parity audit is a refusal (exit 2,
'PARITY LAW BROKEN'), not a silent fix-up: the driver stops rather than
converting a payload whose expert placement no one priced.
The w5nl64 campaign measures against the vendor checkpoint, not the Swift-1.5
finetune: the served payload (qwen3.8-next-flash-fp8-iq4r-moe) is quantized from
this base, so fidelity must be scored against the same reference. 335.3 GiB
across 10 shards.
Verifier gates on the index's own shard list, each shard's declared byte count,
and the Hub's declared total - a curl without -f writes the 401 body as the
'shard' and reports success. --fetch refuses when the destination cannot hold
the checkpoint, because a 335 GiB download that fills the disk mid-shard costs
the whole campaign window.
iptables -I prepends, so the install block's final -I of the blanket DROP
landed it at position 1, above all three ACCEPTs: every packet to
18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each
ACCEPT). The host's own health probe timed out beside a LISTENing socket
and the whole dankclaw drain aborted at the alertmanager gate twice.
Install now rebuilds the tail: delete any blanket DROP wherever it sits,
keep the ACCEPTs at the head, append the DROP last - converges on hosts
poisoned by the old order. New bats case runs install_signal_relay for
real against an iptables sandbox seeded with the poisoned order (red on
the old code, green on this one).
Three stacked lessons, each proved live 2026-10-21:
- the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the
binary made it a positional arg: 'unexpected /bin/blackbox_exporter'.
- the podman gateway 169.254.1.2 is NOT bindable from the host netns - it
exists only inside each rootless netns: 'bind: cannot assign requested
address'. The scrape path does not need it bound: a bridge container
dialing the gateway lands on the host LOOPBACK (proved 200 from the
prometheus container), so the exporter binds 127.0.0.1 + LAN.
- HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token
and quadlet folds the following --health-interval/--health-retries (and
the image) into the command string; the plain exec form the sibling units
use is correct.
Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a
bind address); deploy-remote gate stays loopback, which is now honest.
PR #69 wired the copier, planner entry and health gate but missed the
three dispatch gates: the client valid_service, the remote
service_is_allowed, and deploy_service's case list. The client gate
rejected --service blackbox-exporter outright; even past it the
dispatcher's catch-all hard-fails on dankclaw (no compose since
2026-09-16). New seam pins the class: every dankclaw service the
remote gate allows must carry a deploy_service case - proven red by
planting a gated, undispatched name.
copy_alloy_config now validates the rendered config with the very
alloy image that will run it, before installing: a misplaced attribute
crash-loops the unit behind a green deploy, and no textual seam can
see it. copy_blackbox_config ships the prober config; the planner maps
the new unit and shared/blackbox/** to the blackbox-exporter service,
ordered before prometheus.
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.
Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/
as well; our mounts filled only /opt/xilinx/xrt/lib and the install path.
Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming
the missing prefix ('No such library
/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine
supervisor takes the WHOLE container down on a dead lane — measured
2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the
GPU lines looked healthy.
Each XRT library now mounts THREE times from the same real file (never a
symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename
so the bats fixture proves the derivation. Proven end-to-end before commit:
production-shaped scratch boot + the three extra mounts answers the full
boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling.
A dead lane is not survivable on this engine — arm it only with all three
prefixes (page law updated).
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.
Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.
The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.
Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.
The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.
Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.
TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.
BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.
Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1
source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding
sequence and 1/8 of the budget per waiting request, and refuses to shrink the
chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment
carried a 512 that is not this build's auto value (0 until the held-step
counter asks; 64 in help mode). The serve header still described three decode
slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.
scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
Frank
dank