comments: replace law idiom with the mechanism it encoded

Every comment that cited a named law, an incident date, or an internal nickname now states the underlying mechanism and why it matters, generically: curl exit-0 artifacts and byte-count gates, argv-in-/proc secret exposure, tag mutability vs digest pinning, shape-specialized kernels refusing uncompiled slot counts, MTP heads as file tensors no flag can supply, drafter/slot coupling as an engine property, pkill -f matching its invoker, pinned-RAM pairing with LimitMEMLOCK, one-encoding-per-table, the short-run first-token noise floor, and the protected-expert rule. No behavior change: comment/string text only; suite green (371 tests).
docs(operations): LLM usage-attribution research catalog
2026-10-14 13:25:41 -07:00 · 2026-10-14 13:25:41 -07:00 · 2026-10-13 20:51:32 -07:00 · 2026-10-13 20:33:02 -07:00 · 2026-10-13 20:18:41 -07:00 · 2026-10-13 20:00:39 -07:00
29 changed files with 1147 additions and 220 deletions
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
+| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
 | Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
 | Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |

@@ -525,21 +525,43 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
 Rollback: revert the swap commit — the fp8-native `.rad`
 (`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.

-**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`,
+**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
 `--max-model-len 200000` (the harness contextWindow must quote this),
 `--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
-partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
+partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
 replaced `--expert-vs-cache-ratio`; the two are ONE budget —
-`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
-consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
-**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`,
-`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
-disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
-v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
-`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16
-tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's
-386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool
-10,375 slots, cards settled 1.85 GiB free.
+`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
+32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
+209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
+claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
+hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
+because it is the only dtype that funds the slot count at this window**,
+`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
+8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
+**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
+(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
+ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
+all at 200000 tokens at once, which is the worst case` — `kv_attn` page
+26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
+`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
+(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
+(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
+563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
+every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
+the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
+settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
+4.22 GiB to the expert cache. Prefill tracks this slab only up to a
+**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
+(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
+(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
+(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
+not a slope.
+
+**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
+single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
+4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
+aggregate**, so the slot count is paid in decode concurrency (a fused-step
+lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.

 Shape notes carried from the fp8-native era and unchanged: wht6 gains,
 MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
@@ -551,35 +573,52 @@ on the fp8 shape (512-token streamed probe). The campaign also
 measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
 accept 0.7246).

-**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` —
-one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what
-frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on
-both halves. Measured per rank from the two boot ledgers: `kv_attn` (13
-layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13
-layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not
-attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at
-bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms
-that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB
-of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` =
-462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is
-paid twice on this engine, and half-refunded: idle KV VRAM is also the expert
-cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the
-fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots
-× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to
-the retired tcclaviger fork's torch.compile, not radiance) and it also drops
-the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is
-bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
-buffers + arena onto the weights side, so `weights + kv` is capped near
-29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
+**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
+`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
+llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
+equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
+the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
+KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
+block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
+and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
+dtype-independent terms that live in the same pool (~110 MiB a sequence of
+recurrent state + 448 MiB of prefix checkpoints):
+`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
+engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
+gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
+the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
+bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
+where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
+half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
+pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
+`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
+25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
+the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
+law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
+weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
+pair, never `--kv-cache-size-mib`.
+
+Shape notes carried from the fp8-native era and unchanged: wht6 gains,
+MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
+prefix cache, and the rule that a prompt over `--max-model-len` is
+refused, never truncated. What changed: the fp8-native era's NVMe expert
+streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
+host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
+on the fp8 shape (512-token streamed probe). The campaign also
+measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
+accept 0.7246).

 **The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
 pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
 the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
 `RLIMIT_MEMLOCK` — the TTM floor is set just above it.
-`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
-script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host
-204,800) refuses to start until both knobs are raised in the stop window
-(sysfs write first, kernel cmdline + reboot only if read-only).
+`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
+runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
+by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
+before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
+and the preflight re-refused 6,919 times. A manual sysfs write in the stop
+window is a convenience, not the repair; kernel cmdline persistence
+(`ttm.pages_limit=52428800`) is the belt to it.

 **Contract facts** (source-verified): the HTTP listener binds AFTER the
 full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -761,6 +800,17 @@ linked runbooks.
  the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
+- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
+  to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
+  2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
+  frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
+  and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
+  `activating (auto-restart)` that exits 1 in 50 ms with no container and no
+  engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
+  `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
+  to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
+  after a write the kernel rejected. Never prescribe a manual sysfs write as the
+  recovery procedure for a service that can boot before a human reaches it.
 - **Silent RAM corruption outranks every software theory.** frank-archon's
  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
@@ -27,7 +27,8 @@ CALIB=/home/dank/corpora/calib
 OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
 LOG=/home/dank/logs/swift15_convert.log
 # gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
-# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
+ # the format assigns ONE encoding per expert table, so a per-tensor block-fp8
+ # down tier cannot be served beside a w4 gate_up tier -- see the recipe header).
 MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}

 # Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
@@ -947,7 +947,7 @@ deploy_plex() {
    # accepted and then answered with nothing (curl 52, Empty reply) against a
    # perfectly healthy server. Measured 2026-10-02 — the gate burned its full
    # 600s and failed a plex whose https://127.0.0.1:32400/identity answered
-    # 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
+    # 200 in 44 ms the whole time. The same requirement is why the Caddyfile dials the
    # plex upstream over TLS (tests/caddyfile.bats pins it).
    # -k: PMS presents its own self-signed cert on 32400.
    #
@@ -1207,8 +1207,9 @@ deploy_llm_dragoon() {
    # on the FULL container name, which appears in the argv of a manual
    # `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
    # this deployer's own command line (`bash deploy-remote.sh
-    # frank-dragoon`) — so it cannot shoot itself in the foot (the fleet
-    # pkill law), and it can never match frank-zealot's llama-server.
+    # frank-dragoon`) — so the sweep can never match the deployer's own argv
+    # (`pkill -f` kills the invoker when the pattern appears in the invoker's
+    # own command line), and it can never match frank-zealot's llama-server.
    run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
    pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
    podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
@@ -2,7 +2,7 @@
 """derive-protected-experts.py — rewrite the recipe's protected-expert section
 from a calibration recording. RUNS ON DANKSERVER (venv: numpy).

-The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
+ The protection rule (docs/MOE-W4.md "Protected experts", restated in the recipe):
 an expert whose share of its layer's down-input energy is at least 3% AND
 at least three times its share of the routing is kept bf16; the count must
 be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
@@ -14,7 +14,7 @@
 # Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
 # listing/resume downloader, curl+jq only): per-file size checks against
 # the Hub listing, `-C -` resume, pickle-format duplicates skipped.
-# Verify (default) is the AGENTS.md law in three parts: every listed file
+ # Verify (default) enforces the download contract in three parts: every listed file
 # present with the listed size, the shard count matches the weight-map
 # index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
 # only the public API.
@@ -12,7 +12,7 @@
 # Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
 # listing/resume downloader, curl+jq only): per-file size checks against
 # the Hub listing, `-C -` resume, pickle-format duplicates skipped.
-# Verify (default) is the AGENTS.md law in three parts: every listed file
+ # Verify (default) enforces the download contract in three parts: every listed file
 # present with the listed size, the shard count matches the weight-map
 # index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
 # only the public API.
@@ -4,9 +4,10 @@
 # .hgn, the REQUIRED lookup table and the optional vision tower from
 # peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
 # card pins), and the flat tokenizer the two repos share. All public, no
-# token — and per the fleet secret law, none would ever touch argv.
+ # token — and a token never rides a command line: argv is readable by every
+ # process on the host through /proc, so a key placed there is a key exposed.
 #
-# The fetch law, same as every sibling script: curl -f (an access error
+ # The fetch discipline, same as every sibling script: curl -f (an access error
 # must not become a "successful" 153-byte artifact), exact byte count from
 # the Hub tree API, LFS sha256 verification, --verify-only to re-check an
 # existing payload without downloading.
@@ -26,7 +27,8 @@
 #   tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
 #     already on disk (tokenizer.json LFS
 #     0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
-#     fetched fresh rather than symlinked, per the no-symlink law.
+ #     fetched fresh rather than symlinked, so the payload dir is
+ #     self-contained — every gate and every mount sees real files inside it.
 set -euo pipefail

 DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
@@ -1,10 +1,11 @@
 #!/usr/bin/env bash
 # download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
 # the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
-# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
-# secret law, none would ever touch argv).
-#
-# The fetch law, same as every sibling script: curl -f (an access error
+ # PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and a token never
+ # rides a command line: argv is readable by every process on the host
+ # through /proc, so a key placed there is a key exposed).
+ #
+ # The fetch discipline, same as every sibling script: curl -f (an access error
 # must not become a "successful" 153-byte artifact), exact byte count from
 # the model card, LFS sha256 verification, --verify-only to re-check an
 # existing payload without downloading.
@@ -11,8 +11,8 @@ Tensor extents are DERIVED from consecutive GGUF header offsets — no quant
 type-size table (those are easy to get subtly wrong; a wrong per-element size
 silently under-covers each tensor and dumps tail samples into a junk bucket).

-Fleet law this instrument established (2026-09-26, 75 s window, 4 mixed
-streams, 2.7M expert-tensor samples): per-BLOCK read volume is FLAT
+ What this instrument shows (75 s window, 4 mixed streams, 2.7M
+ expert-tensor samples): per-BLOCK read volume is FLAT
 (0.93x-1.08x of mean across all 40 CPU-resident blocks) because MoE top-k
 routing fires every block every token — there are no "most active layers" at
 `-ot` granularity. The real skew is per-EXPERT (top 10% of (block,expert)
@@ -3,10 +3,19 @@
 # built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
 # dankserver :7770.
 #
-# SHAPE (2026-10-11): 2 decode slots x 200,000 tokens with an UNQUANTIZED
-# (bf16) KV cache — the pool is sized for the pair, and the weights slab pays
-# for it (see the KV block). The fp8-native payload this file was written for
-# is the rollback target and keeps its own trio under the fp8-native runbook.
+# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
+# with the per-card budget spent on resident experts: 25,600 MiB of slab +
+# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
+# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
+# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
+# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
+# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
+# +25 % at the full window) while KV beyond the contract buys nothing.
+# Consequence, deliberate: a FOURTH full-height conversation does not fit this
+# pair, and the engine refuses it — it never truncates. See the KV block for the
+ # arithmetic and the measured per-dtype cost. The fp8-native payload this file was
+# written for is the rollback target and keeps its own trio under the
+# fp8-native runbook.
 #
 # 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
 # pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
@@ -113,18 +122,28 @@ FRANK_ARCHON_TP=2
 # +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
 # engine default.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots: TWO, so the served window is a CAPACITY CONTRACT — two
+# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
 # concurrent conversations of --max-model-len fit the KV pool (see the KV
-# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
-# tcclaviger vLLM fork's torch.compile shape specialization on
-# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
-# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
-# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
-# (30.9-33.1 t/s, inside the +-3% noise floor). Side benefit: the fused TP
-# all-reduce is declared over seqs x (1 + spec) = 6 rows, under the 25-row
-# bound at which the wht6 wire stays EXACT (single-sequence decode and any
-# step of <= 25 rows is bit-exact; only busier steps are lossy).
-FRANK_ARCHON_MAX_SEQS=2
+ # block below). A runtime that compiles its kernels specialized to the exact
+ # batch shape can refuse a slot count it has never compiled for; a
+ # shape-agnostic engine accepts any count that fits memory. Radiance is the
+ # latter: seqs 2 boots, and measured decode at seqs 2 and 3 is statistically
+ # tied (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys
+ # concurrency, not speed, so a slot costs KV, never decode.
+# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
+# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
+# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
+# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
+# steps are lossy).
+# The pool is sized to this contract and nothing more (see the KV block):
+# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
+# the resident expert slab. The trade is explicit and it is Frank's call:
+# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
+# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
+# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
+# traffic that wants four full-height sessions is a --vram-kv-mib change, not
+# a code change.
+FRANK_ARCHON_MAX_SEQS=3
 # Served window. Native is 262,144; 200,000 is the publisher's production
 # value (compose) and what models.yml's contextWindow must quote — a longer
 # prompt is refused by the engine, never truncated. The engine treats
@@ -145,25 +164,52 @@ FRANK_ARCHON_HOST_POOL_MIB=44928
 # VRAM left unclaimed per card (publisher default 96 MiB).
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
 # VRAM split of claimed memory per card, MiB (0 = derive from the card).
-# THE CEILING, measured on this host 2026-10-11: card 31,856 MiB, a boot
-# consumes 1,710 MiB OUTSIDE both budgets (driver context + activation arena
-# 990 + hardware queues 100 + headroom 96 + step staging), and the fp8 pair
-# settled at 1,379 MiB free — so `vram_weights + vram_kv <= 29,120 MiB` keeps
-# >= 1 GiB free. The pair below is exactly that sum.
+# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
+# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
+# neither budgeted nor accounted — the driver's context, the activation arena
+# and --gpu-headroom-mib come out of it; anything past those is throughput on
+# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
+# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
+# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
+# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
+# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
+# the engine's; the pair below spends the difference and lands 458 MiB above
+# the engine's floor. `weights + kv <= 30,969` is the gate.
 # THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
 # expert cache's spillover (`mover: 2275 slots can be lent out of the KV
 # cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
 # so a bigger KV pool steals weight residency from the slab AND from the
 # lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
 # a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
-# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. Expect -3..-8%
-# vs the fp8 control (136 t/s median at 44.5k depth, acceptance 0.7246,
-# ~190 t/s on a live 512-token probe); re-tune from a boot ledger + probe,
-# never from that estimate.
-FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208
-FRANK_ARCHON_VRAM_KV_MIB=6912
-# KV cache UNQUANTIZED (bf16). This engine's flag takes ONLY bf16|fp8 — there
-# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
+# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
+# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
+# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
+# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
+# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
+# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
+# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
+# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
+# The margin is thin by design (608,027 predicted tokens against the 600,000
+# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
+# once` line is the referee for THIS pair and the concurrent probe is its
+# regression test.
+# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
+# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
+# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
+# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
+# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
+# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
+# pair landed inside noise of EACH OTHER, which is what made it read as a
+# plateau; two points are not a curve.
+# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
+# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
+# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
+# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
+# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
+FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
+FRANK_ARCHON_VRAM_KV_MIB=5120
+# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
+# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
 # --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
 # UNSERVABLE here: the rung below bf16 is fp8 on both halves.
 # Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
@@ -177,45 +223,57 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
 # The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
 # copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
 # checkpoint region 8 x 55.99 = 448 MiB. So:
-#   pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
-#   6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
-#   reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
-#   and the ledger says it outright: "backs 2 sequences all at 200000 tokens
-#   at once, which is the worst case".
-# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
-# sessions, and a third long conversation preempted to the host/disk tiers).
-# Secondary cost: the prefix-cache entry grows with the attention plane
-# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
-# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
-# the same number of parked conversations, out of RAM the host has
-# (104 GiB available with the service loaded).
-FRANK_ARCHON_KV_DTYPE=bf16
+#   pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
+#   3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
+#   and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
+#   three-slot contract with 1.3 % slack, every MiB above 5,063 having been
+#   handed to the expert slab instead. The ledger line is the referee ("backs N
+#   sequences all at 200000 tokens at once") — a pair this tight is read off the
+#   boot, never assumed.
+#   At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
+#   the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
+#   11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
+#   slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
+#   tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
+#   sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
+# Secondary saving: the prefix-cache entry shrinks with the attention plane
+# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
+# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
+# many conversations as it did at bf16 — no re-size needed to park three slots.
+FRANK_ARCHON_KV_DTYPE=fp8
 # Spec decode: the checkpoint's own MTP head at the publisher's depth-3
 # operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
 # flag states it so the ledger records the intent).
 FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
-# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
-# The host tier is sized in ENTRIES, and one entry doubles with the cache
-# dtype (58.50 KiB at fp8 -> 117 KiB at bf16, per the ledger's `kv tier
-# transfer` line): 4096 MiB at bf16 parks ~1.4 sessions of 200k where it
-# parked ~2.9 at fp8. Doubled to keep the capacity, at RAM the host has
-# (104 GiB available with the service loaded).
+# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
+# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
+# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
+# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
+# parked set fits with the tier as it stands — at RAM the host has (104 GiB
+# available with the service loaded).
 FRANK_ARCHON_PREFIX_HOST_MIB=8192
 FRANK_ARCHON_PREFIX_DISK_MIB=65536
-# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
-# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
-# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
+ # The PLE n-gram table (47.7 GiB) serves from pinned host RAM the cards
+ # read directly, so it cannot be evicted or page-faulted mid-token. `ram`
+ # placement REFUSES the start if the memlock limit is too small, and the
+ # unit's LimitMEMLOCK=infinity is the matching half of that pairing.
 FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
 # prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
 # pages_limit (the current kernel exposes NO pages_max; the cap function
 # takes the min over the knobs that exist). Below the plan the load dies
-# mid-pin with an allocator error instead of this named refusal — raise the
-# knob first (runtime sysfs write; kernel cmdline fallback + one reboot only
-# if the file is read-only), see the runbook. Cap raised 2026-10-05 to
-# 52,428,800 pages = 200 GiB, which still clears this floor.
+# mid-pin with an allocator error instead of this named refusal.
+# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
+# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
+# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
+# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
+# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
+# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
+# root unit is privileged; the knob is a 0644 root module parameter), so the
+# recovery is a boot, not a visit to the machine.
 FRANK_ARCHON_TTM_MIN_MIB=150000
+FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
 # multi-flag values as one string; the serve script expands it unquoted on
 # purpose.
@@ -35,6 +35,13 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
+# The `+` prefix runs this as root even though the service is User=dank — the
+# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
+# unusable below it. After a reboot the knob is back at its default (half of
+# RAM), so the raise must be part of the start, not a note in a runbook: the
+# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
+# times. The serve script re-verifies the cap after this hook.
+ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
 # systemd stop: wait alone cannot stop the container child; stop it
 # explicitly. No graceful drain contract is claimed (the OpenAI front end
 # holds no queue to drain — in-flight decodes are lost, which a stop
@@ -42,10 +42,10 @@
 #   form of the engine's default, so the seam test can pin the product.
 #   KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
 #   HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
-#   drafter is NOT slot-locked on this engine (batching and speculation
-#   coexist — DFlash-era slots>1 law does NOT carry over; the card's own
-#   quickstart ships 2 slots WITH the head), so the second slot is
-#   concurrent capacity, not a speculation tax.
+ #   drafter is NOT slot-locked on this engine: some engines silently
+ #   disable drafting the moment a second slot is enabled, this one does not
+ #   (the card's own quickstart ships 2 slots WITH the head), so the second
+ #   slot is concurrent capacity, not a speculation tax.
 #   Memory ledger against the box's 121.2 GiB unified (the n-gram table is
 #   mmap'd page cache beside the checkpoint, not a pinned charge — the
 #   card's own quickstart arithmetic does not bill it):
@@ -88,8 +88,9 @@
 #     before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
 #     amdgpu.noretry=0 (check at deploy).

-# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
-# is what was qualified). 0.16.2 = the version the model card names as its
+ # Image: digest-pinned (a tag is a mutable pointer — the same tag can
+ # silently deliver different bits — so the digest pins exactly the bits
+ # that were qualified). 0.16.2 = the version the model card names as its
 # quickstart pin; RepoDigest captured at pull on 2026-10-06.
 FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
 # Container identity: kept from the 27B era (unit ExecStop names it).
@@ -16,7 +16,7 @@
 # custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
 # in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
 #
-# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
+ # KV rate arithmetic: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
 # token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
 # 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
 # recurrent state per SLOT (not per token) — that is why two 200k slots fit
@@ -29,7 +29,8 @@
 #     its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
 #     getattr_static()s each one off the instance, breaking on the first miss.
 #     So BOTH wrappers need all three attributes.
-#   * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
+ #   * Method-resolution order forbids the obvious inheritance: adding
+ #     SupportsEagle3 as a base dies at class creation with
 #     "Cannot create a consistent method resolution order (MRO) for bases ..."
 #     (GLM's wrappers already carry four/five Protocol-derived bases).  The
 #     attribute route avoids the MRO entirely.  Also proven by simulation:
@@ -1,22 +1,31 @@
 #!/usr/bin/env python3
-"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE
+"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
 frank-archon (radiance) service. Run it ON dankserver: it talks to
 127.0.0.1:7770 and builds its prompts from the repo checkout there.
 No scratch container, no mask: the service under test is the service.

    ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'

-Three probes, in increasing cost:
+Four probes, in increasing cost:
  A  decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
     Reports the engine's own `timings` (prefill, decode, draft acceptance).
-     The bench law applies: >=128 generated tokens, greedy, real text.
-  B  the capacity contract — TWO concurrent chats of ~197k prompt tokens
-     (~394k tokens resident) each carrying a unique marker at its head. Both
-     must answer their own marker. This is what a 2 x 200,000 endpoint owes:
-     two full-height conversations resident at once, exact retrieval in both.
+     Measurement discipline: generate at least 128 tokens (a short run
+     measures first-token noise, not the server), greedy, real text.
+  B  the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
+     the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
+     host), each carrying a unique marker at its head. All must answer their own
+     marker. This is what the endpoint owes: N full-height conversations
+     resident at once, exact retrieval in every one of them.
  C  prefix-cache restore — probe A's cold prompt again, so the disk tier has
     to give the blocks back (cache_n > 0, TTFT collapses).
-
+  D  concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
+     the window. This is the cost a slot count imposes on DECODE, which no
+     prefill number shows: the fused-step schedule is unfair, so every stream is
+     printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
+     single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
+     window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
+     Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
+     shape hunt missed this for a whole day because leg B generates 24 tokens.
 PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
 `reasoning_content`, so a chat request whose budget is spent inside
 `<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
@@ -114,23 +123,51 @@ def show(tag, j, wall, expect=None):
    print(line)


+def deployed_shape():
+    """The contract is the deployed env, not a constant: slot count and window
+    come from the checkout on this host, so the probe cannot drift from the
+    service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
+    2 x 200,000, and the fp8 era 3 x 200,000)."""
+    shape = {}
+    with open(f"{REPO}/scripts/frank-archon.env") as fh:
+        for line in fh:
+            line = line.strip()
+            if line.startswith("FRANK_ARCHON_") and "=" in line:
+                key, _, val = line.partition("=")
+                shape[key] = val.strip().strip('"')
+    return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
+            int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
+            shape["FRANK_ARCHON_KV_DTYPE"])
+
+
 def main():
+    seqs, maxlen, dtype = deployed_shape()
+    print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
    print("health:", json.dumps(get("/health")))
    print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
    deep = corpus(160_000)
    wide = corpus(1_100_000)

-    facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"}
-    bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()}
+    # 3.28 chars/token is the measured ratio of this corpus through the
+    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
+    # deploy); 98.5 % of the window leaves the response room under the ceiling.
+    chars = int(0.985 * maxlen * 3.28)
+    # Each marker's FIRST hyphen-separated word is unique, because show()
+    # scores the hit on that word — a shared prefix would score every slot HIT
+    # on any answer that says the word.
+    words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
+    facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
+             for i in range(seqs)}
+    bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
    print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
-          f"request, x2 concurrent")
+          f"request, x{seqs} concurrent")

-    # A + C first (cheap), B last (the expensive pair).
+    # A + C first (cheap), B last (the expensive set).
    for tag, salt in (("A-cold", "A1-9137"), ):
        j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
        show(tag, j, wall, "zephyr-lantern-7")
    t0 = time.time()
-    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
+    with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
        futs = {s: ex.submit(post, b) for s, b in bodies.items()}
        for s, fut in futs.items():
            try:
@@ -139,6 +176,24 @@ def main():
            except Exception as e:  # a preemption/timeout lands here
                print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
    print(f"[B] total {time.time() - t0:.1f}s")
+    # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
+    dchars = int(0.25 * maxlen * 3.28)
+    dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
+               for i in range(seqs)]
+    print(f"[D] {seqs} concurrent streams x 512 tokens, "
+          f"{dchars:,} chars each")
+    t0 = time.time()
+    with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
+        futs = [ex.submit(post, b) for b in dbodies]
+        for i, fut in enumerate(futs, 1):
+            try:
+                j, wall, _ = fut.result()
+                show(f"D-s{i}", j, wall)
+            except Exception as e:  # a preemption/timeout lands here
+                print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
+    tot = time.time() - t0
+    print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
+          f"{seqs * 512 / tot:.1f} t/s aggregate decode")
    j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
    show("C-prefix-hit", j, wall, "zephyr-lantern-7")

@@ -1,9 +1,9 @@
 # qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
 # (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
 # base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
-# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
+ # through GPTQ into the w4nl64a8h table. The format assigns exactly ONE encoding per table
 # ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
-# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
+ # beside it", qwen4exp_fp8 rad_arch_declare — so the original split tier (down block-fp8 +
 # gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
 # The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
 # and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
@@ -26,8 +26,8 @@
 # The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
 # CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
 # The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
-# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
-# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
+ # this section from Swift's OWN calibration recording before this recipe is ever run, by the
+ # protection rule -- an expert whose share of its layer's down-input energy is at least 3% and at least three
 # times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
 # takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
 # PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
@@ -13,9 +13,11 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
-# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
-# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
+# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
+# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
+# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
+# own budget table makes claimable, spent on the slab because prefill is
+# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
 # 8 GiB + disk 64 GiB, PLE pinned in RAM.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
@@ -61,6 +63,7 @@ set +a
 : "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
 : "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
 : "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
+: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
 : "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"

 # The image gate is the digest-pinned ref itself: a name@sha256 ref exists
@@ -117,34 +120,19 @@ if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
    exit 1
 fi

-# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
-# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
-# pages_limit (the current kernel exposes NO pages_max; the cap function
-# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
-# the load dies mid-pin with an allocator error instead of this named
-# refusal — raise the knob first (runtime sysfs write; kernel cmdline
-# fallback + one reboot only if the file is read-only), see the runbook.
-# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
-ttm_cap_mib() {
-    # min over the knobs THIS kernel exposes — the current kernel has
-    # pages_limit only (no pages_max); a knob that is absent is skipped,
-    # not a zero (a zero would refuse a healthy cap).
-    local v cur=""
-    for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
-        [ -r "$p" ] || continue
-        v="$(cat "$p" 2>/dev/null)" || continue
-        [ -n "$v" ] || continue
-        v=$(( v * 4 / 1024 ))
-        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
-    done
-    echo "${cur:-0}"
-}
-TTM_CAP="$(ttm_cap_mib)"
-if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
-    echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
-    echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
-    exit 1
-fi
+# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
+# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
+# pages, capped by ttm's pages_limit (the current kernel exposes NO
+# pages_max; the cap function takes the min over the knobs that exist).
+# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
+# error, so the gate refuses first — and RAISES the knob before refusing:
+# the cap is a runtime knob whose default is half of RAM, so a power failure
+# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
+# and Restart=always re-ran the refusal 6,919 times over ~20 h).
+# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
+# shellcheck source=/dev/null
+source "$SCRIPT_DIR/ttm-preflight.sh"
+ttm_ensure_cap || exit 1

 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
@@ -70,7 +70,9 @@ fi

 # Size-exact gates on all three payload files: a truncated curl exits 0 on
 # some failures and a partial .hgn loads far enough to look alive, so the
-# byte counts from the Hub tree API are the contract (fleet fetch law).
+ # byte counts from the Hub tree API are the contract: curl exits 0 on an
+ # HTTP error and writes the error body as the file, so a truncated download
+ # is caught by size here and never boots.
 ck() {
    # ck <env-name> <path> <expected-bytes>
    if [ ! -f "$2" ]; then
@@ -63,8 +63,9 @@ fi
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
 # Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
-# there is no in-file head (the law: MTP is a property of the file, not of the
-# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
+ # there is no in-file head — this kind of drafter exists only as tensors
+ # inside the checkpoint file, and no flag can supply one the file lacks.
+ # SPEC_TYPE=none strips BOTH speculative flags (they move as a set);
 # naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
 # assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
 : "${SPEC_TYPE:=none}"
@@ -298,7 +298,7 @@ def main():
                 and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
    if not scale_law:
        log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
-            f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
+            f"down {ddt}{dsh} scale {dsd}{dssh} (128x128 block scales)")
        return 2
    shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
              "gs": tuple(gssh), "ds": tuple(dssh)}
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
+#
+# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
+# radiance expert pools + the PLE n-gram table) has its pinned pages charged
+# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
+# (and pages_max on kernels that expose one). The cap function below takes the
+# MIN over the knobs the running kernel actually exposes — an absent knob is
+# skipped, never read as zero, because a zero would refuse a healthy cap.
+#
+# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
+# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
+# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
+# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
+# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
+# therefore not an acceptable repair procedure: the gate raises the knob itself
+# (the units that need it run as root, and the knob is a 0644 root-owned module
+# parameter), re-reads, and only then refuses.
+#
+# Contract (set by the sourcing script's env file):
+#   FRANK_ARCHON_TTM_MIN_MIB     floor the pinned ledger needs; refuse below it
+#   FRANK_ARCHON_TTM_TARGET_MIB  value to raise the cap to when it is below the
+#                                floor (the recorded host ceiling — writing the
+#                                floor itself would leave the ledger no slack)
+#   FRANK_ARCHON_TTM_SYS_DIR     knob directory; defaults to the real sysfs
+#                                path, a test points it at a temp dir.
+# Pages are 4 KiB: pages = MiB x 256.
+
+: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
+
+# Prints the effective cap in MiB, 0 when no knob is readable.
+ttm_cap_mib() {
+    local p v cur=""
+    for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
+             "$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
+        [ -r "$p" ] || continue
+        v="$(cat "$p" 2>/dev/null)" || continue
+        [ -n "$v" ] || continue
+        v=$(( v * 4 / 1024 ))
+        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
+    done
+    echo "${cur:-0}"
+}
+
+# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
+# swallowed: the caller re-reads the cap, and a write the kernel refused shows
+# up there (ttm refuses a limit below the pages it has already pinned).
+#
+# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
+# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
+# canonical privileged path is the unit's ExecStartPre with the `+` prefix
+# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
+# human running the gate from a shell (passwordless sudo on both hosts).
+ttm_raise_cap() {
+    local pages=$(( $1 * 256 )) p
+    for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
+             "$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
+        [ -e "$p" ] || continue
+        if [ -w "$p" ]; then
+            echo "$pages" > "$p" 2>/dev/null || true
+        elif command -v sudo >/dev/null 2>&1; then
+            sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
+        fi
+    done
+}
+
+# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
+# needed; returns 1 with a named refusal when it cannot be made to.
+ttm_ensure_cap() {
+    local cap
+    cap="$(ttm_cap_mib)"
+    if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
+        echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
+             "floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
+             "$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
+        ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
+        cap="$(ttm_cap_mib)"
+    fi
+    if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
+        echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
+        echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
+        return 1
+    fi
+    echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
+}
+
+# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
+# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
+# the `Restart=always` loop can never outrun it.
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    set -euo pipefail
+    case "${1:-}" in
+        --raise)
+            TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+            # shellcheck source=/dev/null
+            source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
+            ttm_ensure_cap
+            ;;
+        *)
+            echo "usage: ${0##*/} --raise   (raise the ttm pages cap, then verify)" >&2
+            exit 2
+            ;;
+    esac
+fi
@@ -145,11 +145,18 @@ dtype`) so a future resize cannot silently under-provision the pool.

 ### Why the weights slab pays — and why that is not pure loss

-The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
-boot consumes outside both (driver context, the 990 MiB activation arena,
-hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
-**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
-resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
+The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
+hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
+24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
+context, the activation arena and --gpu-headroom-mib come out of it; anything
+past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
+against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
+static region)`. Claimable reserve above the ENGINE's floor:
+2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
+written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
+figure of 31,856 MiB; the engine never asked for that reserve and the seam now
+derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
+the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
 into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
 slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
 floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
@@ -253,6 +260,240 @@ from the scored configuration — bf16 can only reduce the attention-state
 error, and a re-score is a drained-window job if the gate agreement is ever
 needed in the record.

+## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
+
+### The outage: a refusal, not a crash
+
+Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
+never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
+after starting, with one journal line on repeat:
+
+```text
+ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
+frank-archon: raise /sys/module/ttm/parameters/pages_limit …
+```
+
+**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
+the name with a zero start time), so there was no engine, no port, and no engine
+log — the signature of a preflight refusal, which every deployer/monitor view
+outside the journal renders as an unexplained crash loop.
+
+The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
+expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
+shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
+runtime state, and its default is half of RAM** — so the manual raise of
+2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
+fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
+script enforces. The fp8-native era had planned the persistent half of this
+("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
+reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
+a power failure is not scheduled.
+
+### The repair: the gate raises the knob itself
+
+`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
+the MIN over the knobs the running kernel exposes, and when it is below
+`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
+52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
+The unit is root and the knob is a 0644 root-owned module parameter, so the write
+is legal; a kernel that refuses the move produces the same named refusal as
+before, and a reboot is now a boot rather than a visit to the machine.
+
+| Knob | Power-failure default | After the gate |
+|---|---|---|
+| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
+
+Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
+refuses when the write fails* and *the ttm cap counts only the knobs the kernel
+exposes* drive the gate against a temp knob directory — a below-floor writable
+knob ends at the target and the start passes; a read-only knob leaves the cap
+untouched and the start refuses by name; an absent `pages_max` is skipped, never
+read as zero (a zero reading would refuse a healthy host).
+
+### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
+
+Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
+concurrent 200,000-token conversations with quantized KV.
+
+fp8 is not a preference at four slots, it is the only dtype that affords them.
+The pool must back `seqs × max-model-len` (the engine's hard limit), so the
+fourth slot is bought in KV, out of the one per-card budget
+(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
+
+| dtype | pool for 4 × 200,000 | weights slab left |
+|---|---|---|
+| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
+| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
+
+So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
+2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
+44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
+move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
+**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
+(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
+1.93).
+
+Terms that move with the dtype and the slot count:
+
+- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
+  +110 MiB per slot, 440 MiB for four (already in the arithmetic above).
+- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
+  **12 rows**, still inside the 25-row bound at which the `wht6` wire is
+  bit-exact — so four slots cost nothing in wire exactness.
+- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
+  58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
+  bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
+- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
+  (200,000), which did not change; only the concurrency did.
+
+Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
+(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
+cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
+~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
+26,071 MiB slab and does not come back by quantizing KV.
+
+### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
+
+The sizing law's prediction, checked against the engine's own arithmetic:
+predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
+
+| Group | Page / state | Size | Tokens |
+|---|---|---|---|
+| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
+| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
+| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
+| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
+
+`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
+the worst case` — the capacity contract, in the engine's words. The rest of the
+boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
+/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
+2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
+4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
+lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
+
+Two ledger lines that confirm the derived consequences: the sampler prints
+**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
+`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
+(58.50 KiB, half of the bf16 entry).
+
+### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
+
+Frank's call, twice in one evening. Both are one-env-pair changes, and both were
+read off the boot ledger rather than assumed.
+
+**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
+3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
+was backing 858,969 tokens — 43 % more window than the contract asks for.
+Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
+1,792 MiB to the slab and drops the all-reduce to 9 rows.
+
+**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
+`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
+--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
+the ENGINE's own floor, with the budget table calling the surplus `throughput on
+the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
+`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
+
+| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
+|---|---|---|---|---|
+| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
+| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
+| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
+| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
+
+The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
+~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
+slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
+plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
+the card is throughput (the ledger shows the transfer directly: slab
+1,313 slots, pinned pool −1,313).
+
+The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
+for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
+30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
+**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
+it is a capacity decision, not a tuning one.
+
+Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
+of **199,984 tokens** each (599,952 tokens) were accepted and completed on
+`/v1/completions`; a fourth full-height request is refused as "too large to ever
+fit" — radiance never truncates.
+
+### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
+
+Frank's challenge — "why are you not tracking decode?" — was right, and the
+answer is a number now. 512 generated tokens (the bench law), greedy, cold
+prompts, this shape (3 x 200,000, fp8, slab 25,600):
+
+```
+1-stream  43.5k: decode  99.0 t/s  (prefill 4,042, accept 0.952)
+1-stream   197k: decode  93.3 t/s  (prefill 4,826, accept 0.938)
+3-stream  43.5k: decode 113.9 / 43.3 / 10.9 t/s   wall 83.4s
+3-stream   197k: decode  88.9 / 16.3 / 64.6 t/s   wall 122.4s
+```
+
+Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
+93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
+prefill, not tokens-per-second. **Concurrency is where the slot count is
+paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
+at 197k against 99 single-stream, and the split is a lottery
+(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
+to whoever it reached first, so one stream finishes at 113 t/s while its
+sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
+number shows it.
+
+Corpus caveat, stated at the claim: this probe's filler is a repetitive
+ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
+table below; these rows flatter decode relative to that table and are the
+baseline for *this* corpus, not a replacement for those rows.
+`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
+tokens, which is why a day of shape work priced only prefill).
+
+### Recovery evidence
+
+```text
+Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
+  is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
+  204800 MiB (52428800 pages)
+Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
+  204800 MiB clears the pinned-ledger floor 150000
+Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
+```
+
+`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
+the boot, i.e. the root `ExecStartPre` hook did the write the service process
+(User=dank) cannot.
+
+### Capacity probe (2026-10-13, live service)
+
+`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
+from the deployed env on the host, so the probe cannot drift from the service:
+it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
+of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
+carrying a marker whose first word is unique.
+
+| Stream | prompt | wall | prefill | retrieval |
+|---|---|---|---|---|
+| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
+| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
+| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
+| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
+| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
+
+All four answered their own marker; nothing was preempted, refused, or truncated
+(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
+prefill equals the single-stream rate measured in the same probe (3,397 t/s at
+43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
+ceiling and the engine never ran short of KV. Prefix cache intact: the same
+43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
+
+The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
+make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
+rate). This probe answers one question only: does the endpoint owe four
+full-height conversations at once, and answer each one exactly. It does.
+
 ## Gates (measured at cutover)

 - **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
@@ -2,13 +2,14 @@

 **Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
 (digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
-driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
-`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
-(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
-the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
-experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
-47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
-2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
+driving `podman run` directly, root systemd unit, :7770):
+`ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
+(`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
+TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
+(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
+30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
+table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
+froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
 and frank-archon-vllm-shapeG-20261003),
 and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
 (frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
@@ -36,10 +37,10 @@ graph LR
 gfx1201, 32 GiB each]
        B2[radiance StillDeadcode, digest-pinned image
 podman run, frank-archon, port 7770]
-        C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
-2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
-PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
-2 x 200000, froggeric v22.5]
+        C[Swift1.5-Qwen3.8-Flash-Next GPTQ w4 .rad
+2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
+PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
+4 x 200000, froggeric v22.5]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -155,12 +156,24 @@ frank-zealot, port 7771]
 - [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
  Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
  1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
-  the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
-  UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
-  462,752 tokens), weights slab 26071 → 22208, host pool 44928,
-  `--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
-  dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
-  NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
+  the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
+  pair re-cut twice** — `--kv-cache-dtype fp8` first at
+  `--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
+  with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
+  raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
+  settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
+  1 GiB reserve invented. fp8 is what funds the slots — the pool must back
+  `seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
+  Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
+  **25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
+  prediction that the slab was a threshold below 26 GiB died on its own
+  measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
+  arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
+  and the 4-slot fp8 shape are the documented rollbacks.
+  **2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
+  itself** — the 2026-10-12 power failure
+  reset the knob to its default (125,988 MiB) and the endpoint refused to boot
+  6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
 - [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
  w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
  (root systemd unit, port 7770): the StillDeadcode **radiance** engine
@@ -0,0 +1,199 @@
+---
+type: reference
+title: LLM Usage Attribution
+description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.
+tags: [llm, observability, auth, gateway, litellm, tracking, identity]
+timestamp: 2026-10-14T00:00:00-07:00
+---
+
+# LLM Usage Attribution
+
+Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the
+fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web
+citations; live-fleet measurements marked **(measured)**.
+
+## Baseline: what the fleet answers today
+
+- All three endpoints are `auth: none`, plain HTTP, LAN-only
+  (`models.yml` providers; no vhost on 7770/7771).
+- **(measured)** Live `/metrics` label census: `frank-archon` exports
+  `vllm:*`+`radiance:*` with labels only `{model_name, engine, le,
+  finished_reason, position, class}` — zero identity labels.
+  `frank-dragoon` exports `halogen:*`+`llamacpp:*`.
+  `frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes
+  `--metrics`, so the Prometheus job is structurally never-green
+  (`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array.
+- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing
+  only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki,
+  so what lands there has no identity either.
+- Caddy has no `log` directive anywhere → no per-request access logs on the
+  OpenWebUI vhost.
+- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER
+  identity exists nowhere.
+
+## The identity layers (pick the layer, get its resolution)
+
+| # | Layer | Identity | Answers |
+|---|---|---|---|
+| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN |
+| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name |
+| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software |
+| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected |
+| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who |
+| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session |
+| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host |
+
+Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they
+prove nothing about a human (halogen's own docs warn this verbatim:
+"they group requests, and they prove nothing about who sent them").
+
+## Catalog
+
+### A. Engine-native (per-consumer identity inside the inference server)
+
+| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes |
+|---|---|---|---|---|---|
+| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) |
+| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) |
+| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed |
+| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. |
+| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap |
+| Ollama | single key | request log | `ollama:` no key label | none | not in fleet |
+
+Wire fields usable from clients today (all client-declared): OpenAI
+`user`, `safety_identifier`, `prompt_cache_key`, Messages
+`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions
+(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`).
+
+### B. LLM gateways (front door with virtual keys — the only layer with
+per-consumer TOKEN counts)
+
+| Solution | Identity | Usage lands | License | Footprint |
+|---|---|---|---|---|
+| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres |
+| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB |
+| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB |
+| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB |
+| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB |
+| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ |
+| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native |
+| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy |
+| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge |
+
+([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys),
+[prometheus](https://docs.litellm.ai/docs/proxy/prometheus),
+[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/),
+[Bifrost](https://github.com/maximhq/bifrost),
+[Kong AI](https://developer.konghq.com/index/ai-gateway/))
+
+### C. Auth proxies / identity infrastructure (identity without token counts)
+
+- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/
+  client-credentials → header mutation (`X-User: {{ .Subject }}`) +
+  decision logs.
+- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials,
+  `--pass-user-headers` → `X-Forwarded-User`; ~100 MiB.
+- **Authentik itself**: proxy-provider custom headers (already injects
+  `X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves
+  the pattern on this host); event log + webhook forwarding.
+- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields —
+  Caddy already fronts both hosts.
+- **Smallstep step-ca**: short-lived per-device certs, ACME-style.
+- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets).
+- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs,
+  headscale for self-hosted control. Best for off-LAN laptops.
+- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar.
+
+### D. LLM observability / tracing (prompt-level per-user audit)
+
+| Platform | Identity | Content | Self-host | Shape |
+|---|---|---|---|---|
+| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK |
+| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself |
+| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector |
+| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse |
+| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP |
+| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel |
+| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — |
+
+OTel GenAI semconv (Development-stability): `gen_ai.operation.name`,
+`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/
+`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id`
+from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via
+external SDK — **the fleet's engines are OTel-silent at request level**,
+which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.
+
+### E. Network/host passive attribution (no client cooperation)
+
+- **Pi-hole v6** (already on Dankclaw): query log API per device,
+  `pihole6_exporter` + query-logger→Loki (GPLv3 exporter).
+- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the
+  auth chain it carries `X-Authentik-Username` for browser users.
+- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API
+  (`/api/v1/analytics/*`) — per-USER identity exists here already (MIT).
+- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771),
+  ship via audisp→Loki.
+- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow,
+  bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped
+  for single hosts, but agents.plugin-style tools work.
+- **nftables named counters** per consumer IP (bytes; reload resets).
+- **netdata apps.plugin**: per-process network bytes, auto-discovered.
+- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is
+  plaintext HTTP today → use the HTTP **User-Agent census** instead (free
+  from any access log).
+- **DHCP/ARP census** (leases + `ip neigh`): device roster.
+
+## Phased plan for this fleet
+
+**Phase 0 — repo-only, no new services**
+1. frank-zealot: add `--metrics` (fixes the measured 501 + the
+   structurally-red Prometheus job + the firing `LlmEndpointDown`).
+2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) +
+   mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to
+   `all` if prompt audit is wanted) + Alloy ship of `turns/`.
+3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed
+   build); radiance: probe for `--api-key` support (contract facts are
+   silent — add to `fleet-probes`).
+4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider
+   schema supports apiKey (pi-era conversion note).
+5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON →
+   Loki).
+   Limits: keys gate access, and only the engines' request records (halogen)
+   see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.
+
+**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.**
+One virtual key per consumer (omp-<workstation> ×N, openwebui, hermes,
+agents, bench). Per-key/user/team Prometheus labels into the existing
+Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per
+key. This is the ONLY layer that yields tokens-per-who without engine
+cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone
+the proxy-native-observability alternative.)
+
+**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM
+callback: per-user traces, sessions, cost; masking + retention knobs.
+
+**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device;
+Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the
+agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser
+users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline
+can forward `X-Forwarded-User` so the gateway sees them too.
+
+## Cautions
+
+- Client-declared identity (keys, `user`, headers) groups traffic; it does
+  not authenticate humans. Authenticated identity = Authentik (browser) /
+  mTLS (device) / Tailscale (node).
+- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as
+  identity labels on the LAN, not secrets, until TLS/mTLS at the door.
+- Prompt-content logging stores a plaintext copy of everything — set
+  retention caps (halogen GIB/days; Langfuse masking; Loki retention) and
+  gitignore the record dirs.
+- Household-scale cardinality is safe for per-key Prometheus labels;
+  the engines' refusal of key labels is a multi-tenant concern, not ours.
+
+## Related
+- [[Fleet Audit and Posture]] — monitoring gaps census
+- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus
+  pipes every sink above uses
+- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0
+  edits touch)
@@ -23,6 +23,9 @@ Procedures for operating the fleet. The LLM-specific pages live under
  posture, whole-fleet audit, bulk redaction, stash audits
 - [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
  login shells, GUI-hang diagnosis over ssh
+- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
+  engine-native identity, gateways, tracing, network/host attribution, and
+  the phased plan

 ## Related
 - [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
@@ -134,6 +134,9 @@ skill-cluster. See [log.md](log.md) for the wave plan and results.
  posture, whole-fleet audit, bulk redaction
 - [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
  GUI-hang diagnosis over ssh
+- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
+  engine-native identity, gateways, tracing, network/host attribution,
+  phased plan

 **Architecture:**

@@ -1,5 +1,99 @@
 # Bundle Change Log

+## 2026-10-13
+- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
+  power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
+  from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
+  below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
+  preflight named the refusal and exited 1, and `Restart=always` re-ran it
+  **6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
+  container, no engine log. Law adopted: **a sysfs raise is a countdown to the
+  next reboot**, so the gate raises the knob itself. New
+  `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
+  re-verify, then refuse by name) and is also the unit's
+  `ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
+  and the knob is root-owned 0644, which is what the first attempt (a plain
+  write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
+  records the host ceiling.
+- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
+  `--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
+  measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
+  at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
+  to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
+  fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
+  `models.yml` needed no change: the per-request window is unchanged, only the
+  concurrency.
+- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
+  backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
+  sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
+  of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
+  (now reading the shape from the deployed env): four concurrent 196,960-token
+  chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
+  aggregate prefill, no preemption; prefix cache verified (repeat answered in
+  1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
+  homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
+- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
+  4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
+  was funding 858,969 tokens — 43 % over the contract. Then slab
+  24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
+  OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
+  96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
+  real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
+  invented on 2026-10-11. The pair now spends 30,720 and settles
+  `unclaimed 477.30 MiB` (cards ~336 MiB free).
+- **The prediction shipped with that arm was falsified by its own measurement,
+  which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
+  below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
+  other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
+  cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
+  193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
+  pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
+  points inside noise of each other are not a plateau. The cliff is above
+  25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
+  slots (26,112 + 5,120 > the ceiling) and free at two.
+- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
+  accepted and completed (599,952 tokens resident), ledger
+  `backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
+  law's 608,027. A fourth full-height request is refused as too large to ever
+  fit — radiance never truncates. Probe law: the oversized-prompt counter is
+  FREE (the refusal fires when PROMPT tokens alone exceed the window —
+  `prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
+  window is ~514,286 characters at the measured 2.5714 chars/token, so a
+  calibration probe below that pays a real prefill instead of reporting.
+- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
+  exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
+  conversation. Untested arm; capacity decision, not tuning.
+- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
+  owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
+  measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
+  own test. 371/371 tests and e2e 33/33 on both deploys.
+- **Power-failure casualties, NOT fixed (outside the archon window):**
+  `mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
+  raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
+  media libraries on it read empty. `frank-job-digest.service` (frankbook
+  job-hunt digest) is in `list-units --state=failed`.
+- **The slab is a threshold, not a slope** — measured, same method (probe A
+  leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
+  22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
+  bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
+  (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
+  would clear it (pool 3,072 + slab 27,897) remains untested — and the
+  paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
+  not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
+- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
+  cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
+  is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
+  give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
+  The slot count is paid in decode concurrency — a fused-step lottery, not a
+  fair share — which is exactly the axis the shape hunt was blind to (probe
+  leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
+  is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
+  table, so these rows are the baseline for this corpus, not a replacement.
+- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
+  tokens, per-stream + aggregate) so no future shape change can be priced on
+  prefill alone.
+
+
 ## 2026-10-10
 - **Fleet service rename: the StarCraft names became the SERVICE names.**
  The 2026-10-09 wire alias moved onto every repo artifact: units, serve
@@ -9,11 +9,12 @@
 # flags alone do not say: the served window clears the harness worst case,
 # the KV pool backs seqs x max-model-len at the declared cache dtype (an
 # engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
-# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
-# the serve script ENFORCES it, the served window is what models.yml must
+# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
+# it and RAISES the knob itself when a reboot reset it (the cap's default is
+# half of RAM), the served window is what models.yml must
 # quote, the host-networking contract (no --publish), the image gate on a
 # name@sha256 ref, the .rad payload gate with its size floor, the
-# chat-template override mount, the self-healing law (the run returns before
+ # chat-template override mount, the self-healing requirement (the run returns before
 # the engine is up, so the serve script must wait on the container for
 # Restart=always to mean anything), the deployer's false-green MainPID check,
 # the vendored hfget fetch path (no token for this ungated repo), and the
@@ -24,6 +25,7 @@ load helpers

 UNIT="$REPO_ROOT/scripts/frank-archon.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
+TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
@@ -31,9 +33,20 @@ BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
 HFGET="$REPO_ROOT/scripts/hfget.sh"
 TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"

+# A PATH with `cat` (the gate reads knobs with it) and NO `sudo`, so the
+# privileged fallback cannot rescue a write the test means to fail.
+nosudo_bin() {
+    local bin="$BATS_TEST_TMPDIR/nosudo-bin"
+    mkdir -p "$bin"
+    ln -sf "$(command -v cat)" "$bin/cat"
+    ln -sf "$(command -v bash)" "$bin/bash"
+    echo "$bin"
+}
+
@test "frank-archon unit, serve script, env, fetch and build scripts and template exist" {
    [ -f "$UNIT" ]
    [ -x "$SERVE" ]
+    [ -f "$TTM_LIB" ]
    [ -f "$ENV_FILE" ]
    [ -f "$DOWNLOAD" ]
    [ -x "$BUILD" ]
@@ -120,37 +133,41 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
    grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
    ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
    [ "$ml" -ge 177356 ]
-    # Concurrency: 2 slots x the served window is the capacity contract, and
-    # the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
+    # Concurrency: 3 slots x the served window is the capacity contract, and
+    # the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
    # under the 25-row bound at which the wht6 wire stays bit-exact. The
-    # "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
-    # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
-    # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
-    # tied inside the +-3% noise floor.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    # "seqs 2 cannot boot" is the failure mode of a runtime that compiles
+    # kernels specialized to the batch shape — it refuses a slot count it has
+    # never compiled for. This engine is shape-agnostic: seqs 2 boots, and
+    # measured decode ties seqs 2 and 3 inside the +-3% noise floor — slots
+    # buy concurrency, not speed.
+    grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
    # expert_tiered placement: vram_weights + the pinned host pool serve the
    # per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
-    # KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
-    # only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
-    # in host RAM per Frank's law.
+    # KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
+    # only bf16|fp8 — no per-tensor K/V split), and the PLE n-gram table is
+    # pinned in host RAM so the cards read it without page-faulting mid-token.
    grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
    # Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
    grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
    # The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
    # under the ttm pages cap, and the env states the floor the serve
-    # script enforces (>= 140,000 MiB — the current cap ~123 GiB fails it,
-    # which is the point).
+    # script enforces (>= 140,000 MiB — the kernel's own default cap, half of
+    # RAM, fails it, which is the point) plus the value it raises the knob to
+    # when a reboot reset it (the target must clear the floor with slack).
    grep -q '^FRANK_ARCHON_TTM_MIN_MIB=' "$ENV_FILE"
    ttmm=$(awk -F= '/^FRANK_ARCHON_TTM_MIN_MIB=/{print $2}' "$ENV_FILE")
+    ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
+    [ "$ttmt" -gt "$ttmm" ]
    pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
    ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
@@ -188,15 +205,20 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
 }

@test "the per-card VRAM pair stays inside the measured card ceiling" {
-    # Measured on Dankserver 2026-10-11: card 31,856 MiB, a boot consumes
-    # 1,710 MiB OUTSIDE both budgets (driver context, the 990 MiB activation
-    # arena, hardware queues, headroom, step staging) and the fp8 pair settled
-    # at 1,379 MiB free. Claiming more than 29,120 MiB in total is an
-    # allocation failure at graph capture, not a warning — so the KV pool and
-    # the weights slab are one budget, and this seam is the guard.
+    # The engine's OWN budget table, not a hand-picked reserve. Boot of
+    # 2026-10-13 at the pair 24000/5120: card 32,624 MiB (31.86 GiB), budgets
+    # claimed 29,120, `3.42 GiB is neither budgeted nor accounted — the
+    # driver's context, the activation arena and --gpu-headroom-mib come out of
+    # it; anything past those is throughput on the floor`, and `card 0 settled
+    # at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and
+    # 113.19 MiB kept for the static region)`. Claimable reserve above the
+    # engine's floor: 2,058 - 209 = 1,849 MiB, so the ceiling is 30,969 MiB.
+    # The 2026-10-11 bound of 29,120 ("keep >= 1 GiB free", card read as 31,856
+    # MiB) was MY reserve, which the engine never asked for; the shipped pair
+    # spends the difference and lands ~458 MiB above the floor.
    w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
    kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
-    [ $(( w + kv )) -le 29120 ]
+    [ $(( w + kv )) -le 30969 ]
 }

@test "the serve script wires the engine flags from the env" {
@@ -272,6 +294,83 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
    grep -q -- '--device /dev/dri' "$SERVE"
 }

+@test "the ttm gate raises a reset pages cap and refuses when the write fails" {
+    # The cap is a RUNTIME knob: the kernel default is half of RAM, so every
+    # reboot resets it (measured: the 2026-10-12 23:17 power failure on
+    # dankserver took it 200 GiB -> 125,988 MiB, and the preflight refusal was
+    # re-run 6,919 times by Restart=always over ~20 h). Behaviour required of
+    # the gate, in both directions:
+    #   knob below the floor + writable  -> cap ends at the target, start passes
+    #   knob below the floor + unwritable -> cap stays low, start REFUSES by name
+    local dir="$BATS_TEST_TMPDIR/ttm"
+    mkdir -p "$dir"
+    export FRANK_ARCHON_TTM_SYS_DIR="$dir"
+    export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
+
+    echo 32253039 > "$dir/pages_limit"    # 125,988 MiB: the kernel default here
+    run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
+    [ "$status" -eq 0 ]
+    [ "$(cat "$dir/pages_limit")" = "52428800" ]   # 204,800 MiB in 4 KiB pages
+
+    echo 32253039 > "$dir/pages_limit"
+    chmod 444 "$dir/pages_limit"          # the write refused: cap cannot move
+    # The refusal case must not have `sudo` on PATH: the gate's privileged
+    # fallback runs `sudo -n`, and on a passwordless host root can write a 444
+    # file, which would silently test nothing.
+    run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"ERROR: ttm pages cap is 125988 MiB"* ]]
+    [ "$(cat "$dir/pages_limit")" = "32253039" ]
+}
+
+@test "the ttm cap counts only the knobs the kernel exposes" {
+    # An absent knob is not a zero: a zero-cap reading would refuse a healthy
+    # host. The effective cap is the MIN over the knobs that exist; pages_max,
+    # when a kernel exposes one, is the binding one.
+    local dir="$BATS_TEST_TMPDIR/ttm2"
+    mkdir -p "$dir"
+    export FRANK_ARCHON_TTM_SYS_DIR="$dir"
+    export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
+
+    echo 51200000 > "$dir/pages_limit"    # 200,000 MiB, no pages_max knob
+    run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
+    [ "$status" -eq 0 ]
+    [ "$output" = "200000" ]
+
+    echo 30000000 > "$dir/pages_max"      # 117,187 MiB: now the binding knob
+    run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
+    [ "$output" = "117187" ]
+
+    chmod 444 "$dir/pages_limit" "$dir/pages_max"   # both writes refused
+    run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"ERROR: ttm pages cap is 117187 MiB"* ]]
+    [ "$(cat "$dir/pages_max")" = "30000000" ]
+}
+
+@test "the gate runs as a root pre-start hook, where the knob write is legal" {
+    # The knob is root-owned 0644 and the service runs as User=dank, so the
+    # gate CANNOT raise it from the service process. systemd's `+` prefix on
+    # ExecStartPre runs as root even under User= — the canonical privileged
+    # path — and the gate's own --raise mode is what that line calls.
+    grep -q '^ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise$' "$UNIT"
+
+    # Behaviour of that entry point: sourced env values, knob raised, exit 0.
+    local dir="$BATS_TEST_TMPDIR/ttm3" envf="$BATS_TEST_TMPDIR/ttm3.env"
+    mkdir -p "$dir"
+    printf 'FRANK_ARCHON_TTM_MIN_MIB=150000\nFRANK_ARCHON_TTM_TARGET_MIB=204800\n' > "$envf"
+    echo 32253039 > "$dir/pages_limit"
+    run env FRANK_ARCHON_ENV_FILE="$envf" FRANK_ARCHON_TTM_SYS_DIR="$dir" \
+        bash "$TTM_LIB" --raise
+    [ "$status" -eq 0 ]
+    [ "$(cat "$dir/pages_limit")" = "52428800" ]
+
+    # Usage is a named failure, not a silent no-op.
+    run bash "$TTM_LIB"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"--raise"* ]]
+}
+
@test "the serve script clears stale containers and self-heals via podman wait" {
    # A crashed unit can leave the container name registered forever after;
    # the legacy identities hold the same GPUs and die here too. Then block
@@ -11,8 +11,9 @@
 # vision file names AND byte counts inside the /models mount, and the
 # tokenizer's flat-file shape (a HF cache snapshot's symlinks dangle inside
 # the container). A serve gate and a deployer gate that disagree is the
-# 2026-10-01 crash-loop; both sides are pinned here, plus the one engine
-# law the card is emphatic about: HALOGEN_SPEC_ADAPT=0 — the default
+ # crash-loop class this file exists to kill: both sides are pinned here,
+ # plus the one engine rule the card is emphatic about: HALOGEN_SPEC_ADAPT=0 —
+ # the default
 # adaptive policy silently disables the MTP head per request and HALVES
 # decode.
 #
@@ -111,9 +112,10 @@ good_env() {
@test "the head stays on: adaptive spec must stay 0 and the window shape pinned" {
    # The card is emphatic: the default adaptive policy turns the MTP head
    # off for whole requests after low acceptance and HALVES decode. The
-    # two-slot law does NOT carry over from the 27B engine — on this line
-    # batching and speculation coexist (the card's quickstart ships 2
-    # slots WITH the head), so 2 is concurrent capacity, not a tax.
+    # two-slot coupling of the 27B engine — which disabled drafting whenever a
+    # second slot existed — does NOT carry over: on this line batching and
+    # speculation coexist (the card's quickstart ships 2 slots WITH the head),
+    # so 2 is concurrent capacity, not a tax.
    grep -qE '^HALOGEN_SPEC_ADAPT=0$' "$ENV_FILE"
    grep -qE '^HALOGEN_KV_SLOTS=2$' "$ENV_FILE"
    # The cap/timeout pair is coupled (FLAGS.md table): a request at the
@@ -178,8 +180,9 @@ good_env() {
 }

@test "the serve script refuses a checkpoint of the wrong byte count" {
-    # A truncated curl exit-0 artifact must not boot; the Hub's byte
-    # count is the contract (fleet fetch law).
+    # A truncated curl exit-0 artifact must not boot: curl exits 0 on an HTTP
+    # error and writes the error body as the file, so the Hub's declared byte
+    # count is the contract that catches it.
    podman_stub
    good_env
    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
@@ -280,7 +283,7 @@ $(cat "$TEST_TMP/serve.log")"
    ! grep -q 'PODMAN wait' "$TEST_TMP/serve.log"
 }

-@test "the shipped env keeps the harness fit law inside the ceiling" {
+ @test "the shipped env keeps the harness fit budget inside the engine ceiling" {
    # 262144 (the checkpoint's native window) is the per-slot hard ceiling
    # (a longer prompt is a 400 naming the limit, never a silent
    # truncation). The harness clamps max_tokens = min(45056, 217088 − est
Author	SHA1	Message	Date
Frank	2e86445fc5	comments: replace law idiom with the mechanism it encoded Every comment that cited a named law, an incident date, or an internal nickname now states the underlying mechanism and why it matters, generically: curl exit-0 artifacts and byte-count gates, argv-in-/proc secret exposure, tag mutability vs digest pinning, shape-specialized kernels refusing uncompiled slot counts, MTP heads as file tensors no flag can supply, drafter/slot coupling as an engine property, pkill -f matching its invoker, pinned-RAM pairing with LimitMEMLOCK, one-encoding-per-table, the short-run first-token noise floor, and the protected-expert rule. No behavior change: comment/string text only; suite green (371 tests).	2026-10-14 13:25:41 -07:00
Frank	cf6319aad7	docs(operations): LLM usage-attribution research catalog Deep-research reference for per-consumer tracking of the fleet endpoints: measured baseline (no identity anywhere; frank-zealot /metrics 501), the seven identity layers, engine-native/gateway/tracing/network catalogs with citations, and a phased adoption plan.	2026-10-14 13:25:41 -07:00
Frank	a9d45e7012	frank-archon: the slab is a threshold, and decode is where slots cost Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.	2026-10-13 20:51:32 -07:00
Frank	2581b8d4b6	docs(frank-archon): the slab is a slope, and the ceiling was mine not the engine's Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 -> 3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %, wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s. Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at 29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable). The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the number, so the next resize starts from the engine's arithmetic. Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism, the pinned pool is the slow path. Capacity contract live-verified at 3 slots: 3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too large to ever fit. Probe law learned: the oversized-prompt refusal names the prompt's token count BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.	2026-10-13 20:33:02 -07:00
Frank	09a0251d2d	llm(frank-archon): claim the card's settling slack, slab 24000 -> 25600 The 2 GiB/card that \`rocm-smi\` shows free is not idle: the engine's budget table calls it driver context + the 896 MiB activation arena + 96 MiB headroom, and says the rest is `throughput on the floor`. Its own settle line is the number that matters — `card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)` — so 1,849 MiB above the ENGINE's floor is claimable and the real ceiling is 30,969 MiB, not the 29,120 my 1 GiB reserve invented on 2026-10-11. Shipped pair 25600/5120 = 30,720: +1,600 MiB of slab, ~458 MiB above the floor. The slab is already saturated at 24,000 (mover took all of it, 451 KiB slack), so this is the only direction left — KV is at its 3-slot floor (5,063 needed of 5,120). Named prediction, because today's two points say the slab is a THRESHOLD and not a slope (22,208 -> 3,397 t/s, 24,000 -> 3,290 t/s at 43.5k single-stream): this arm buys ~1,600 resident expert slots and ~0 prefill. The point that moves prefill is 26,071 (2026-10-11: 4,379 t/s), fundable only at two slots. Prefill moving > +-3 % here falsifies the threshold reading. Seam: the ceiling gate now derives from the engine's budget table (30,969) and is the single owner of the pair arithmetic; the duplicate I put in the shape test is removed.	2026-10-13 20:18:41 -07:00
Frank	066813ea95	llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB Frank's call: three slots, and spend the KV the third slot doesn't need on resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`, `--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured 29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %. The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the contract buys nothing. At fp8 the slab curve measured on this host is 22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so 24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check. The cost is named, not hidden: a FOURTH full-height conversation no longer fits (4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit — radiance never truncates. Reverting to four slots is one env pair. Seam: two new arithmetic gates, because both failures are boot-then-400 shape faults that no flag check catches — the pair must respect the per-card ceiling, and the pool must back seqs x max-model-len at the dtype's measured rate plus the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of prefix checkpoints).	2026-10-13 20:00:39 -07:00
Frank	7daa886dd6	docs(frank-archon): work-log entry and the 4-way capacity probe Probe on the live service: four concurrent 196,960-token chats, 4/4 exact marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the single-stream ceiling, so the four streams shared one pipeline and the KV pool never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008. Recorded with the caveat that 8 generated tokens do not make a decode rate.	2026-10-13 19:30:46 -07:00
Frank	bae78e7dae	docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law, the shape arithmetic, the boot ledger that confirms it: 843,276 tokens = 4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise lesson under Key Lessons, and the pinned-RAM paragraph now points at scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.	2026-10-13 19:28:25 -07:00
Frank	a6f55f4cc6	fix(frank-archon): raise the ttm knob from a root ExecStartPre — User=dank cannot write it a9da65e put the raise inside the serve script, which runs as `User=dank`: the knob is root-owned 0644, so `-w` is false and the write silently failed — the gate printed its refusal on a cap it had just failed to move. The privileged path is systemd's `+` prefix: `ExecStartPre=+scripts/ttm-preflight.sh --raise` runs as root even under `User=`, before ExecStart, so `Restart=always` can never outrun it. The lib is now also executable (`--raise` sources the env file and verifies), and keeps a `sudo -n` fallback for a human running the gate from a shell. Seam: the refusal cases run with a PATH that has `cat`+`bash` and NO `sudo` (root writes a 444 file, so a passwordless dev box would have tested nothing), and both now assert the gate's own exit 1, not merely non-zero. New seam covers the unit line and the `--raise` entry point end to end (writable knob -> raised, no args -> usage exit 2).	2026-10-13 19:20:33 -07:00
Frank	a9da6589a6	fix(frank-archon): `:" guard needs a space —` :"${VAR:?}"` runs :204800 The TTM_TARGET_MIB guard I added on `1a6a78e` was written `:"...` (no space), so bash parsed the expansion as one word and executed `cmd :204800` as the command `:204800`: exit 127 `command not found` on every start, before the ttm gate ran. Caught on the deploy: the unit crash-looped 34 times in ~4 min with exactly that journal line. probe-frank-archon-kv.py now reads the slot count, window and KV dtype from the deployed env on the host, so probe B exercises the capacity contract that is actually live (4 x 200,000 today) instead of a hardcoded pair, and each probe's marker starts with a unique word because show() scores the hit on that word.	2026-10-13 19:16:48 -07:00
Frank	1a6a78ef73	fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve script's preflight named the refusal and exited 1; Restart=always re-ran it 6,919 times over ~20 hours and frank-archon served nobody. scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob itself before refusing: the unit is root, the knob is a 0644 root module parameter, and the cap is a runtime knob whose default is half of RAM — so a manual sysfs write is not a repair procedure, it is a countdown to the next reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate re-reads the cap after writing and only then refuses, so a knob the kernel will not move is still a named failure. Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8 (quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the 25-row bound where the wht6 wire is bit-exact.	2026-10-13 19:09:58 -07:00