governance: bench is removed from the model-swap procedure

Model changes verify by driving the live service — boot ledger, per-request timings, one real request per harness role. No bench window is ever scheduled for a swap. The stop-and-mask policy stays for deliberate standalone bench windows (it is host protection, not a swap step). Codified in AGENTS.md (policy lead + summary row), Model Swap and Rollback (No bench step law, §8 close-out source, Related rescope).
frank-archon: the slab is a threshold, and decode is where slots cost
2026-10-15 03:07:46 -07:00 · 2026-10-13 20:51:32 -07:00 · 2026-10-13 20:33:02 -07:00 · 2026-10-13 20:18:41 -07:00 · 2026-10-13 20:00:39 -07:00 · 2026-10-13 19:30:46 -07:00
11 changed files with 913 additions and 169 deletions
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
+| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
 | Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
 | Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
@@ -226,6 +226,13 @@ archive.
 ## Benchmarking Policy (MANDATORY)
 **Benchmarking is NOT part of the model-swap procedure** (2026-10-15). A
 model change is verified by driving the live service — boot ledger,
 per-request `timings`/`Server-Timing`, one real request per harness role
 (`serverAppConfigs-docs/LLM/Model Swap and Rollback.md` §6/§8). No bench
 window is scheduled for a swap, ever. The rest of this policy governs
 *deliberate* bench windows (flag A/Bs, shape sweeps), which are separate
 operations on a live host:
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
@@ -525,21 +532,43 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
 Rollback: revert the swap commit — the fp8-native `.rad`
 (`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
-**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`,
+**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
 `--max-model-len 200000` (the harness contextWindow must quote this),
 `--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
-partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
+partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
 replaced `--expert-vs-cache-ratio`; the two are ONE budget —
-`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
+`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
-consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
+32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
-**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`,
+209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
-`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
+claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
-disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
+hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
-v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
+because it is the only dtype that funds the slot count at this window**,
-`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16
+`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
-tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's
+8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
-386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool
+**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
-10,375 slots, cards settled 1.85 GiB free.
+(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
 ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
 all at 200000 tokens at once, which is the worst case` — `kv_attn` page
 26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
 `kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
 (3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
 (3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
 563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
 every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
 the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
 settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
 4.22 GiB to the expert cache. Prefill tracks this slab only up to a
 **threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
 (±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
 (the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
 (26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
 not a slope.
 **Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
 single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
 4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
 aggregate**, so the slot count is paid in decode concurrency (a fused-step
 lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
 Shape notes carried from the fp8-native era and unchanged: wht6 gains,
 MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
@@ -551,35 +580,52 @@ on the fp8 shape (512-token streamed probe). The campaign also
 measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
 accept 0.7246).
-**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` —
+**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
-one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what
+`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
-frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on
+llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
-both halves. Measured per rank from the two boot ledgers: `kv_attn` (13
+equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
-layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13
+the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
-layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not
+KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
-attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at
+block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
-bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms
+and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
-that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB
+dtype-independent terms that live in the same pool (~110 MiB a sequence of
-of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` =
+recurrent state + 448 MiB of prefix checkpoints):
-462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is
+`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
-paid twice on this engine, and half-refunded: idle KV VRAM is also the expert
+engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
-cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the
+gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
-fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots
+the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
-× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to
+bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
-the retired tcclaviger fork's torch.compile, not radiance) and it also drops
+where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
-the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is
+half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
-bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
+pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
-buffers + arena onto the weights side, so `weights + kv` is capped near
+`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
-29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
+25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
 the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
 law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
 weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
 pair, never `--kv-cache-size-mib`.
 Shape notes carried from the fp8-native era and unchanged: wht6 gains,
 MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
 prefix cache, and the rule that a prompt over `--max-model-len` is
 refused, never truncated. What changed: the fp8-native era's NVMe expert
 streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
 host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
 on the fp8 shape (512-token streamed probe). The campaign also
 measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
 accept 0.7246).
 **The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
 pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
 the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
 `RLIMIT_MEMLOCK` — the TTM floor is set just above it.
-`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
+`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
-script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host
+runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
-204,800) refuses to start until both knobs are raised in the stop window
+by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
-(sysfs write first, kernel cmdline + reboot only if read-only).
+before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
 and the preflight re-refused 6,919 times. A manual sysfs write in the stop
 window is a convenience, not the repair; kernel cmdline persistence
 (`ttm.pages_limit=52428800`) is the belt to it.
 **Contract facts** (source-verified): the HTTP listener binds AFTER the
 full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -761,6 +807,17 @@ linked runbooks.
  the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
 - **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
  to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
  2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
  frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
  and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
  `activating (auto-restart)` that exits 1 in 50 ms with no container and no
  engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
  `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
  to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
  after a write the kernel rejected. Never prescribe a manual sysfs write as the
  recovery procedure for a service that can boot before a human reaches it.
 - **Silent RAM corruption outranks every software theory.** frank-archon's
  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
@@ -1107,7 +1164,7 @@ history.
 | Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
 | Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
 | Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
-| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. |
+| Benchmarking | **Not part of a model swap** — swaps verify on the live service. A deliberate bench window: **stop and mask the service first**, bench, unmask, restart, verify health. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
 | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
@@ -3,10 +3,19 @@
 # built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
 # dankserver :7770.
 #
-# SHAPE (2026-10-11): 2 decode slots x 200,000 tokens with an UNQUANTIZED
+# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
-# (bf16) KV cache — the pool is sized for the pair, and the weights slab pays
+# with the per-card budget spent on resident experts: 25,600 MiB of slab +
-# for it (see the KV block). The fp8-native payload this file was written for
+# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
-# is the rollback target and keeps its own trio under the fp8-native runbook.
+# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
 # to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
 # predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
 # is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
 # +25 % at the full window) while KV beyond the contract buys nothing.
 # Consequence, deliberate: a FOURTH full-height conversation does not fit this
 # pair, and the engine refuses it — it never truncates. See the KV block for the
 # arithmetic and the measured dtype law. The fp8-native payload this file was
 # written for is the rollback target and keeps its own trio under the
 # fp8-native runbook.
 #
 # 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
 # pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
@@ -113,18 +122,29 @@ FRANK_ARCHON_TP=2
 # +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
 # engine default.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots: TWO, so the served window is a CAPACITY CONTRACT — two
+# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
 # concurrent conversations of --max-model-len fit the KV pool (see the KV
 # block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
 # tcclaviger vLLM fork's torch.compile shape specialization on
 # query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
 # with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
 # and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
-# (30.9-33.1 t/s, inside the +-3% noise floor). Side benefit: the fused TP
+# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
-# all-reduce is declared over seqs x (1 + spec) = 6 rows, under the 25-row
+# not speed, so a slot costs KV, never decode.
-# bound at which the wht6 wire stays EXACT (single-sequence decode and any
+# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
-# step of <= 25 rows is bit-exact; only busier steps are lossy).
+# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
-FRANK_ARCHON_MAX_SEQS=2
+# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
 # (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
 # steps are lossy).
 # The pool is sized to this contract and nothing more (see the KV block):
 # 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
 # the resident expert slab. The trade is explicit and it is Frank's call:
 # a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
 # ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
 # as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
 # traffic that wants four full-height sessions is a --vram-kv-mib change, not
 # a code change.
 FRANK_ARCHON_MAX_SEQS=3
 # Served window. Native is 262,144; 200,000 is the publisher's production
 # value (compose) and what models.yml's contextWindow must quote — a longer
 # prompt is refused by the engine, never truncated. The engine treats
@@ -145,25 +165,52 @@ FRANK_ARCHON_HOST_POOL_MIB=44928
 # VRAM left unclaimed per card (publisher default 96 MiB).
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
 # VRAM split of claimed memory per card, MiB (0 = derive from the card).
-# THE CEILING, measured on this host 2026-10-11: card 31,856 MiB, a boot
+# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
-# consumes 1,710 MiB OUTSIDE both budgets (driver context + activation arena
+# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
-# 990 + hardware queues 100 + headroom 96 + step staging), and the fp8 pair
+# neither budgeted nor accounted — the driver's context, the activation arena
-# settled at 1,379 MiB free — so `vram_weights + vram_kv <= 29,120 MiB` keeps
+# and --gpu-headroom-mib come out of it; anything past those is throughput on
-# >= 1 GiB free. The pair below is exactly that sum.
+# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
 # 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
 # So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
 # ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
 # ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
 # the engine's; the pair below spends the difference and lands 458 MiB above
 # the engine's floor. `weights + kv <= 30,969` is the gate.
 # THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
 # expert cache's spillover (`mover: 2275 slots can be lent out of the KV
 # cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
 # so a bigger KV pool steals weight residency from the slab AND from the
 # lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
 # a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
-# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. Expect -3..-8%
+# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
-# vs the fp8 control (136 t/s median at 44.5k depth, acceptance 0.7246,
+# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
-# ~190 t/s on a live 512-token probe); re-tune from a boot ledger + probe,
+# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
-# never from that estimate.
+# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
-FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208
+# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
-FRANK_ARCHON_VRAM_KV_MIB=6912
+# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
-# KV cache UNQUANTIZED (bf16). This engine's flag takes ONLY bf16|fp8 — there
+# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
-# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
+# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
 # The margin is thin by design (608,027 predicted tokens against the 600,000
 # contract), so the boot ledger's `backs N sequences all at 200000 tokens at
 # once` line is the referee for THIS pair and the concurrent probe is its
 # regression test.
 # VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
 # the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
 # 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
 # the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
 # 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
 # +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
 # pair landed inside noise of EACH OTHER, which is what made it read as a
 # plateau; two points are not a curve.
 # The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
 # 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
 # slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
 # vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
 # pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
 FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
 FRANK_ARCHON_VRAM_KV_MIB=5120
 # KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
 # ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
 # --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
 # UNSERVABLE here: the rung below bf16 is fp8 on both halves.
 # Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
@@ -177,30 +224,35 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
 # The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
 # copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
 # checkpoint region 8 x 55.99 = 448 MiB. So:
-#   pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
+#   pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
-#   6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
+#   3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
-#   reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
+#   and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
-#   and the ledger says it outright: "backs 2 sequences all at 200000 tokens
+#   three-slot contract with 1.3 % slack, every MiB above 5,063 having been
-#   at once, which is the worst case".
+#   handed to the expert slab instead. The ledger line is the referee ("backs N
-# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
+#   sequences all at 200000 tokens at once") — a pair this tight is read off the
-# sessions, and a third long conversation preempted to the host/disk tiers).
+#   boot, never assumed.
-# Secondary cost: the prefix-cache entry grows with the attention plane
+#   At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
-# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
+#   the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
-# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
+#   11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
-# the same number of parked conversations, out of RAM the host has
+#   slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
-# (104 GiB available with the service loaded).
+#   tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
-FRANK_ARCHON_KV_DTYPE=bf16
+#   sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
 # Secondary saving: the prefix-cache entry shrinks with the attention plane
 # (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
 # ledgers' `kv tier transfer` lines), so the host tier below parks twice as
 # many conversations as it did at bf16 — no re-size needed to park three slots.
 FRANK_ARCHON_KV_DTYPE=fp8
 # Spec decode: the checkpoint's own MTP head at the publisher's depth-3
 # operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
 # flag states it so the ledger records the intent).
 FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
-# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
+# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
-# The host tier is sized in ENTRIES, and one entry doubles with the cache
+# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
-# dtype (58.50 KiB at fp8 -> 117 KiB at bf16, per the ledger's `kv tier
+# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
-# transfer` line): 4096 MiB at bf16 parks ~1.4 sessions of 200k where it
+# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
-# parked ~2.9 at fp8. Doubled to keep the capacity, at RAM the host has
+# parked set fits with the tier as it stands — at RAM the host has (104 GiB
-# (104 GiB available with the service loaded).
+# available with the service loaded).
 FRANK_ARCHON_PREFIX_HOST_MIB=8192
 FRANK_ARCHON_PREFIX_DISK_MIB=65536
 # Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
@@ -211,11 +263,17 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
 # pages_limit (the current kernel exposes NO pages_max; the cap function
 # takes the min over the knobs that exist). Below the plan the load dies
-# mid-pin with an allocator error instead of this named refusal — raise the
+# mid-pin with an allocator error instead of this named refusal.
-# knob first (runtime sysfs write; kernel cmdline fallback + one reboot only
+# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
-# if the file is read-only), see the runbook. Cap raised 2026-10-05 to
+# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
-# 52,428,800 pages = 200 GiB, which still clears this floor.
+# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
 # FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
 # start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
 # scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
 # root unit is privileged; the knob is a 0644 root module parameter), so the
 # recovery is a boot, not a visit to the machine.
 FRANK_ARCHON_TTM_MIN_MIB=150000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
 # multi-flag values as one string; the serve script expands it unquoted on
 # purpose.
@@ -35,6 +35,13 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
 # The `+` prefix runs this as root even though the service is User=dank — the
 # ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
 # unusable below it. After a reboot the knob is back at its default (half of
 # RAM), so the raise must be part of the start, not a note in a runbook: the
 # 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
 # times. The serve script re-verifies the cap after this hook.
 ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
 # systemd stop: wait alone cannot stop the container child; stop it
 # explicitly. No graceful drain contract is claimed (the OpenAI front end
 # holds no queue to drain — in-flight decodes are lost, which a stop
@@ -1,22 +1,30 @@
 #!/usr/bin/env python3
-"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE
+"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
 frank-archon (radiance) service. Run it ON dankserver: it talks to
 127.0.0.1:7770 and builds its prompts from the repo checkout there.
 No scratch container, no mask: the service under test is the service.
    ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
-Three probes, in increasing cost:
+Four probes, in increasing cost:
  A  decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
     Reports the engine's own `timings` (prefill, decode, draft acceptance).
     The bench law applies: >=128 generated tokens, greedy, real text.
-  B  the capacity contract — TWO concurrent chats of ~197k prompt tokens
+  B  the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
-     (~394k tokens resident) each carrying a unique marker at its head. Both
+     the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
-     must answer their own marker. This is what a 2 x 200,000 endpoint owes:
+     host), each carrying a unique marker at its head. All must answer their own
-     two full-height conversations resident at once, exact retrieval in both.
+     marker. This is what the endpoint owes: N full-height conversations
     resident at once, exact retrieval in every one of them.
  C  prefix-cache restore — probe A's cold prompt again, so the disk tier has
     to give the blocks back (cache_n > 0, TTFT collapses).
-
+  D  concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
     the window. This is the cost a slot count imposes on DECODE, which no
     prefill number shows: the fused-step schedule is unfair, so every stream is
     printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
     single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
     window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
     Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
     shape hunt missed this for a whole day because leg B generates 24 tokens.
 PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
 `reasoning_content`, so a chat request whose budget is spent inside
 `<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
@@ -114,23 +122,51 @@ def show(tag, j, wall, expect=None):
    print(line)
 def deployed_shape():
    """The contract is the deployed env, not a constant: slot count and window
    come from the checkout on this host, so the probe cannot drift from the
    service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
    2 x 200,000, and the fp8 era 3 x 200,000)."""
    shape = {}
    with open(f"{REPO}/scripts/frank-archon.env") as fh:
        for line in fh:
            line = line.strip()
            if line.startswith("FRANK_ARCHON_") and "=" in line:
                key, _, val = line.partition("=")
                shape[key] = val.strip().strip('"')
    return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
            int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
            shape["FRANK_ARCHON_KV_DTYPE"])
 def main():
    seqs, maxlen, dtype = deployed_shape()
    print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
    print("health:", json.dumps(get("/health")))
    print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
    deep = corpus(160_000)
    wide = corpus(1_100_000)
-    facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"}
+    # 3.28 chars/token is the measured ratio of this corpus through the
-    bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()}
+    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
    # deploy); 98.5 % of the window leaves the response room under the ceiling.
    chars = int(0.985 * maxlen * 3.28)
    # Each marker's FIRST hyphen-separated word is unique, because show()
    # scores the hit on that word — a shared prefix would score every slot HIT
    # on any answer that says the word.
    words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
    facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
             for i in range(seqs)}
    bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
    print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
-          f"request, x2 concurrent")
+          f"request, x{seqs} concurrent")
-    # A + C first (cheap), B last (the expensive pair).
+    # A + C first (cheap), B last (the expensive set).
    for tag, salt in (("A-cold", "A1-9137"), ):
        j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
        show(tag, j, wall, "zephyr-lantern-7")
    t0 = time.time()
-    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
+    with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
        futs = {s: ex.submit(post, b) for s, b in bodies.items()}
        for s, fut in futs.items():
            try:
@@ -139,6 +175,24 @@ def main():
            except Exception as e:  # a preemption/timeout lands here
                print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
    print(f"[B] total {time.time() - t0:.1f}s")
    # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
    dchars = int(0.25 * maxlen * 3.28)
    dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
               for i in range(seqs)]
    print(f"[D] {seqs} concurrent streams x 512 tokens, "
          f"{dchars:,} chars each")
    t0 = time.time()
    with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
        futs = [ex.submit(post, b) for b in dbodies]
        for i, fut in enumerate(futs, 1):
            try:
                j, wall, _ = fut.result()
                show(f"D-s{i}", j, wall)
            except Exception as e:  # a preemption/timeout lands here
                print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
    tot = time.time() - t0
    print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
          f"{seqs * 512 / tot:.1f} t/s aggregate decode")
    j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
    show("C-prefix-hit", j, wall, "zephyr-lantern-7")
@@ -13,9 +13,11 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
+# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
-# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
+# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
-# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
+# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
 # own budget table makes claimable, spent on the slab because prefill is
 # slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
 # 8 GiB + disk 64 GiB, PLE pinned in RAM.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
@@ -61,6 +63,7 @@ set +a
 : "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
 : "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
 : "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
 : "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
 : "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
 # The image gate is the digest-pinned ref itself: a name@sha256 ref exists
@@ -117,34 +120,19 @@ if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
    exit 1
 fi
-# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
-# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
+# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
-# pages_limit (the current kernel exposes NO pages_max; the cap function
+# pages, capped by ttm's pages_limit (the current kernel exposes NO
-# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
+# pages_max; the cap function takes the min over the knobs that exist).
-# the load dies mid-pin with an allocator error instead of this named
+# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
-# refusal — raise the knob first (runtime sysfs write; kernel cmdline
+# error, so the gate refuses first — and RAISES the knob before refusing:
-# fallback + one reboot only if the file is read-only), see the runbook.
+# the cap is a runtime knob whose default is half of RAM, so a power failure
-# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
+# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
-ttm_cap_mib() {
+# and Restart=always re-ran the refusal 6,919 times over ~20 h).
-    # min over the knobs THIS kernel exposes — the current kernel has
+# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
-    # pages_limit only (no pages_max); a knob that is absent is skipped,
+# shellcheck source=/dev/null
-    # not a zero (a zero would refuse a healthy cap).
+source "$SCRIPT_DIR/ttm-preflight.sh"
-    local v cur=""
+ttm_ensure_cap || exit 1
    for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
        [ -r "$p" ] || continue
        v="$(cat "$p" 2>/dev/null)" || continue
        [ -n "$v" ] || continue
        v=$(( v * 4 / 1024 ))
        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
    done
    echo "${cur:-0}"
 }
 TTM_CAP="$(ttm_cap_mib)"
 if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
    echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
    echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
    exit 1
 fi
 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
@@ -0,0 +1,104 @@
 #!/usr/bin/env bash
 # ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
 #
 # Sourced, never executed. An engine that pins tens of GiB of host RAM (the
 # radiance expert pools + the PLE n-gram table) has its pinned pages charged
 # against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
 # (and pages_max on kernels that expose one). The cap function below takes the
 # MIN over the knobs the running kernel actually exposes — an absent knob is
 # skipped, never read as zero, because a zero would refuse a healthy cap.
 #
 # THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
 # half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
 # failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
 # preflight refused the start — 6,919 crash-restarts over ~20 hours, with
 # `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
 # therefore not an acceptable repair procedure: the gate raises the knob itself
 # (the units that need it run as root, and the knob is a 0644 root-owned module
 # parameter), re-reads, and only then refuses.
 #
 # Contract (set by the sourcing script's env file):
 #   FRANK_ARCHON_TTM_MIN_MIB     floor the pinned ledger needs; refuse below it
 #   FRANK_ARCHON_TTM_TARGET_MIB  value to raise the cap to when it is below the
 #                                floor (the recorded host ceiling — writing the
 #                                floor itself would leave the ledger no slack)
 #   FRANK_ARCHON_TTM_SYS_DIR     knob directory; defaults to the real sysfs
 #                                path, a test points it at a temp dir.
 # Pages are 4 KiB: pages = MiB x 256.
 : "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
 # Prints the effective cap in MiB, 0 when no knob is readable.
 ttm_cap_mib() {
    local p v cur=""
    for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
             "$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
        [ -r "$p" ] || continue
        v="$(cat "$p" 2>/dev/null)" || continue
        [ -n "$v" ] || continue
        v=$(( v * 4 / 1024 ))
        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
    done
    echo "${cur:-0}"
 }
 # Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
 # swallowed: the caller re-reads the cap, and a write the kernel refused shows
 # up there (ttm refuses a limit below the pages it has already pinned).
 #
 # PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
 # process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
 # canonical privileged path is the unit's ExecStartPre with the `+` prefix
 # (systemd runs it as root even under User=); `sudo -n` is the fallback for a
 # human running the gate from a shell (passwordless sudo on both hosts).
 ttm_raise_cap() {
    local pages=$(( $1 * 256 )) p
    for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
             "$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
        [ -e "$p" ] || continue
        if [ -w "$p" ]; then
            echo "$pages" > "$p" 2>/dev/null || true
        elif command -v sudo >/dev/null 2>&1; then
            sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
        fi
    done
 }
 # Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
 # needed; returns 1 with a named refusal when it cannot be made to.
 ttm_ensure_cap() {
    local cap
    cap="$(ttm_cap_mib)"
    if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
        echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
             "floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
             "$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
        ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
        cap="$(ttm_cap_mib)"
    fi
    if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
        echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
        echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
        return 1
    fi
    echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
 }
 # Executed (not sourced) it is the privileged pre-start hook: systemd runs this
 # as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
 # the `Restart=always` loop can never outrun it.
 if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
    set -euo pipefail
    case "${1:-}" in
        --raise)
            TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
            # shellcheck source=/dev/null
            source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
            ttm_ensure_cap
            ;;
        *)
            echo "usage: ${0##*/} --raise   (raise the ttm pages cap, then verify)" >&2
            exit 2
            ;;
    esac
 fi
@@ -3,7 +3,7 @@ type: reference
 title: Model Swap and Rollback
 description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
 tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-15T00:00:00-07:00
 ---
 # Model Swap and Rollback
@@ -12,6 +12,15 @@ Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon /
 frank-archon). Repo law: all config flows through git; the live models are
 host state under `/home/dank/models/<name>/`.
 **No bench step (law, 2026-10-15).** A model swap NEVER schedules a bench
 window. Performance evidence comes from the deployed service itself: the
 engine's boot ledger, per-request timings (`/completion` `timings`,
 `Server-Timing`), and one real request per harness role (§6, §8). A
 deliberate standalone bench (flag A/B, shape sweep) is OUT OF PROCEDURE —
 a separate operation that follows the AGENTS.md Benchmarking Policy
 (stop & mask). The env-file hook (§5) stays so those out-of-procedure
 arms remain possible.
 ## 1. Fit preflight — never deploy a swap without confirming it FITS
 1. Get the checkpoint's on-disk/weight size AND precision from the HF card
@@ -144,7 +153,8 @@ recoverable, drift is not.
 ## 8. Close-out (mandatory)
 1. Docs: active runbook updated (build recipe, profile, validation numbers
-   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
+   incl. real-workload prefill/decode measured on the LIVE service — no
   bench window); superseded runbook gets a RETIRED
   banner and stays as the rollback record; `LLM/index.md` mermaid + active
   list; AGENTS.md fleet table + runtime section + history chain; work-log
   entry in `serverAppConfigs-docs/log.md`.
@@ -277,5 +287,6 @@ predecessor "stopped" (the port race above).
 - [[GGUF Compatibility]] — which runtime can load a payload at all
 - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
+- [[LLM Bench Windows and Measurement]] — out-of-procedure bench windows
  only; a new deployment is measured by §6 live validation, not by a bench
 - [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
@@ -145,11 +145,18 @@ dtype`) so a future resize cannot silently under-provision the pool.
 ### Why the weights slab pays — and why that is not pure loss
-The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
+The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
-boot consumes outside both (driver context, the 990 MiB activation arena,
+hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
-hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
+24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
-**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
+context, the activation arena and --gpu-headroom-mib come out of it; anything
-resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
+past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
 against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
 static region)`. Claimable reserve above the ENGINE's floor:
 2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
 written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
 figure of 31,856 MiB; the engine never asked for that reserve and the seam now
 derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
 the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
 into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
 slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
 floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
@@ -253,6 +260,240 @@ from the scored configuration — bf16 can only reduce the attention-state
 error, and a re-score is a drained-window job if the gate agreement is ever
 needed in the record.
 ## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
 ### The outage: a refusal, not a crash
 Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
 never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
 after starting, with one journal line on repeat:
 ```text
 ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
 frank-archon: raise /sys/module/ttm/parameters/pages_limit …
 ```
 **6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
 the name with a zero start time), so there was no engine, no port, and no engine
 log — the signature of a preflight refusal, which every deployer/monitor view
 outside the journal renders as an unexplained crash loop.
 The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
 expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
 shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
 runtime state, and its default is half of RAM** — so the manual raise of
 2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
 fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
 script enforces. The fp8-native era had planned the persistent half of this
 ("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
 reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
 a power failure is not scheduled.
 ### The repair: the gate raises the knob itself
 `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
 the MIN over the knobs the running kernel exposes, and when it is below
 `FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
 52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
 The unit is root and the knob is a 0644 root-owned module parameter, so the write
 is legal; a kernel that refuses the move produces the same named refusal as
 before, and a reboot is now a boot rather than a visit to the machine.
 | Knob | Power-failure default | After the gate |
 |---|---|---|
 | `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
 Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
 refuses when the write fails* and *the ttm cap counts only the knobs the kernel
 exposes* drive the gate against a temp knob directory — a below-floor writable
 knob ends at the target and the start passes; a read-only knob leaves the cap
 untouched and the start refuses by name; an absent `pages_max` is skipped, never
 read as zero (a zero reading would refuse a healthy host).
 ### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
 Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
 concurrent 200,000-token conversations with quantized KV.
 fp8 is not a preference at four slots, it is the only dtype that affords them.
 The pool must back `seqs × max-model-len` (the engine's hard limit), so the
 fourth slot is bought in KV, out of the one per-card budget
 (`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
 | dtype | pool for 4 × 200,000 | weights slab left |
 |---|---|---|
 | bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
 | **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
 So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
 2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
 44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
 move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
 **843,566 predicted tokens** = four full-height conversations with 5.4 % slack
 (the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
 1.93).
 Terms that move with the dtype and the slot count:
 - **Per-slot recurrent state** is dtype-independent and lives inside the pool:
  +110 MiB per slot, 440 MiB for four (already in the arithmetic above).
 - **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
  **12 rows**, still inside the 25-row bound at which the `wht6` wire is
  bit-exact — so four slots cost nothing in wire exactness.
 - **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
  58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
  bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
 - **`models.yml` is untouched**: `contextWindow` quotes the per-request window
  (200,000), which did not change; only the concurrency did.
 Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
 (same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
 cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
 ~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
 26,071 MiB slab and does not come back by quantizing KV.
 ### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
 The sizing law's prediction, checked against the engine's own arithmetic:
 predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
 | Group | Page / state | Size | Tokens |
 |---|---|---|---|
 | `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
 | `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
 | `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
 | `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
 `kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
 the worst case` — the capacity contract, in the engine's words. The rest of the
 boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
 / 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
 2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
 4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
 lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
 Two ledger lines that confirm the derived consequences: the sampler prints
 **12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
 `wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
 (58.50 KiB, half of the bf16 entry).
 ### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
 Frank's call, twice in one evening. Both are one-env-pair changes, and both were
 read off the boot ledger rather than assumed.
 **4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
 3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
 was backing 858,969 tokens — 43 % more window than the contract asks for.
 Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
 1,792 MiB to the slab and drops the all-reduce to 9 rows.
 **24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
 `card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
 --gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
 the ENGINE's own floor, with the budget table calling the surplus `throughput on
 the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
 `unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
 | pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
 |---|---|---|---|---|
 | 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
 | 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
 | **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
 | 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
 The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
 ~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
 +8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
 slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
 plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
 the card is throughput (the ledger shows the transfer directly: slab
 +1,313 slots, pinned pool −1,313).
 The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
 for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
 30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
 **27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
 it is a capacity decision, not a tuning one.
 Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
 of **199,984 tokens** each (599,952 tokens) were accepted and completed on
 `/v1/completions`; a fourth full-height request is refused as "too large to ever
 fit" — radiance never truncates.
 ### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
 Frank's challenge — "why are you not tracking decode?" — was right, and the
 answer is a number now. 512 generated tokens (the bench law), greedy, cold
 prompts, this shape (3 x 200,000, fp8, slab 25,600):
 ```
 1-stream  43.5k: decode  99.0 t/s  (prefill 4,042, accept 0.952)
 1-stream   197k: decode  93.3 t/s  (prefill 4,826, accept 0.938)
 3-stream  43.5k: decode 113.9 / 43.3 / 10.9 t/s   wall 83.4s
 3-stream   197k: decode  88.9 / 16.3 / 64.6 t/s   wall 122.4s
 ```
 Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
 93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
 prefill, not tokens-per-second. **Concurrency is where the slot count is
 paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
 at 197k against 99 single-stream, and the split is a lottery
 (113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
 to whoever it reached first, so one stream finishes at 113 t/s while its
 sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
 number shows it.
 Corpus caveat, stated at the claim: this probe's filler is a repetitive
 ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
 table below; these rows flatter decode relative to that table and are the
 baseline for *this* corpus, not a replacement for those rows.
 `probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
 tokens, which is why a day of shape work priced only prefill).
 ### Recovery evidence
 ```text
 Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
  is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
  204800 MiB (52428800 pages)
 Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
  204800 MiB clears the pinned-ledger floor 150000
 Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
 ```
 `cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
 the boot, i.e. the root `ExecStartPre` hook did the write the service process
 (User=dank) cannot.
 ### Capacity probe (2026-10-13, live service)
 `scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
 from the deployed env on the host, so the probe cannot drift from the service:
 it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
 of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
 carrying a marker whose first word is unique.
 | Stream | prompt | wall | prefill | retrieval |
 |---|---|---|---|---|
 | B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
 | B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
 | B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
 | B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
 | **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
 All four answered their own marker; nothing was preempted, refused, or truncated
 (`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
 prefill equals the single-stream rate measured in the same probe (3,397 t/s at
 43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
 ceiling and the engine never ran short of KV. Prefix cache intact: the same
 43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
 The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
 make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
 rate). This probe answers one question only: does the endpoint owe four
 full-height conversations at once, and answer each one exactly. It does.
 ## Gates (measured at cutover)
 - **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
@@ -2,13 +2,14 @@
 **Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
 (digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
-driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
+driving `podman run` directly, root systemd unit, :7770):
-`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
+`ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
-(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
+(`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
-the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
+TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
-experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
+(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
-47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
+30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
-2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
+table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
 froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
 and frank-archon-vllm-shapeG-20261003),
 and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
 (frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
@@ -36,10 +37,10 @@ graph LR
 gfx1201, 32 GiB each]
        B2[radiance StillDeadcode, digest-pinned image
 podman run, frank-archon, port 7770]
-        C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
+        C[Swift1.5-Qwen3.8-Flash-Next GPTQ w4 .rad
-2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
+2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
-PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
+PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
-2 x 200000, froggeric v22.5]
+4 x 200000, froggeric v22.5]
    end
    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -155,12 +156,24 @@ frank-zealot, port 7771]
 - [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
  Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
  1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
-  the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
+  the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
-  UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
+  pair re-cut twice** — `--kv-cache-dtype fp8` first at
-  462,752 tokens), weights slab 26071 → 22208, host pool 44928,
+  `--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
-  `--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
+  with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
-  dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
+  raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
-  NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
+  settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
  1 GiB reserve invented. fp8 is what funds the slots — the pool must back
  `seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
  Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
  **25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
  prediction that the slab was a threshold below 26 GiB died on its own
  measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
  arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
  and the 4-slot fp8 shape are the documented rollbacks.
  **2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
  itself** — the 2026-10-12 power failure
  reset the knob to its default (125,988 MiB) and the endpoint refused to boot
  6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
 - [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
  w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
  (root systemd unit, port 7770): the StillDeadcode **radiance** engine
@@ -1,5 +1,99 @@
 # Bundle Change Log
 ## 2026-10-13
 - **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
  power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
  from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
  below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
  preflight named the refusal and exited 1, and `Restart=always` re-ran it
  **6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
  container, no engine log. Law adopted: **a sysfs raise is a countdown to the
  next reboot**, so the gate raises the knob itself. New
  `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
  re-verify, then refuse by name) and is also the unit's
  `ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
  and the knob is root-owned 0644, which is what the first attempt (a plain
  write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
  records the host ceiling.
 - **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
  `--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
  measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
  at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
  to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
  fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
  `models.yml` needed no change: the per-request window is unchanged, only the
  concurrency.
 - **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
  backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
  sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
  of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
  (now reading the shape from the deployed env): four concurrent 196,960-token
  chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
  aggregate prefill, no preemption; prefix cache verified (repeat answered in
  1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
  homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
 - **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
  4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
  was funding 858,969 tokens — 43 % over the contract. Then slab
  24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
  OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
  96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
  real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
  invented on 2026-10-11. The pair now spends 30,720 and settles
  `unclaimed 477.30 MiB` (cards ~336 MiB free).
 - **The prediction shipped with that arm was falsified by its own measurement,
  which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
  below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
  other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
  cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
  193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
  pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
  points inside noise of each other are not a plateau. The cliff is above
  25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
  slots (26,112 + 5,120 > the ceiling) and free at two.
 - **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
  accepted and completed (599,952 tokens resident), ledger
  `backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
  law's 608,027. A fourth full-height request is refused as too large to ever
  fit — radiance never truncates. Probe law: the oversized-prompt counter is
  FREE (the refusal fires when PROMPT tokens alone exceed the window —
  `prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
  window is ~514,286 characters at the measured 2.5714 chars/token, so a
  calibration probe below that pays a real prefill instead of reporting.
 - **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
  exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
  conversation. Untested arm; capacity decision, not tuning.
 - Seam: the pair ceiling is derived from the engine's budget table (30,969) and
  owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
  measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
  own test. 371/371 tests and e2e 33/33 on both deploys.
 - **Power-failure casualties, NOT fixed (outside the archon window):**
  `mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
  raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
  media libraries on it read empty. `frank-job-digest.service` (frankbook
  job-hunt digest) is in `list-units --state=failed`.
 - **The slab is a threshold, not a slope** — measured, same method (probe A
  leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
  22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
  bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
  (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
  would clear it (pool 3,072 + slab 27,897) remains untested — and the
  paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
  not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
 - **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
  cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
  is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
  give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
  The slot count is paid in decode concurrency — a fused-step lottery, not a
  fair share — which is exactly the axis the shape hunt was blind to (probe
  leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
  is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
  table, so these rows are the baseline for this corpus, not a replacement.
 - **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
  tokens, per-stream + aggregate) so no future shape change can be priced on
  prefill alone.
 ## 2026-10-10
 - **Fleet service rename: the StarCraft names became the SERVICE names.**
  The 2026-10-09 wire alias moved onto every repo artifact: units, serve
@@ -1651,3 +1745,22 @@ every restart since the cutover.
  [[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
  (the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
 ## 2026-10-15 — governance: bench removed from the model-change procedure
 - Frank's call: a model swap never carries a bench window. Codified as law:
  - `AGENTS.md` Benchmarking Policy opens with "Benchmarking is NOT part of
    the model-swap procedure"; swap verification is the live service (boot
    ledger, `timings`/`Server-Timing`, one real request per harness role).
    The stop-and-mask rule stays for DELIBERATE bench windows (flag A/Bs,
    shape sweeps) — it is host protection (2026-09-16 hard-down precedent),
    not a swap step. Summary-table row updated to match.
  - [[Model Swap and Rollback]]: "No bench step (law, 2026-10-15)" up top;
    §8 close-out numbers must be measured on the live service; the Related
    link to [[LLM Bench Windows and Measurement]] rescoped to
    out-of-procedure windows only.
 - Untouched on purpose: the `FRANK_*_ENV_FILE` hook (staged/out-of-procedure
  arms), the Fleet Ops Playbook bench-window mechanics (deliberate windows
  still need the procedure), `deployment.md` (already says bench work is a
  SEPARATE procedure). Managed skill `llamacpp-endpoint-model-swap` verifies
  on the live engine — it never had a bench step.
@@ -9,8 +9,9 @@
 # flags alone do not say: the served window clears the harness worst case,
 # the KV pool backs seqs x max-model-len at the declared cache dtype (an
 # engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
-# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
+# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
-# the serve script ENFORCES it, the served window is what models.yml must
+# it and RAISES the knob itself when a reboot reset it (the cap's default is
 # half of RAM), the served window is what models.yml must
 # quote, the host-networking contract (no --publish), the image gate on a
 # name@sha256 ref, the .rad payload gate with its size floor, the
 # chat-template override mount, the self-healing law (the run returns before
@@ -24,6 +25,7 @@ load helpers
 UNIT="$REPO_ROOT/scripts/frank-archon.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
 TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
@@ -31,9 +33,20 @@ BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
 HFGET="$REPO_ROOT/scripts/hfget.sh"
 TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
 # A PATH with `cat` (the gate reads knobs with it) and NO `sudo`, so the
 # privileged fallback cannot rescue a write the test means to fail.
 nosudo_bin() {
    local bin="$BATS_TEST_TMPDIR/nosudo-bin"
    mkdir -p "$bin"
    ln -sf "$(command -v cat)" "$bin/cat"
    ln -sf "$(command -v bash)" "$bin/bash"
    echo "$bin"
 }
@test "frank-archon unit, serve script, env, fetch and build scripts and template exist" {
    [ -f "$UNIT" ]
    [ -x "$SERVE" ]
    [ -f "$TTM_LIB" ]
    [ -f "$ENV_FILE" ]
    [ -f "$DOWNLOAD" ]
    [ -x "$BUILD" ]
@@ -120,37 +133,40 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
    grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
    ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
    [ "$ml" -ge 177356 ]
-    # Concurrency: 2 slots x the served window is the capacity contract, and
+    # Concurrency: 3 slots x the served window is the capacity contract, and
-    # the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
+    # the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
    # under the 25-row bound at which the wht6 wire stays bit-exact. The
    # "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
    # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
    # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
-    # tied inside the +-3% noise floor.
+    # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
    # expert_tiered placement: vram_weights + the pinned host pool serve the
    # per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
-    # KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
+    # KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
    # only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
    # in host RAM per Frank's law.
    grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
    # Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
    grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
    # The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
    # under the ttm pages cap, and the env states the floor the serve
-    # script enforces (>= 140,000 MiB — the current cap ~123 GiB fails it,
+    # script enforces (>= 140,000 MiB — the kernel's own default cap, half of
-    # which is the point).
+    # RAM, fails it, which is the point) plus the value it raises the knob to
    # when a reboot reset it (the target must clear the floor with slack).
    grep -q '^FRANK_ARCHON_TTM_MIN_MIB=' "$ENV_FILE"
    ttmm=$(awk -F= '/^FRANK_ARCHON_TTM_MIN_MIB=/{print $2}' "$ENV_FILE")
    ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
    [ "$ttmt" -gt "$ttmm" ]
    pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
    ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
@@ -188,15 +204,20 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
 }
@test "the per-card VRAM pair stays inside the measured card ceiling" {
-    # Measured on Dankserver 2026-10-11: card 31,856 MiB, a boot consumes
+    # The engine's OWN budget table, not a hand-picked reserve. Boot of
-    # 1,710 MiB OUTSIDE both budgets (driver context, the 990 MiB activation
+    # 2026-10-13 at the pair 24000/5120: card 32,624 MiB (31.86 GiB), budgets
-    # arena, hardware queues, headroom, step staging) and the fp8 pair settled
+    # claimed 29,120, `3.42 GiB is neither budgeted nor accounted — the
-    # at 1,379 MiB free. Claiming more than 29,120 MiB in total is an
+    # driver's context, the activation arena and --gpu-headroom-mib come out of
-    # allocation failure at graph capture, not a warning — so the KV pool and
+    # it; anything past those is throughput on the floor`, and `card 0 settled
-    # the weights slab are one budget, and this seam is the guard.
+    # at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and
    # 113.19 MiB kept for the static region)`. Claimable reserve above the
    # engine's floor: 2,058 - 209 = 1,849 MiB, so the ceiling is 30,969 MiB.
    # The 2026-10-11 bound of 29,120 ("keep >= 1 GiB free", card read as 31,856
    # MiB) was MY reserve, which the engine never asked for; the shipped pair
    # spends the difference and lands ~458 MiB above the floor.
    w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
    kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
-    [ $(( w + kv )) -le 29120 ]
+    [ $(( w + kv )) -le 30969 ]
 }
@test "the serve script wires the engine flags from the env" {
@@ -272,6 +293,83 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
    grep -q -- '--device /dev/dri' "$SERVE"
 }
@test "the ttm gate raises a reset pages cap and refuses when the write fails" {
    # The cap is a RUNTIME knob: the kernel default is half of RAM, so every
    # reboot resets it (measured: the 2026-10-12 23:17 power failure on
    # dankserver took it 200 GiB -> 125,988 MiB, and the preflight refusal was
    # re-run 6,919 times by Restart=always over ~20 h). Behaviour required of
    # the gate, in both directions:
    #   knob below the floor + writable  -> cap ends at the target, start passes
    #   knob below the floor + unwritable -> cap stays low, start REFUSES by name
    local dir="$BATS_TEST_TMPDIR/ttm"
    mkdir -p "$dir"
    export FRANK_ARCHON_TTM_SYS_DIR="$dir"
    export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
    echo 32253039 > "$dir/pages_limit"    # 125,988 MiB: the kernel default here
    run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
    [ "$status" -eq 0 ]
    [ "$(cat "$dir/pages_limit")" = "52428800" ]   # 204,800 MiB in 4 KiB pages
    echo 32253039 > "$dir/pages_limit"
    chmod 444 "$dir/pages_limit"          # the write refused: cap cannot move
    # The refusal case must not have `sudo` on PATH: the gate's privileged
    # fallback runs `sudo -n`, and on a passwordless host root can write a 444
    # file, which would silently test nothing.
    run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
    [ "$status" -eq 1 ]
    [[ "$output" == *"ERROR: ttm pages cap is 125988 MiB"* ]]
    [ "$(cat "$dir/pages_limit")" = "32253039" ]
 }
@test "the ttm cap counts only the knobs the kernel exposes" {
    # An absent knob is not a zero: a zero-cap reading would refuse a healthy
    # host. The effective cap is the MIN over the knobs that exist; pages_max,
    # when a kernel exposes one, is the binding one.
    local dir="$BATS_TEST_TMPDIR/ttm2"
    mkdir -p "$dir"
    export FRANK_ARCHON_TTM_SYS_DIR="$dir"
    export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
    echo 51200000 > "$dir/pages_limit"    # 200,000 MiB, no pages_max knob
    run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
    [ "$status" -eq 0 ]
    [ "$output" = "200000" ]
    echo 30000000 > "$dir/pages_max"      # 117,187 MiB: now the binding knob
    run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
    [ "$output" = "117187" ]
    chmod 444 "$dir/pages_limit" "$dir/pages_max"   # both writes refused
    run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
    [ "$status" -eq 1 ]
    [[ "$output" == *"ERROR: ttm pages cap is 117187 MiB"* ]]
    [ "$(cat "$dir/pages_max")" = "30000000" ]
 }
@test "the gate runs as a root pre-start hook, where the knob write is legal" {
    # The knob is root-owned 0644 and the service runs as User=dank, so the
    # gate CANNOT raise it from the service process. systemd's `+` prefix on
    # ExecStartPre runs as root even under User= — the canonical privileged
    # path — and the gate's own --raise mode is what that line calls.
    grep -q '^ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise$' "$UNIT"
    # Behaviour of that entry point: sourced env values, knob raised, exit 0.
    local dir="$BATS_TEST_TMPDIR/ttm3" envf="$BATS_TEST_TMPDIR/ttm3.env"
    mkdir -p "$dir"
    printf 'FRANK_ARCHON_TTM_MIN_MIB=150000\nFRANK_ARCHON_TTM_TARGET_MIB=204800\n' > "$envf"
    echo 32253039 > "$dir/pages_limit"
    run env FRANK_ARCHON_ENV_FILE="$envf" FRANK_ARCHON_TTM_SYS_DIR="$dir" \
        bash "$TTM_LIB" --raise
    [ "$status" -eq 0 ]
    [ "$(cat "$dir/pages_limit")" = "52428800" ]
    # Usage is a named failure, not a silent no-op.
    run bash "$TTM_LIB"
    [ "$status" -eq 2 ]
    [[ "$output" == *"--raise"* ]]
 }
@test "the serve script clears stale containers and self-heals via podman wait" {
    # A crashed unit can leave the container name registered forever after;
    # the legacy identities hold the same GPUs and die here too. Then block
Author	SHA1	Message	Date
Frank	39b8396794	governance: bench is removed from the model-swap procedure Model changes verify by driving the live service — boot ledger, per-request timings, one real request per harness role. No bench window is ever scheduled for a swap. The stop-and-mask policy stays for deliberate standalone bench windows (it is host protection, not a swap step). Codified in AGENTS.md (policy lead + summary row), Model Swap and Rollback (No bench step law, §8 close-out source, Related rescope).	2026-10-15 03:07:46 -07:00
Frank	a9d45e7012	frank-archon: the slab is a threshold, and decode is where slots cost Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.	2026-10-13 20:51:32 -07:00
Frank	2581b8d4b6	docs(frank-archon): the slab is a slope, and the ceiling was mine not the engine's Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 -> 3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %, wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s. Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at 29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable). The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the number, so the next resize starts from the engine's arithmetic. Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism, the pinned pool is the slow path. Capacity contract live-verified at 3 slots: 3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too large to ever fit. Probe law learned: the oversized-prompt refusal names the prompt's token count BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.	2026-10-13 20:33:02 -07:00
Frank	09a0251d2d	llm(frank-archon): claim the card's settling slack, slab 24000 -> 25600 The 2 GiB/card that \`rocm-smi\` shows free is not idle: the engine's budget table calls it driver context + the 896 MiB activation arena + 96 MiB headroom, and says the rest is `throughput on the floor`. Its own settle line is the number that matters — `card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)` — so 1,849 MiB above the ENGINE's floor is claimable and the real ceiling is 30,969 MiB, not the 29,120 my 1 GiB reserve invented on 2026-10-11. Shipped pair 25600/5120 = 30,720: +1,600 MiB of slab, ~458 MiB above the floor. The slab is already saturated at 24,000 (mover took all of it, 451 KiB slack), so this is the only direction left — KV is at its 3-slot floor (5,063 needed of 5,120). Named prediction, because today's two points say the slab is a THRESHOLD and not a slope (22,208 -> 3,397 t/s, 24,000 -> 3,290 t/s at 43.5k single-stream): this arm buys ~1,600 resident expert slots and ~0 prefill. The point that moves prefill is 26,071 (2026-10-11: 4,379 t/s), fundable only at two slots. Prefill moving > +-3 % here falsifies the threshold reading. Seam: the ceiling gate now derives from the engine's budget table (30,969) and is the single owner of the pair arithmetic; the duplicate I put in the shape test is removed.	2026-10-13 20:18:41 -07:00
Frank	066813ea95	llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB Frank's call: three slots, and spend the KV the third slot doesn't need on resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`, `--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured 29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %. The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the contract buys nothing. At fp8 the slab curve measured on this host is 22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so 24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check. The cost is named, not hidden: a FOURTH full-height conversation no longer fits (4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit — radiance never truncates. Reverting to four slots is one env pair. Seam: two new arithmetic gates, because both failures are boot-then-400 shape faults that no flag check catches — the pair must respect the per-card ceiling, and the pool must back seqs x max-model-len at the dtype's measured rate plus the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of prefix checkpoints).	2026-10-13 20:00:39 -07:00
Frank	7daa886dd6	docs(frank-archon): work-log entry and the 4-way capacity probe Probe on the live service: four concurrent 196,960-token chats, 4/4 exact marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the single-stream ceiling, so the four streams shared one pipeline and the KV pool never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008. Recorded with the caveat that 8 generated tokens do not make a decode rate.	2026-10-13 19:30:46 -07:00
Frank	bae78e7dae	docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law, the shape arithmetic, the boot ledger that confirms it: 843,276 tokens = 4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise lesson under Key Lessons, and the pinned-RAM paragraph now points at scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.	2026-10-13 19:28:25 -07:00
Frank	a6f55f4cc6	fix(frank-archon): raise the ttm knob from a root ExecStartPre — User=dank cannot write it a9da65e put the raise inside the serve script, which runs as `User=dank`: the knob is root-owned 0644, so `-w` is false and the write silently failed — the gate printed its refusal on a cap it had just failed to move. The privileged path is systemd's `+` prefix: `ExecStartPre=+scripts/ttm-preflight.sh --raise` runs as root even under `User=`, before ExecStart, so `Restart=always` can never outrun it. The lib is now also executable (`--raise` sources the env file and verifies), and keeps a `sudo -n` fallback for a human running the gate from a shell. Seam: the refusal cases run with a PATH that has `cat`+`bash` and NO `sudo` (root writes a 444 file, so a passwordless dev box would have tested nothing), and both now assert the gate's own exit 1, not merely non-zero. New seam covers the unit line and the `--raise` entry point end to end (writable knob -> raised, no args -> usage exit 2).	2026-10-13 19:20:33 -07:00
Frank	a9da6589a6	fix(frank-archon): `:" guard needs a space —` :"${VAR:?}"` runs :204800 The TTM_TARGET_MIB guard I added on `1a6a78e` was written `:"...` (no space), so bash parsed the expansion as one word and executed `cmd :204800` as the command `:204800`: exit 127 `command not found` on every start, before the ttm gate ran. Caught on the deploy: the unit crash-looped 34 times in ~4 min with exactly that journal line. probe-frank-archon-kv.py now reads the slot count, window and KV dtype from the deployed env on the host, so probe B exercises the capacity contract that is actually live (4 x 200,000 today) instead of a hardcoded pair, and each probe's marker starts with a unique word because show() scores the hit on that word.	2026-10-13 19:16:48 -07:00
Frank	1a6a78ef73	fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve script's preflight named the refusal and exited 1; Restart=always re-ran it 6,919 times over ~20 hours and frank-archon served nobody. scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob itself before refusing: the unit is root, the knob is a 0644 root module parameter, and the cap is a runtime knob whose default is half of RAM — so a manual sysfs write is not a repair procedure, it is a countdown to the next reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate re-reads the cap after writing and only then refuses, so a knob the kernel will not move is still a named failure. Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8 (quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the 25-row bound where the wht6 wire is bit-exact.	2026-10-13 19:09:58 -07:00