serverAppConfigs/serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md

# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2

**Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2

The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert)
is replaced by a **GPTQ-calibrated 4-bit container** of
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB, abliterated
finetune) converted with the recipe
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
Same port, alias, container name, systemd unit. Rollback: revert the
`MODEL_FILE` line in `scripts/frank-archon.env` + redeploy —
`/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash
policy; 173 GiB, purge when the window closes).

## Why this swap

- **Context economics at production memory budgets**: the FP8-native
  experts (56.4 GiB/rank slice) forced a 40 GiB/rank pinned host pool and
  left KV ~3.5 GiB. W4Dn8 experts cut the resident set to ~1/4, freeing
  host and VRAM budget for KV and slots at equal-or-better measured
  decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
  fleet-personality call; frank-dragoon serves the same family).
- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
  measurably worse; GPTQ with Hessians from a 10M-token corpus of this
  model's own distribution recovers most of it, verified end-to-end by a
  KLD gate against the BF16 container itself (KL ≤ 0.03, top-1 ≥ 95 %).

## The payload chain

1. **Fetch**: `scripts/download-frank-archon-swift15.sh` →
   `/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
   GiB, per-shard LFS sha256 verified, no token on the command line).
   48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
   PLE 20M-vocab table 95.37 GiB, MTP head, vision tower.
2. **BF16 container**: `run-swift15-bf16.sh convert` →
   `rad-convert` no-recipe (pure copy) at `--max-ctx 4096` →
   `qwen4exp-swift15-bf16.rad` (335.28 GiB). Serves on 1.0.13 as the
   KLD reference engine (disk-tier placement; boot ~35 min).
3. **KLD reference**: `run-swift15-bf16.sh kld` (1.0.13) →
   `/home/dank/corpora/kld-ref` — `radiance-kld-1` format, `rows:
   50235`, 197 source-labelled docs, top-16 log-probs (f16), corpus
   fnv1a64 `a9d62b33c18b3d9c`. ~4 h at 32k prefill steps
   (`--max-num-batched-tokens 32768` — a 65536 step OOMs an arena
   tensor at `--max-model-len 16384`). **The wrapper's exit line is the
   receipt: never scp this script while it runs** (bash lazy-read;
   a mid-run swap killed a clean engine exit with rc=127, third fleet
   occurrence of the script-swap law). `kld-then-calib.sh SKIP_KLD=1`
   resumes from the `rows: 50235` receipt.
4. **Calibration**: `run-swift15-bf16.sh calib` on **1.2.0** (the only
   engine with `RADIANCE_CALIB_TOKENS`; grams are engine-invariant
   bf16 activations — RADGRAM2 format byte-identical across versions)
   + `calib-push.py` pushing `calib-corpus.jsonl` (10,014,932 tokens,
   1,294 prompts, assembled from real HF sources, chat-templated,
   source-labelled). The tap (in the `qwen4exp_fp8.so` architecture
   plugin) writes per-expert down-proj Gram matrices
   `gram.r<rank>.blk.L.ffn_down_exps.<e>.bin` (32-byte RADGRAM2 header +
   640×640 f32; `rows` = routing count, `trace` = activation energy)
   and stops the serve at the token threshold. TP=2: whole experts per
   rank (256/rank); gate_up input is rank-sliced (plugin warning — the
   recipe only casts the down projection).
5. **Protected-expert derivation**: `derive-protected-experts.py
   /home/dank/corpora/calib` → the 24 highest-energy experts (the law
   is data-driven; Swift drew 24, not the fp8-era 10 — its routing is
   spikier: `blk.47 e445` alone carries 32 % of its layer's down energy
   at 70× routing share) promoted to whole-expert bf16.
   Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count
   per rank half; odd half gains the next most energetic.
   `apply-protected-cast.sh` splices the derived names between the
   `# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive).
6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend
   CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) —
   gates: ≥48 gram files present, bf16 container already deleted
   (disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the
   gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on
   49 GPTQ layers, load-average 8), output **116,416 MiB**.
   The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT
   TABLE (`one table of experts takes one encoding, and only a whole
   expert kept plain bf16 is served beside it`) — a split tier
   (down=fp8, gate_up=w4) is unservable: the kernel serves a single
   w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated,
   gate_up uncalibrated), protected experts bf16 beside them.

## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe)

| Tier | Encoding | Why |
|---|---|---|
| MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) |
| 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) |
| shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law |
| hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law |
| lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) |
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |

## The shape (`scripts/frank-archon.env`)

TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law —
2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric
v22.5 template override. `MODEL_FILE` points at
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB
measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that
change is documented in its own section below.

## 2026-10-11 — unquantized KV at 2 × 200,000

The ask: two concurrent 200,000-token conversations on exact (unquantized)
attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93
sessions of the served window**, so two full-height conversations overflow it
and the scheduler preempts the overflow to the host/disk prefix tiers
(re-prefilling on return). Measured, that overflow is survivable — the fp8
shape passed the same 2 × 197k pair probe — so this is a **headroom** change,
not the repair of a hard failure. The hard limit the engine states is per
request, not per pair: a request longer than the pool's token count is
*"accepted by the API and then refused as too large to ever fit — this is a
hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request
ever hit it.

### The dtype law

`--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole
cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as
`--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the
rung under bf16 is fp8 on both halves. Evidence: the flag table in the image
itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any
`cache-type-k`/`-v` symbol in the binary.

bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table:

| Group | fp8 page | bf16 page | cost per token, per rank |
|---|---|---|---|
| `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB |
| `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed |
| `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** |
| prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed |

→ `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces
both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904;
bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded
as a test seam (`the KV pool backs seqs x max-model-len at the declared cache
dtype`) so a future resize cannot silently under-provision the pool.

### Why the weights slab pays — and why that is not pure loss

The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
context, the activation arena and --gpu-headroom-mib come out of it; anything
past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
static region)`. Claimable reserve above the ENGINE's floor:
2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
figure of 31,856 MiB; the engine never asked for that reserve and the seam now
derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).

Half of that loss is refunded by an engine behaviour worth knowing: **idle KV
VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert
slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of
the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is
a competing cache, and every KV resize is also an expert-residency resize.

### seqs 3 → 2

`--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM
fork's torch.compile constraint (`ConstraintViolationError: Constraints
violated (L['query_start_loc'].size()[0])`), retired with that runtime — see
[[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the
fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied
inside the ±3 % noise floor. Two slots × the window is the capacity contract;
it also saves 110 MiB of per-slot recurrent state and drops the fused TP
all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the
wht6 wire is bit-exact.

### Boot ledger (deployed 2026-10-11, commit `0bf26b0`)

```text
kv cache           6.75 GiB   349% of it -- backs 2 sequences all at 200000
                                tokens at once, which is the worst case
vram_weights      21.69 GiB   stated by --vram-weights-mib
vram_kv            6.75 GiB   stated by --vram-kv-mib
unclaimed          2.03 GiB
kv_attn      full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens)
kv_qsa_bkey  full 13 layers block 4 tok page  3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens)
load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB
mover: 5146 slots can be lent out of the KV cache (6.08 GiB)
card 0 settled at 1.85 GiB free
serving on http://:::7770
```

### Verification (live service, no scratch container)

| Check | Result |
|---|---|
| `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` |
| 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s |
| decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) |
| fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section |

Probe law re-learned, worth recording: radiance routes the thinking block to
`reasoning_content`, so a chat request whose budget is spent inside `<think>`
comes back with **empty `content` and `finish_reason: stop`** — a probe that
forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both
fields is the only honest shape. The first pair probe scored 0/2 on exactly
that artifact, not on a KV fault.

### Measured cost — A/B on the live service

Method (no scratch container, no mask — the unit file is a relink symlink and
`mask` refuses it): write a variant env file on the host
(`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it
(`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own
`Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the
drop-in, restart. Control = the deployed 2026-10-08 shape, verified from
`podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071
--vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB).
Identical probes, same salts, loopback, greedy, 512 generated tokens.

| Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ |
|---|---|---|---|
| prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** |
| prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** |
| decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % |
| decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % |
| decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % |
| draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** |
| 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall |

Read it as: **the price of exact KV on this engine is prefill, not decode.**
Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so
time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s);
both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is
inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass
−24 % is the largest single number, the least reproducible, and it moves the
same direction as prefill — the cold arm is paying expert streaming, not
attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact
attention state makes the target agree with the MTP drafter more often, which
pays on every token of every conversation, and it is the strongest argument for
the shape.

Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8
variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8
for a 105 MiB pool bump and ≈ no decode cost. The tool is committed:
`scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its
prompts from the repo checkout there).

### Rollback

Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3,
host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min.
The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away*
from the scored configuration — bf16 can only reduce the attention-state
error, and a re-score is a drained-window job if the gate agreement is ever
needed in the record.

## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV

### The outage: a refusal, not a crash

Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
after starting, with one journal line on repeat:

```text
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
```

**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
the name with a zero start time), so there was no engine, no port, and no engine
log — the signature of a preflight refusal, which every deployer/monitor view
outside the journal renders as an unexplained crash loop.

The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
runtime state, and its default is half of RAM** — so the manual raise of
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
script enforces. The fp8-native era had planned the persistent half of this
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
a power failure is not scheduled.

### The repair: the gate raises the knob itself

`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
the MIN over the knobs the running kernel exposes, and when it is below
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
The unit is root and the knob is a 0644 root-owned module parameter, so the write
is legal; a kernel that refuses the move produces the same named refusal as
before, and a reboot is now a boot rather than a visit to the machine.

| Knob | Power-failure default | After the gate |
|---|---|---|
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |

Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
exposes* drive the gate against a temp knob directory — a below-floor writable
knob ends at the target and the start passes; a read-only knob leaves the cap
untouched and the start refuses by name; an absent `pages_max` is skipped, never
read as zero (a zero reading would refuse a healthy host).

### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)

Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
concurrent 200,000-token conversations with quantized KV.

fp8 is not a preference at four slots, it is the only dtype that affords them.
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
fourth slot is bought in KV, out of the one per-card budget
(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):

| dtype | pool for 4 × 200,000 | weights slab left |
|---|---|---|
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |

So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
1.93).

Terms that move with the dtype and the slot count:

- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
  +110 MiB per slot, 440 MiB for four (already in the arithmetic above).
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
  **12 rows**, still inside the 25-row bound at which the `wht6` wire is
  bit-exact — so four slots cost nothing in wire exactness.
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
  58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
  bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
  (200,000), which did not change; only the concurrency did.

Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
26,071 MiB slab and does not come back by quantizing KV.

### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)

The sizing law's prediction, checked against the engine's own arithmetic:
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).

| Group | Page / state | Size | Tokens |
|---|---|---|---|
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |

`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
the worst case` — the capacity contract, in the engine's words. The rest of the
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).

Two ledger lines that confirm the derived consequences: the sampler prints
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
(58.50 KiB, half of the bf16 entry).

### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack

Frank's call, twice in one evening. Both are one-env-pair changes, and both were
read off the boot ledger rather than assumed.

**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
was backing 858,969 tokens — 43 % more window than the contract asks for.
Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
1,792 MiB to the slab and drops the all-reduce to 9 rows.

**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
the ENGINE's own floor, with the budget table calling the surplus `throughput on
the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).

| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
|---|---|---|---|---|
| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |

The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
the card is throughput (the ledger shows the transfer directly: slab
+1,313 slots, pinned pool −1,313).

The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
it is a capacity decision, not a tuning one.

Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
of **199,984 tokens** each (599,952 tokens) were accepted and completed on
`/v1/completions`; a fourth full-height request is refused as "too large to ever
fit" — radiance never truncates.

### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill

Frank's challenge — "why are you not tracking decode?" — was right, and the
answer is a number now. 512 generated tokens (the bench law), greedy, cold
prompts, this shape (3 x 200,000, fp8, slab 25,600):

```
1-stream  43.5k: decode  99.0 t/s  (prefill 4,042, accept 0.952)
1-stream   197k: decode  93.3 t/s  (prefill 4,826, accept 0.938)
3-stream  43.5k: decode 113.9 / 43.3 / 10.9 t/s   wall 83.4s
3-stream   197k: decode  88.9 / 16.3 / 64.6 t/s   wall 122.4s
```

Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
prefill, not tokens-per-second. **Concurrency is where the slot count is
paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
at 197k against 99 single-stream, and the split is a lottery
(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
to whoever it reached first, so one stream finishes at 113 t/s while its
sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
number shows it.

Corpus caveat, stated at the claim: this probe's filler is a repetitive
ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
table below; these rows flatter decode relative to that table and are the
baseline for *this* corpus, not a replacement for those rows.
`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
tokens, which is why a day of shape work priced only prefill).

### Recovery evidence

```text
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
  is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
  204800 MiB (52428800 pages)
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
  204800 MiB clears the pinned-ledger floor 150000
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
```

`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
the boot, i.e. the root `ExecStartPre` hook did the write the service process
(User=dank) cannot.

### Capacity probe (2026-10-13, live service)

`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
from the deployed env on the host, so the probe cannot drift from the service:
it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
carrying a marker whose first word is unique.

| Stream | prompt | wall | prefill | retrieval |
|---|---|---|---|---|
| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |

All four answered their own marker; nothing was preempted, refused, or truncated
(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
prefill equals the single-stream rate measured in the same probe (3,397 t/s at
43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
ceiling and the engine never ran short of KV. Prefix cache intact: the same
43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.

The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
rate). This probe answers one question only: does the endpoint owe four
full-height conversations at once, and answer each one exactly. It does.

## Gates (measured at cutover)

- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
  candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126,
  p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**,
  **ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235
  positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json`
  (+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072
  (the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena,
  refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
  fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
  scoring is placement-invariant; scored at 28 positions/s.
- **KLD gate re-PASS (2026-10-08 04:49) at the production-legal shape**:
  MNBT 2048 (the w4 file-tier law: experts stream in place only at
  MNBT ≤ 2048 — the recording's 32768 demands a widened form the disk
  tier refuses), pool 40960, vram-kv 3072 fp8, tp 2, headroom 768.
  **mean KL 0.0131, top-1 97.92 %**. The Δ vs the first PASS (0.0015) is
  the float-order noise band of the chunk-size change — the verdict is
  chunk-size invariant, ~2.3× inside the 0.03 bar on both shapes.
- **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth
  (streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve
  shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**,
  draft accept **0.7246** (350/483); arm C (deployed vram-pair shape
  26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control
  (fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm.
  The w4 expert table is resident — the fp8-native era's NVMe expert
  streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60
  t/s base decode. Conversation battery (short context): geomean
  172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot:
  the w4 resident set fits fully inside the 40 GiB pool at 0.82
  (ledger: mover full fill). Depth-output audit: the 44.5k-depth
  summary is a coherent ordered index of the concatenated corpus
  (no repetition, no NaN-token garbage) — the depth-corruption class
  of 2026-10-04 is clean.
- Fleet probes: PASS, re-run on the deployed prod — vision (solid-red
  512 px → "Red"), tool_calls (`{"expr":"23 * 47"}`), reasoning_effort
  xhigh/medium/low/high all 200 (the Swift1.5 froggeric template accepts
  the full set), reasoning_content present, response_format json_object
  honoured. **Vision probe law:** budget ≥ 256 tokens and read
  `reasoning_content` as fallback — at 64 tokens thinking exhausts the
  budget and `content` comes back empty (the answer lands in the
  reasoning channel).
- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
  --service frank-archon` green incl. Playwright e2e 33/33; unit
  `active`, `/health` 200, `/v1/models` id `frank-archon` (the harness
  alias is the served-model-name, independent of the file — models.yml
  untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
  seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
  pools 57.84 GiB, cards settled 1.41 GiB free, `serving on :7770`.
  Live decode probe (512 tokens, streamed, temp 0): ~190 t/s.

### Wire rename + the bench drop-in trap (2026-10-09)

`FRANK_ARCHON_SERVED_NAME` moved `frank-archon` → **frank-archon**; the
service, unit, scripts and env file keep the old name. Radiance is the only
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
**requested** model name in chat responses — so a chat probe that returns 200
proves nothing about the served id. Always read `/v1/models`.

The first deploy after the edit came up green and `/v1/models` still said
`frank-archon`. Cause: a campaign leftover drop-in
`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
`deploy.sh dankserver --service frank-archon` since then restarted the
staging tree: green health gate, old configuration, silently. Verified before
removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
are byte-identical to the checkout, its `frank-archon.env` differed only by
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`.
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
26071/3543).

**Deployer blind spot:** `deploy.sh` plans from repo paths and gates on the
port. It cannot see host-side drop-ins, so unit drift is invisible to a green
run. After an LLM deploy, verify the served configuration behaviourally
(here: `podman inspect <name> --format {{json .Config.Cmd}}` and `/v1/models`).



## Deploy + verify

```bash
sudo systemctl stop frank-archon          # drained window
bash convert-swift15-w4dn8.sh              # if not already run
./scripts/deploy.sh dankserver --service frank-archon
```

Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
(expert pool coverage now w4-sized, KV pool should be larger than the
FP8-native era's 399,824 tokens); MTP draft counts in `timings`; froggeric
probe; ≥256 px `image_url` vision probe answers the color; tool-call
probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
+ Playwright e2e green.

## Rollback

| Layer | Path |
|---|---|
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-archon.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |

Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24
GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs
`/home/dank/logs/swift15_*.log`. The BF16 `.rad` container is deleted
before the final convert (disk gate) — the raw checkpoint is the
re-convert source if the window ever reopens.

## Links

- [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era
- [[LLM/index]] — fleet index
- [[swift15-campaign-worklog]] — campaign record
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here