# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2 **Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2 The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert) is replaced by a **GPTQ-calibrated 4-bit container** of `ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB, abliterated finetune) converted with the recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as the FP8-native era; 1.2.0 was used ONLY for the calibration serve). Same port, alias, container name, systemd unit. Rollback: revert the `MODEL_FILE` line in `scripts/frank-archon.env` + redeploy — `/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash policy; 173 GiB, purge when the window closes). ## Why this swap - **Context economics at production memory budgets**: the FP8-native experts (56.4 GiB/rank slice) forced a 40 GiB/rank pinned host pool and left KV ~3.5 GiB. W4Dn8 experts cut the resident set to ~1/4, freeing host and VRAM budget for KV and slots at equal-or-better measured decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control). - **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's fleet-personality call; frank-dragoon serves the same family). - **GPTQ closes the precision gap**: RTN alone on BF16 weights lands measurably worse; GPTQ with Hessians from a 10M-token corpus of this model's own distribution recovers most of it, verified end-to-end by a KLD gate against the BF16 container itself (KL ≤ 0.03, top-1 ≥ 95 %). ## The payload chain 1. **Fetch**: `scripts/download-frank-archon-swift15.sh` → `/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3 GiB, per-shard LFS sha256 verified, no token on the command line). 48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560, PLE 20M-vocab table 95.37 GiB, MTP head, vision tower. 2. **BF16 container**: `run-swift15-bf16.sh convert` → `rad-convert` no-recipe (pure copy) at `--max-ctx 4096` → `qwen4exp-swift15-bf16.rad` (335.28 GiB). Serves on 1.0.13 as the KLD reference engine (disk-tier placement; boot ~35 min). 3. **KLD reference**: `run-swift15-bf16.sh kld` (1.0.13) → `/home/dank/corpora/kld-ref` — `radiance-kld-1` format, `rows: 50235`, 197 source-labelled docs, top-16 log-probs (f16), corpus fnv1a64 `a9d62b33c18b3d9c`. ~4 h at 32k prefill steps (`--max-num-batched-tokens 32768` — a 65536 step OOMs an arena tensor at `--max-model-len 16384`). **The wrapper's exit line is the receipt: never scp this script while it runs** (bash lazy-read; a mid-run swap killed a clean engine exit with rc=127, third fleet occurrence of the script-swap law). `kld-then-calib.sh SKIP_KLD=1` resumes from the `rows: 50235` receipt. 4. **Calibration**: `run-swift15-bf16.sh calib` on **1.2.0** (the only engine with `RADIANCE_CALIB_TOKENS`; grams are engine-invariant bf16 activations — RADGRAM2 format byte-identical across versions) + `calib-push.py` pushing `calib-corpus.jsonl` (10,014,932 tokens, 1,294 prompts, assembled from real HF sources, chat-templated, source-labelled). The tap (in the `qwen4exp_fp8.so` architecture plugin) writes per-expert down-proj Gram matrices `gram.r.blk.L.ffn_down_exps..bin` (32-byte RADGRAM2 header + 640×640 f32; `rows` = routing count, `trace` = activation energy) and stops the serve at the token threshold. TP=2: whole experts per rank (256/rank); gate_up input is rank-sliced (plugin warning — the recipe only casts the down projection). 5. **Protected-expert derivation**: `derive-protected-experts.py /home/dank/corpora/calib` → the 24 highest-energy experts (the law is data-driven; Swift drew 24, not the fp8-era 10 — its routing is spikier: `blk.47 e445` alone carries 32 % of its layer's down energy at 70× routing share) promoted to whole-expert bf16. Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count per rank half; odd half gains the next most energetic. `apply-protected-cast.sh` splices the derived names between the `# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive). 6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) — gates: ≥48 gram files present, bf16 container already deleted (disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on 49 GPTQ layers, load-average 8), output **116,416 MiB**. The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT TABLE (`one table of experts takes one encoding, and only a whole expert kept plain bf16 is served beside it`) — a split tier (down=fp8, gate_up=w4) is unservable: the kernel serves a single w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated, gate_up uncalibrated), protected experts bf16 beside them. ## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe) | Tier | Encoding | Why | |---|---|---| | MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) | | 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) | | shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law | | hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law | | lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) | | PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau | | embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched | ## The shape (`scripts/frank-archon.env`) TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric v22.5 template override. `MODEL_FILE` points at `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves 177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that change is documented in its own section below. ## 2026-10-11 — unquantized KV at 2 × 200,000 The ask: two concurrent 200,000-token conversations on exact (unquantized) attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93 sessions of the served window**, so two full-height conversations overflow it and the scheduler preempts the overflow to the host/disk prefix tiers (re-prefilling on return). Measured, that overflow is survivable — the fp8 shape passed the same 2 × 197k pair probe — so this is a **headroom** change, not the repair of a hard failure. The hard limit the engine states is per request, not per pair: a request longer than the pool's token count is *"accepted by the API and then refused as too large to ever fit — this is a hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request ever hit it. ### The dtype law `--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as `--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the rung under bf16 is fp8 on both halves. Evidence: the flag table in the image itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any `cache-type-k`/`-v` symbol in the binary. bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table: | Group | fp8 page | bf16 page | cost per token, per rank | |---|---|---|---| | `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB | | `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed | | `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** | | prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed | → `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904; bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded as a test seam (`the KV pool backs seqs x max-model-len at the declared cache dtype`) so a future resize cannot silently under-provision the pool. ### Why the weights slab pays — and why that is not pure loss The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a boot consumes outside both (driver context, the 990 MiB activation arena, hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒ **`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800). Half of that loss is refunded by an engine behaviour worth knowing: **idle KV VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is a competing cache, and every KV resize is also an expert-residency resize. ### seqs 3 → 2 `--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM fork's torch.compile constraint (`ConstraintViolationError: Constraints violated (L['query_start_loc'].size()[0])`), retired with that runtime — see [[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied inside the ±3 % noise floor. Two slots × the window is the capacity contract; it also saves 110 MiB of per-slot recurrent state and drops the fused TP all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the wht6 wire is bit-exact. ### Boot ledger (deployed 2026-10-11, commit `0bf26b0`) ```text kv cache 6.75 GiB 349% of it -- backs 2 sequences all at 200000 tokens at once, which is the worst case vram_weights 21.69 GiB stated by --vram-weights-mib vram_kv 6.75 GiB stated by --vram-kv-mib unclaimed 2.03 GiB kv_attn full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens) kv_qsa_bkey full 13 layers block 4 tok page 3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens) load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB mover: 5146 slots can be lent out of the KV cache (6.08 GiB) card 0 settled at 1.85 GiB free serving on http://:::7770 ``` ### Verification (live service, no scratch container) | Check | Result | |---|---| | `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` | | 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s | | decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) | | fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section | Probe law re-learned, worth recording: radiance routes the thinking block to `reasoning_content`, so a chat request whose budget is spent inside `` comes back with **empty `content` and `finish_reason: stop`** — a probe that forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both fields is the only honest shape. The first pair probe scored 0/2 on exactly that artifact, not on a KV fault. ### Measured cost — A/B on the live service Method (no scratch container, no mask — the unit file is a relink symlink and `mask` refuses it): write a variant env file on the host (`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it (`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own `Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the drop-in, restart. Control = the deployed 2026-10-08 shape, verified from `podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071 --vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB). Identical probes, same salts, loopback, greedy, 512 generated tokens. | Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ | |---|---|---|---| | prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** | | prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** | | decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % | | decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % | | decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % | | draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** | | 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall | Read it as: **the price of exact KV on this engine is prefill, not decode.** Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s); both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass −24 % is the largest single number, the least reproducible, and it moves the same direction as prefill — the cold arm is paying expert streaming, not attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact attention state makes the target agree with the MTP drafter more often, which pays on every token of every conversation, and it is the strongest argument for the shape. Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8 variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8 for a 105 MiB pool bump and ≈ no decode cost. The tool is committed: `scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its prompts from the repo checkout there). ### Rollback Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3, host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min. The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away* from the scored configuration — bf16 can only reduce the attention-state error, and a re-score is a drained-window job if the gate agreement is ever needed in the record. ## Gates (measured at cutover) - **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`, candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126, p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**, **ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235 positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json` (+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072 (the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena, refused), KV 2048, headroom 512, no speculation. The engine ran rank 0 fully on card 0 (25.88 GiB) with rank 1 host-side in this shape — scoring is placement-invariant; scored at 28 positions/s. - **KLD gate re-PASS (2026-10-08 04:49) at the production-legal shape**: MNBT 2048 (the w4 file-tier law: experts stream in place only at MNBT ≤ 2048 — the recording's 32768 demands a widened form the disk tier refuses), pool 40960, vram-kv 3072 fp8, tp 2, headroom 768. **mean KL 0.0131, top-1 97.92 %**. The Δ vs the first PASS (0.0015) is the float-order noise band of the chunk-size change — the verdict is chunk-size invariant, ~2.3× inside the 0.03 bar on both shapes. - **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth (streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**, draft accept **0.7246** (350/483); arm C (deployed vram-pair shape 26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control (fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm. The w4 expert table is resident — the fp8-native era's NVMe expert streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60 t/s base decode. Conversation battery (short context): geomean 172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot: the w4 resident set fits fully inside the 40 GiB pool at 0.82 (ledger: mover full fill). Depth-output audit: the 44.5k-depth summary is a coherent ordered index of the concatenated corpus (no repetition, no NaN-token garbage) — the depth-corruption class of 2026-10-04 is clean. - Fleet probes: PASS, re-run on the deployed prod — vision (solid-red 512 px → "Red"), tool_calls (`{"expr":"23 * 47"}`), reasoning_effort xhigh/medium/low/high all 200 (the Swift1.5 froggeric template accepts the full set), reasoning_content present, response_format json_object honoured. **Vision probe law:** budget ≥ 256 tokens and read `reasoning_content` as fallback — at 64 tokens thinking exhausts the budget and `content` comes back empty (the answer lands in the reasoning channel). - **Deployed verification (2026-10-08)**: `deploy.sh dankserver --service frank-archon` green incl. Playwright e2e 33/33; unit `active`, `/health` 200, `/v1/models` id `frank-archon` (the harness alias is the served-model-name, independent of the file — models.yml untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2, seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert pools 57.84 GiB, cards settled 1.41 GiB free, `serving on :7770`. Live decode probe (512 tokens, streamed, temp 0): ~190 t/s. ### Wire rename + the bench drop-in trap (2026-10-09) `FRANK_ARCHON_SERVED_NAME` moved `frank-archon` → **frank-archon**; the service, unit, scripts and env file keep the old name. Radiance is the only fleet engine whose `/v1/models` listing reflects the flag, and it echoes the **requested** model name in chat responses — so a chat probe that returns 200 proves nothing about the served id. Always read `/v1/models`. The first deploy after the edit came up green and `/v1/models` still said `frank-archon`. Cause: a campaign leftover drop-in `/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09) overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the staging worktree `/home/dank/serverAppConfigs-swift15`. Every `deploy.sh dankserver --service frank-archon` since then restarted the staging tree: green health gate, old configuration, silently. Verified before removing: the staging `serve-frank-archon.sh` and `frank-archon.service` are byte-identical to the checkout, its `frank-archon.env` differed only by the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`. After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair 26071/3543). **Deployer blind spot:** `deploy.sh` plans from repo paths and gates on the port. It cannot see host-side drop-ins, so unit drift is invisible to a green run. After an LLM deploy, verify the served configuration behaviourally (here: `podman inspect --format {{json .Config.Cmd}}` and `/v1/models`). ## Deploy + verify ```bash sudo systemctl stop frank-archon # drained window bash convert-swift15-w4dn8.sh # if not already run ./scripts/deploy.sh dankserver --service frank-archon ``` Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger (expert pool coverage now w4-sized, KV pool should be larger than the FP8-native era's 399,824 tokens); MTP draft counts in `timings`; froggeric probe; ≥256 px `image_url` vision probe answers the color; tool-call probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh` + Playwright e2e green. ## Rollback | Layer | Path | |---|---| | FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-archon.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk | | FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` | | tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` | Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24 GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs `/home/dank/logs/swift15_*.log`. The BF16 `.rad` container is deleted before the final convert (disk gate) — the raw checkpoint is the re-convert source if the window ever reopens. ## Links - [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era - [[LLM/index]] — fleet index - [[swift15-campaign-worklog]] — campaign record - [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here