docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law (only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why bf16 costs 1.89x and not 2x), the pool formula that reproduces both live boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's torch.compile constraint, not radiance's), the boot ledger, and the A/B. The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in: prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes, draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648). Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said ACTIVE), adds the worklog entry, and commits the verification tool as scripts/probe-frank-archon-kv.py — it carries the probe law that cost two false negatives: radiance answers in reasoning_content, so a chat probe with a small budget reads as empty content + finish_reason stop on a healthy server. Tests 368/368.
This commit is contained in:
@@ -1598,3 +1598,56 @@ every restart since the cutover.
|
||||
so the honest claim is ~184 s of inductor/AOT build recovered per restart —
|
||||
not a proportional boot cut.
|
||||
|
||||
## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000
|
||||
|
||||
- Ask: two concurrent 200,000-token conversations on exact attention state.
|
||||
Sized from the live fp8 boot ledger, not arithmetic: the paged groups cost
|
||||
7.3125 KiB/token per rank (`kv_attn` 26.00 KiB per 4-token block +
|
||||
`kv_qsa_bkey` 3.25), so the 3,543 MiB pool bought 386,904 tokens = **1.93
|
||||
sessions of the served window**.
|
||||
- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the
|
||||
whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot
|
||||
runs) has **no radiance equivalent**. Measured bf16 rate is **13.8125
|
||||
KiB/token = 1.89× fp8, not 2×**: the QSA block key (13 layers, 3.25 KiB per
|
||||
4 tokens) does not follow the dtype. Pool formula
|
||||
`tokens × rate + 110 MiB × seqs (recurrent state) + 448 MiB (prefix
|
||||
checkpoints)` reproduces both boots within 0.1 %, and is now a test seam.
|
||||
- Deployed (`0bf26b0` + docs): pool 3543 → **6912** MiB (462,752 tokens; ledger
|
||||
"backs 2 sequences all at 200000 tokens at once"), weights slab 26071 →
|
||||
22208, host pool 40960 → 44928 (the evicted experts land there; pinned pool
|
||||
slots 7,190 → 10,375), prefix host 4096 → 8192 (the entry grows 58.50 →
|
||||
110.50 KiB), TTM floor 140000 → 150000 (pinned ledger ~147 GiB, cap 204,800).
|
||||
`weights + kv` pinned to the measured ceiling **29,120 MiB/card**; cards
|
||||
settled 1.85 GiB free — more than the fp8 boot's 1.39, because the total
|
||||
claim dropped. Half the cost is refunded by the engine: idle KV VRAM is the
|
||||
expert cache's spillover, and the bigger pool *lends* 6.08 GiB where the fp8
|
||||
pool lent 2.69.
|
||||
- `--max-num-seqs 3 → 2`: the fleet's "seqs 2 cannot boot" law is the retired
|
||||
tcclaviger vLLM fork's torch.compile `query_start_loc` constraint, not
|
||||
radiance's — radiance booted seqs 2 in the fp8-native era and the 2026-10-06
|
||||
sweep had 2 and 3 tied inside ±3 %. Bonus: the fused all-reduce is declared
|
||||
over 2 × (1+2) = 6 rows, inside the 25-row bound where the wht6 wire is
|
||||
bit-exact.
|
||||
- Verified live (no scratch container, no bench): `/health` 200, served id
|
||||
`frank-archon`; 2 × 196,919 concurrent conversations each answered the
|
||||
marker buried at the head of its own prompt (393,837 tokens resident, above
|
||||
the fp8 pool's 386,904); decode at 51.5k depth 94.1 cold / 102.3 warm / 117.6
|
||||
prefix-hit t/s, draft acceptance 0.7284 / 0.7525.
|
||||
- fp8 control arm through a transient `FRANK_ARCHON_ENV_FILE` drop-in (the
|
||||
frank-zealot ubatch method, argv + ledger page size verified): prefill
|
||||
**−29 %** (4,379 → 3,126 cold, 4,864 → 3,333 warm), decode inside the noise
|
||||
band on two of three passes, TTFT at the full window **+25 %** (the pair:
|
||||
110.5 s → 138.1 s), draft acceptance **up 0.661 → 0.740**. The price is
|
||||
prefill — a 14.8 % smaller resident slab, and prefill is the pass that
|
||||
touches every expert — the gain is exact state plus acceptance. **Open call
|
||||
for Frank:** keep bf16, or take the cheap fp8 2 × 200k fit (pool 3,648 MiB,
|
||||
≈ free) and leave KV quantized.
|
||||
- Harness law re-learned: radiance puts the thinking block in
|
||||
`reasoning_content`, so a probe that reads `content` on a small budget sees
|
||||
empty text with `finish_reason: stop` — a healthy server scored 0/2 until the
|
||||
probe set `chat_template_kwargs.enable_thinking=false`. Tool committed as
|
||||
`scripts/probe-frank-archon-kv.py`.
|
||||
- Tests 368/368, e2e 33/33 (one unrelated homepage retry). Docs:
|
||||
[[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
|
||||
(the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user