docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe

Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.

The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).

Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
This commit is contained in:
Frank
2026-10-11 01:36:48 -07:00
parent 0bf26b0741
commit c52ba781e3
7 changed files with 431 additions and 46 deletions
+17 -7
View File
@@ -166,19 +166,29 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
# Measured per-rank cost, read out of the boot ledger's group table:
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
# 2026-10-08, bf16 boot 2026-10-11):
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB (fp8) -> 6.50 KiB
# => 7.3125 KiB/token fp8, 14.625 KiB/token bf16
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
# the QSA block key is an index, not attention state, and does
# NOT follow --kv-cache-dtype
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
# not 2x, precisely because the bkey plane is dtype-fixed.
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
# checkpoint region 8 x 55.99 = 448 MiB. So:
# pool_mib >= tokens x 14.625 KiB + 110 x seqs + 448
# 6912 -> (6912 - 220 - 448) x 1024 / 14.625 = 437,320 tokens >= 2 x 200,000
# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
# and the ledger says it outright: "backs 2 sequences all at 200000 tokens
# at once, which is the worst case".
# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
# sessions, and a third long conversation preempted to the host/disk tiers).
# Secondary cost: bf16 doubles the prefix-cache entry (58.50 -> 117 KiB), so
# the host tier below doubles too.
# Secondary cost: the prefix-cache entry grows with the attention plane
# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
# the same number of parked conversations, out of RAM the host has
# (104 GiB available with the service loaded).
FRANK_ARCHON_KV_DTYPE=bf16
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the