docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law (only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why bf16 costs 1.89x and not 2x), the pool formula that reproduces both live boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's torch.compile constraint, not radiance's), the boot ledger, and the A/B. The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in: prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes, draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648). Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said ACTIVE), adds the worklog entry, and commits the verification tool as scripts/probe-frank-archon-kv.py — it carries the probe law that cost two false negatives: radiance answers in reasoning_content, so a chat probe with a small budget reads as empty content + finish_reason stop on a healthy server. Tests 368/368.
This commit is contained in:
@@ -166,19 +166,29 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
|
||||
# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
|
||||
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
|
||||
# Measured per-rank cost, read out of the boot ledger's group table:
|
||||
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
|
||||
# 2026-10-08, bf16 boot 2026-10-11):
|
||||
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
|
||||
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB (fp8) -> 6.50 KiB
|
||||
# => 7.3125 KiB/token fp8, 14.625 KiB/token bf16
|
||||
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
|
||||
# the QSA block key is an index, not attention state, and does
|
||||
# NOT follow --kv-cache-dtype
|
||||
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
|
||||
# not 2x, precisely because the bkey plane is dtype-fixed.
|
||||
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
|
||||
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
|
||||
# checkpoint region 8 x 55.99 = 448 MiB. So:
|
||||
# pool_mib >= tokens x 14.625 KiB + 110 x seqs + 448
|
||||
# 6912 -> (6912 - 220 - 448) x 1024 / 14.625 = 437,320 tokens >= 2 x 200,000
|
||||
# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
|
||||
# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
|
||||
# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
|
||||
# and the ledger says it outright: "backs 2 sequences all at 200000 tokens
|
||||
# at once, which is the worst case".
|
||||
# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
|
||||
# sessions, and a third long conversation preempted to the host/disk tiers).
|
||||
# Secondary cost: bf16 doubles the prefix-cache entry (58.50 -> 117 KiB), so
|
||||
# the host tier below doubles too.
|
||||
# Secondary cost: the prefix-cache entry grows with the attention plane
|
||||
# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
|
||||
# the same number of parked conversations, out of RAM the host has
|
||||
# (104 GiB available with the service loaded).
|
||||
FRANK_ARCHON_KV_DTYPE=bf16
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
|
||||
Reference in New Issue
Block a user