dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe

Browse Source 
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.

The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).

Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
This commit is contained in:
Frank
2026-10-11 01:36:48 -07:00
parent 0bf26b0741
commit c52ba781e3
7 changed files with 431 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_archon_unit.bats
+12 -10
View File
@@ -164,21 +164,23 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
@test "the KV pool backs seqs x max-model-len at the declared cache dtype" {
# The engine's own hard limit: a request longer than the pool's token
# count is "accepted by the API and then refused as too large to ever fit
# -- this is a hard limit, not a backing ratio". Rates measured from the
# live boot ledger's group table, per rank: kv_attn (13 layers) is a
# 26.00 KiB page per 4 tokens at fp8 and kv_qsa_bkey (13 layers) 3.25 KiB
# -> 7,488 B/token, and bf16 doubles both -> 14,976 B/token. The terms
# that do NOT follow the dtype also live in the pool: ~110 MiB a sequence
# of recurrent state (kv_gdn_state 54 MiB x 2 copies + conv/tail/ple) and
# 448 MiB of prefix-cache checkpoints (8 x 55.99 MiB). Verified against
# the fp8 boot it replaces: 3543 - 330 - 448 = 2765 MiB -> 387,240 tokens,
# ledger said 386,904.
# -- this is a hard limit, not a backing ratio". Rates are measured, not
# derived: per rank, kv_attn (13 layers) pages 26.00 KiB per 4 tokens at
# fp8 and 52.00 KiB at bf16, while kv_qsa_bkey (13 layers) pages 3.25 KiB
# at BOTH — the QSA block key is an index, not attention state, and does
# not follow --kv-cache-dtype — so the rate is 7,488 B/token at fp8 and
# 14,144 at bf16 (1.89x, not 2x). The dtype-independent terms also live in
# the pool: ~110 MiB a sequence of recurrent state (kv_gdn_state 54 MiB x 2
# copies + conv/tail/ple) and 448 MiB of prefix checkpoints (8 x 55.99).
# Verified against BOTH live boots: fp8 3543-330-448 = 2765 MiB -> 387,240
# tokens (ledger 386,904); bf16 6912-220-448 = 6244 MiB -> 462,946
# (ledger 462,752). Both within 0.1%.
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
maxlen=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
case "$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE")" in
fp8) bpt=7488 ;;
bf16) bpt=14976 ;;
bf16) bpt=14144 ;;
*) fail "unknown FRANK_ARCHON_KV_DTYPE" ;;
esac
tokens=$(( (kv - 110 * seqs - 448) * 1048576 / bpt ))