From 066813ea951f9d3a8dacbe9596c8ee8dc9f0675b Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 13 Oct 2026 20:00:39 -0700 Subject: [PATCH] llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Frank's call: three slots, and spend the KV the third slot doesn't need on resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`, `--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured 29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %. The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the contract buys nothing. At fp8 the slab curve measured on this host is 22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so 24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check. The cost is named, not hidden: a FOURTH full-height conversation no longer fits (4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit — radiance never truncates. Reverting to four slots is one env pair. Seam: two new arithmetic gates, because both failures are boot-then-400 shape faults that no flag check catches — the pair must respect the per-card ceiling, and the pool must back seqs x max-model-len at the dtype's measured rate plus the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of prefix checkpoints). --- scripts/frank-archon.env | 82 ++++++++++++++++++++++------------- scripts/serve-frank-archon.sh | 7 +-- tests/frank_archon_unit.bats | 30 +++++++++++-- 3 files changed, 82 insertions(+), 37 deletions(-) diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 110a9c0..82f81ad 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -3,13 +3,18 @@ # built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2), # dankserver :7770. # -# SHAPE (2026-10-13): FOUR decode slots x 200,000 tokens with an fp8 -# (quantized) KV cache. fp8 is what makes four slots affordable: the pool must -# back seqs x max-model-len, and at bf16 that costs 11,679 MiB (weights would -# fall to 17,441 MiB, -33 %), at fp8 only 6,601 MiB — so the slab stays at its -# 2026-10-11 size. See the KV block for the arithmetic and the measured dtype -# law. The fp8-native payload this file was written for is the rollback target -# and keeps its own trio under the fp8-native runbook. +# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV, +# with the per-card budget re-cut in the slab's favour: 24,000 MiB of resident +# experts + 5,120 MiB of pool = the measured 29,120 MiB ceiling. The pool is +# sized to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; +# 5,120 predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the +# slab is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, +# TTFT +25 % at the full window) while KV beyond the contract buys nothing. +# Consequence, deliberate: a FOURTH full-height conversation does not fit this +# pair, and the engine refuses it — it never truncates. See the KV block for the +# arithmetic and the measured dtype law. The fp8-native payload this file was +# written for is the rollback target and keeps its own trio under the +# fp8-native runbook. # # 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in # pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the @@ -116,7 +121,7 @@ FRANK_ARCHON_TP=2 # +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the # engine default. FRANK_ARCHON_TP_WIRE=wht6 -# Decode slots: FOUR, so the served window is a CAPACITY CONTRACT — four +# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three # concurrent conversations of --max-model-len fit the KV pool (see the KV # block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the # tcclaviger vLLM fork's torch.compile shape specialization on @@ -124,13 +129,21 @@ FRANK_ARCHON_TP_WIRE=wht6 # with that runtime). Radiance booted seqs 2 at the fp8-native first boot, # and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied # (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency, -# not speed, so the price of the fourth slot is KV, not decode. +# not speed, so a slot costs KV, never decode. # Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV # pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows — -# 4 x 3 = 12 here, still under the 25-row bound at which the wht6 wire stays -# EXACT (single-sequence decode and any step of <= 25 rows is bit-exact; only -# busier steps are lossy). -FRANK_ARCHON_MAX_SEQS=4 +# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT +# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier +# steps are lossy). +# The pool is sized to this contract and nothing more (see the KV block): +# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into +# the resident expert slab. The trade is explicit and it is Frank's call: +# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs +# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused +# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party +# traffic that wants four full-height sessions is a --vram-kv-mib change, not +# a code change. +FRANK_ARCHON_MAX_SEQS=3 # Served window. Native is 262,144; 200,000 is the publisher's production # value (compose) and what models.yml's contextWindow must quote — a longer # prompt is refused by the engine, never truncated. The engine treats @@ -163,13 +176,18 @@ FRANK_ARCHON_GPU_HEADROOM_MIB=96 # lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for # a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's # other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11 -# exact-KV A/B measured the price of this exact slab size (22208): prefill -# -29 %, TTFT at the full window +25 %, decode inside the noise band on 2 of 3 -# passes, draft acceptance UP 0.661 -> 0.740. Since 2026-10-13 the pool is fp8 -# (see the KV block), which is what funds the FOURTH slot at this slab size; -# re-tune from a boot ledger + probe, never from that estimate. -FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208 -FRANK_ARCHON_VRAM_KV_MIB=6912 +# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071): +# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on +# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740. +# 2026-10-13 (late): THREE slots, so the pool is cut to the contract and the +# freed 1,792 MiB goes to the slab (22208 -> 24000, +8.1 %). The direction the +# A/B data points: prefill is slab-limited, KV beyond the contract is not. The +# margin is thin by design (608,027 predicted tokens against the 600,000 +# contract), so the boot ledger's `backs N sequences all at 200000 tokens at +# once` line is the referee for THIS pair and the concurrent probe is its +# regression test. +FRANK_ARCHON_VRAM_WEIGHTS_MIB=24000 +FRANK_ARCHON_VRAM_KV_MIB=5120 # KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is # ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's # --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is @@ -186,20 +204,22 @@ FRANK_ARCHON_VRAM_KV_MIB=6912 # copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix # checkpoint region 8 x 55.99 = 448 MiB. So: # pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448 -# 4 x 200,000 at fp8 needs 5,713 + 440 + 448 = 6,601 MiB; the pool is 6,912, -# and it predicts (6912 - 440 - 448) x 1024 / 7.3125 = 843,566 tokens, i.e. -# FOUR full-height conversations with 5.4 % slack. The ledger line is the -# referee ("backs N sequences all at 200000 tokens at once"). -# At bf16 the same four slots cost 10,791 + 888 = 11,679 MiB, which would -# cut the resident slab to 17,441 MiB (-33 % vs the fp8-era 26,071) — that -# is why fp8 is not a preference here, it is the only dtype that funds four -# slots at this window. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed -# 462,752 tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 -# (1.93 sessions). +# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120, +# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the +# three-slot contract with 1.3 % slack, every MiB above 5,063 having been +# handed to the expert slab instead. The ledger line is the referee ("backs N +# sequences all at 200000 tokens at once") — a pair this tight is read off the +# boot, never assumed. +# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut +# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 = +# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both +# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752 +# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93 +# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276. # Secondary saving: the prefix-cache entry shrinks with the attention plane # (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot # ledgers' `kv tier transfer` lines), so the host tier below parks twice as -# many conversations as it did at bf16 — no re-size needed to park four slots. +# many conversations as it did at bf16 — no re-size needed to park three slots. FRANK_ARCHON_KV_DTYPE=fp8 # Spec decode: the checkpoint's own MTP head at the publisher's depth-3 # operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the diff --git a/scripts/serve-frank-archon.sh b/scripts/serve-frank-archon.sh index 35e38ea..a5725bc 100755 --- a/scripts/serve-frank-archon.sh +++ b/scripts/serve-frank-archon.sh @@ -13,9 +13,10 @@ # log states every budget before anything is allocated (weights vs KV # split, the expert host pool's coverage of the worst case, the PLE pin, # the P2P answer per pair) — read it after every resize. -# Deployed shape (2026-10-13): FOUR decode slots x 200,000 tokens, fp8 -# (quantized) KV in a 6,912 MiB pool — the dtype that funds four slots -# without cutting the weights slab below its 22,208 MiB size, 44 GiB/rank +# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8 +# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a +# 24,000 MiB resident expert slab — the pair is the measured 29,120 MiB/card +# ceiling, spent on the slab because prefill is slab-limited. 44 GiB/rank # pinned expert pool, MTP depth 2, prefix cache host 8 GiB + disk 64 GiB, PLE # pinned in RAM. # diff --git a/tests/frank_archon_unit.bats b/tests/frank_archon_unit.bats index da76b05..5d24c2b 100755 --- a/tests/frank_archon_unit.bats +++ b/tests/frank_archon_unit.bats @@ -133,14 +133,14 @@ nosudo_bin() { grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE" ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE") [ "$ml" -ge 177356 ] - # Concurrency: 4 slots x the served window is the capacity contract, and - # the fused TP all-reduce is declared over seqs x (1 + spec) = 12 rows, + # Concurrency: 3 slots x the served window is the capacity contract, and + # the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows, # under the 25-row bound at which the wht6 wire stays bit-exact. The # "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3 # tied inside the +-3% noise floor — slots buy concurrency, not speed. - grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE" + grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE" grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE" @@ -154,6 +154,30 @@ nosudo_bin() { grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE" grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE" grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE" + # Two arithmetic contracts the engine ENFORCES at request/boot time, so + # they are gates and not lore (a bad pair boots, then 400s every long + # request — the 2026-10-11 pool of 6,912 MiB at seqs 4 accepted 200k-token + # chats and refused them as "too large to ever fit"): + # 1. vram_weights + vram_kv <= the measured 29,120 MiB/card ceiling + # (31,856 card - 1,710 consumed outside both budgets - 1,024 kept free) + # 2. the pool must back seqs x max-model-len at the dtype's rate, plus + # the dtype-INDEPENDENT terms that live in the pool: ~110 MiB of + # recurrent state a sequence and 448 MiB of prefix checkpoints. + # Rates are the two boot ledgers' group tables: 7,488 B/token fp8, + # 14,144 B/token bf16 (the QSA block key stays 3.25 KiB at both). + w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE") + kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE") + seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE") + dt=$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE") + [ $(( w + kv )) -le 29120 ] + case "$dt" in + fp8) rate=7488 ;; + bf16) rate=14144 ;; + *) return 1 ;; + esac + tokens=$(( seqs * ml )) + need=$(( tokens * rate / 1048576 + 110 * seqs + 448 )) + [ "$kv" -ge "$need" ] grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE" # Prefix cache tiers (publisher pattern; disk tier sized for 2 slots). grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"