dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB

Browse Source 
Frank's call: three slots, and spend the KV the third slot doesn't need on
resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`,
`--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured
29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token
contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %.

The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the
contract buys nothing. At fp8 the slab curve measured on this host is
22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so
24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check.

The cost is named, not hidden: a FOURTH full-height conversation no longer fits
(4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit —
radiance never truncates. Reverting to four slots is one env pair.

Seam: two new arithmetic gates, because both failures are boot-then-400 shape
faults that no flag check catches — the pair must respect the per-card ceiling,
and the pool must back seqs x max-model-len at the dtype's measured rate plus
the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of
prefix checkpoints).
This commit is contained in:
Frank
2026-10-13 20:00:39 -07:00
parent 7daa886dd6
commit 066813ea95
3 changed files with 82 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-archon.env
+51 -31
View File
@@ -3,13 +3,18 @@
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
# dankserver :7770.
#
# SHAPE (2026-10-13): FOUR decode slots x 200,000 tokens with an fp8
# (quantized) KV cache. fp8 is what makes four slots affordable: the pool must
# back seqs x max-model-len, and at bf16 that costs 11,679 MiB (weights would
# fall to 17,441 MiB, -33 %), at fp8 only 6,601 MiB — so the slab stays at its
# 2026-10-11 size. See the KV block for the arithmetic and the measured dtype
# law. The fp8-native payload this file was written for is the rollback target
# and keeps its own trio under the fp8-native runbook.
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
# with the per-card budget re-cut in the slab's favour: 24,000 MiB of resident
# experts + 5,120 MiB of pool = the measured 29,120 MiB ceiling. The pool is
# sized to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8;
# 5,120 predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the
# slab is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %,
# TTFT +25 % at the full window) while KV beyond the contract buys nothing.
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
# pair, and the engine refuses it — it never truncates. See the KV block for the
# arithmetic and the measured dtype law. The fp8-native payload this file was
# written for is the rollback target and keeps its own trio under the
# fp8-native runbook.
#
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
@@ -116,7 +121,7 @@ FRANK_ARCHON_TP=2
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
# engine default.
FRANK_ARCHON_TP_WIRE=wht6
# Decode slots: FOUR, so the served window is a CAPACITY CONTRACT — four
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
# concurrent conversations of --max-model-len fit the KV pool (see the KV
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
# tcclaviger vLLM fork's torch.compile shape specialization on
@@ -124,13 +129,21 @@ FRANK_ARCHON_TP_WIRE=wht6
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
# not speed, so the price of the fourth slot is KV, not decode.
# not speed, so a slot costs KV, never decode.
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
# 4 x 3 = 12 here, still under the 25-row bound at which the wht6 wire stays
# EXACT (single-sequence decode and any step of <= 25 rows is bit-exact; only
# busier steps are lossy).
FRANK_ARCHON_MAX_SEQS=4
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
# steps are lossy).
# The pool is sized to this contract and nothing more (see the KV block):
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
# the resident expert slab. The trade is explicit and it is Frank's call:
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
# a code change.
FRANK_ARCHON_MAX_SEQS=3
# Served window. Native is 262,144; 200,000 is the publisher's production
# value (compose) and what models.yml's contextWindow must quote — a longer
# prompt is refused by the engine, never truncated. The engine treats
@@ -163,13 +176,18 @@ FRANK_ARCHON_GPU_HEADROOM_MIB=96
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
# exact-KV A/B measured the price of this exact slab size (22208): prefill
# -29 %, TTFT at the full window +25 %, decode inside the noise band on 2 of 3
# passes, draft acceptance UP 0.661 -> 0.740. Since 2026-10-13 the pool is fp8
# (see the KV block), which is what funds the FOURTH slot at this slab size;
# re-tune from a boot ledger + probe, never from that estimate.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208
FRANK_ARCHON_VRAM_KV_MIB=6912
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract and the
# freed 1,792 MiB goes to the slab (22208 -> 24000, +8.1 %). The direction the
# A/B data points: prefill is slab-limited, KV beyond the contract is not. The
# margin is thin by design (608,027 predicted tokens against the 600,000
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
# once` line is the referee for THIS pair and the concurrent probe is its
# regression test.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=24000
FRANK_ARCHON_VRAM_KV_MIB=5120
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
@@ -186,20 +204,22 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
# checkpoint region 8 x 55.99 = 448 MiB. So:
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
# 4 x 200,000 at fp8 needs 5,713 + 440 + 448 = 6,601 MiB; the pool is 6,912,
# and it predicts (6912 - 440 - 448) x 1024 / 7.3125 = 843,566 tokens, i.e.
# FOUR full-height conversations with 5.4 % slack. The ledger line is the
# referee ("backs N sequences all at 200000 tokens at once").
# At bf16 the same four slots cost 10,791 + 888 = 11,679 MiB, which would
# cut the resident slab to 17,441 MiB (-33 % vs the fp8-era 26,071) — that
# is why fp8 is not a preference here, it is the only dtype that funds four
# slots at this window. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed
# 462,752 tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904
# (1.93 sessions).
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
# handed to the expert slab instead. The ledger line is the referee ("backs N
# sequences all at 200000 tokens at once") — a pair this tight is read off the
# boot, never assumed.
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
# Secondary saving: the prefix-cache entry shrinks with the attention plane
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
# many conversations as it did at bf16 — no re-size needed to park four slots.
# many conversations as it did at bf16 — no re-size needed to park three slots.
FRANK_ARCHON_KV_DTYPE=fp8
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
scripts/serve-frank-archon.sh
+4 -3
View File
@@ -13,9 +13,10 @@
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize.
# Deployed shape (2026-10-13): FOUR decode slots x 200,000 tokens, fp8
# (quantized) KV in a 6,912 MiB pool — the dtype that funds four slots
# without cutting the weights slab below its 22,208 MiB size, 44 GiB/rank
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
# 24,000 MiB resident expert slab — the pair is the measured 29,120 MiB/card
# ceiling, spent on the slab because prefill is slab-limited. 44 GiB/rank
# pinned expert pool, MTP depth 2, prefix cache host 8 GiB + disk 64 GiB, PLE
# pinned in RAM.
#
tests/frank_archon_unit.bats
+27 -3
View File
@@ -133,14 +133,14 @@ nosudo_bin() {
grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
[ "$ml" -ge 177356 ]
# Concurrency: 4 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 12 rows,
# Concurrency: 3 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
# under the 25-row bound at which the wht6 wire stays bit-exact. The
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -154,6 +154,30 @@ nosudo_bin() {
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
# Two arithmetic contracts the engine ENFORCES at request/boot time, so
# they are gates and not lore (a bad pair boots, then 400s every long
# request — the 2026-10-11 pool of 6,912 MiB at seqs 4 accepted 200k-token
# chats and refused them as "too large to ever fit"):
# 1. vram_weights + vram_kv <= the measured 29,120 MiB/card ceiling
# (31,856 card - 1,710 consumed outside both budgets - 1,024 kept free)
# 2. the pool must back seqs x max-model-len at the dtype's rate, plus
# the dtype-INDEPENDENT terms that live in the pool: ~110 MiB of
# recurrent state a sequence and 448 MiB of prefix checkpoints.
# Rates are the two boot ledgers' group tables: 7,488 B/token fp8,
# 14,144 B/token bf16 (the QSA block key stays 3.25 KiB at both).
w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
dt=$(awk -F= '/^FRANK_ARCHON_KV_DTYPE=/{print $2}' "$ENV_FILE")
[ $(( w + kv )) -le 29120 ]
case "$dt" in
fp8) rate=7488 ;;
bf16) rate=14144 ;;
*) return 1 ;;
esac
tokens=$(( seqs * ml ))
need=$(( tokens * rate / 1048576 + 110 * seqs + 448 ))
[ "$kv" -ge "$need" ]
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"