llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB
Frank's call: three slots, and spend the KV the third slot doesn't need on resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`, `--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured 29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %. The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the contract buys nothing. At fp8 the slab curve measured on this host is 22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so 24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check. The cost is named, not hidden: a FOURTH full-height conversation no longer fits (4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit — radiance never truncates. Reverting to four slots is one env pair. Seam: two new arithmetic gates, because both failures are boot-then-400 shape faults that no flag check catches — the pair must respect the per-card ceiling, and the pool must back seqs x max-model-len at the dtype's measured rate plus the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of prefix checkpoints).
This commit is contained in:
1 parent
7daa886dd6
commit
066813ea95
3 files changed
+82
-37
No files matched your search
@@ -13,9 +13,10 @@
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize.
|
||||
# Deployed shape (2026-10-13): FOUR decode slots x 200,000 tokens, fp8
|
||||
# (quantized) KV in a 6,912 MiB pool — the dtype that funds four slots
|
||||
# without cutting the weights slab below its 22,208 MiB size, 44 GiB/rank
|
||||
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
|
||||
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
|
||||
# 24,000 MiB resident expert slab — the pair is the measured 29,120 MiB/card
|
||||
# ceiling, spent on the slab because prefill is slab-limited. 44 GiB/rank
|
||||
# pinned expert pool, MTP depth 2, prefix cache host 8 GiB + disk 64 GiB, PLE
|
||||
# pinned in RAM.
|
||||
#
|
||||
|
||||
Reference in new issue
Block a user