frank-archon: the slab is a threshold, and decode is where slots cost

Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape:
prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k —
the third slab step bought nothing, so the slab is a threshold whose cliff is
the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent
streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in
decode concurrency, the axis the shape hunt never measured because probe leg
B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode,
512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
This commit is contained in:
Frank
2026-10-13 20:51:32 -07:00
parent 2581b8d4b6
commit a9d45e7012
4 changed files with 89 additions and 6 deletions
+12 -4
View File
@@ -550,10 +550,18 @@ all at 200000 tokens at once, which is the worst case` — `kv_attn` page
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
4.22 GiB to the expert cache. Prefill tracks this slab: the 24,000 → 25,600
step measured +8.9 % single-stream (3,290 → 3,584 t/s at 43.5k) and +17.6 %
concurrent (3-way 197k aggregate), so the pinned host pool is the slow path and
every MiB moved from RAM onto the card is throughput.
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
not a slope.
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
aggregate**, so the slot count is paid in decode concurrency (a fused-step
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk