frank-archon: the slab is a threshold, and decode is where slots cost
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
This commit is contained in:
@@ -550,10 +550,18 @@ all at 200000 tokens at once, which is the worst case` — `kv_attn` page
|
||||
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
|
||||
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
|
||||
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
|
||||
4.22 GiB to the expert cache. Prefill tracks this slab: the 24,000 → 25,600
|
||||
step measured +8.9 % single-stream (3,290 → 3,584 t/s at 43.5k) and +17.6 %
|
||||
concurrent (3-way 197k aggregate), so the pinned host pool is the slow path and
|
||||
every MiB moved from RAM onto the card is throughput.
|
||||
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
|
||||
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
|
||||
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
|
||||
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
|
||||
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
|
||||
not a slope.
|
||||
|
||||
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
|
||||
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
|
||||
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
|
||||
aggregate**, so the slot count is paid in decode concurrency (a fused-step
|
||||
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
|
||||
Reference in New Issue
Block a user