frank-archon: the slab is a threshold, and decode is where slots cost

Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
2026-10-13 20:51:32 -07:00
parent 2581b8d4b6
commit a9d45e7012
4 changed files with 89 additions and 6 deletions
@@ -421,6 +421,36 @@ of **199,984 tokens** each (599,952 tokens) were accepted and completed on
 `/v1/completions`; a fourth full-height request is refused as "too large to ever
 fit" — radiance never truncates.

+### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
+
+Frank's challenge — "why are you not tracking decode?" — was right, and the
+answer is a number now. 512 generated tokens (the bench law), greedy, cold
+prompts, this shape (3 x 200,000, fp8, slab 25,600):
+
+```
+1-stream  43.5k: decode  99.0 t/s  (prefill 4,042, accept 0.952)
+1-stream   197k: decode  93.3 t/s  (prefill 4,826, accept 0.938)
+3-stream  43.5k: decode 113.9 / 43.3 / 10.9 t/s   wall 83.4s
+3-stream   197k: decode  88.9 / 16.3 / 64.6 t/s   wall 122.4s
+```
+
+Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
+93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
+prefill, not tokens-per-second. **Concurrency is where the slot count is
+paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
+at 197k against 99 single-stream, and the split is a lottery
+(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
+to whoever it reached first, so one stream finishes at 113 t/s while its
+sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
+number shows it.
+
+Corpus caveat, stated at the claim: this probe's filler is a repetitive
+ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
+table below; these rows flatter decode relative to that table and are the
+baseline for *this* corpus, not a replacement for those rows.
+`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
+tokens, which is why a day of shape work priced only prefill).
+
 ### Recovery evidence

 ```text