frank-archon: the slab is a threshold, and decode is where slots cost
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
This commit is contained in:
@@ -421,6 +421,36 @@ of **199,984 tokens** each (599,952 tokens) were accepted and completed on
|
||||
`/v1/completions`; a fourth full-height request is refused as "too large to ever
|
||||
fit" — radiance never truncates.
|
||||
|
||||
### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
|
||||
|
||||
Frank's challenge — "why are you not tracking decode?" — was right, and the
|
||||
answer is a number now. 512 generated tokens (the bench law), greedy, cold
|
||||
prompts, this shape (3 x 200,000, fp8, slab 25,600):
|
||||
|
||||
```
|
||||
1-stream 43.5k: decode 99.0 t/s (prefill 4,042, accept 0.952)
|
||||
1-stream 197k: decode 93.3 t/s (prefill 4,826, accept 0.938)
|
||||
3-stream 43.5k: decode 113.9 / 43.3 / 10.9 t/s wall 83.4s
|
||||
3-stream 197k: decode 88.9 / 16.3 / 64.6 t/s wall 122.4s
|
||||
```
|
||||
|
||||
Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
|
||||
93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
|
||||
prefill, not tokens-per-second. **Concurrency is where the slot count is
|
||||
paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
|
||||
at 197k against 99 single-stream, and the split is a lottery
|
||||
(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
|
||||
to whoever it reached first, so one stream finishes at 113 t/s while its
|
||||
sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
|
||||
number shows it.
|
||||
|
||||
Corpus caveat, stated at the claim: this probe's filler is a repetitive
|
||||
ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
|
||||
table below; these rows flatter decode relative to that table and are the
|
||||
baseline for *this* corpus, not a replacement for those rows.
|
||||
`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
|
||||
tokens, which is why a day of shape work priced only prefill).
|
||||
|
||||
### Recovery evidence
|
||||
|
||||
```text
|
||||
|
||||
@@ -72,6 +72,26 @@
|
||||
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
|
||||
media libraries on it read empty. `frank-job-digest.service` (frankbook
|
||||
job-hunt digest) is in `list-units --state=failed`.
|
||||
- **The slab is a threshold, not a slope** — measured, same method (probe A
|
||||
leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
|
||||
22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
|
||||
bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
|
||||
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
|
||||
would clear it (pool 3,072 + slab 27,897) remains untested — and the
|
||||
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
|
||||
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
|
||||
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
|
||||
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
|
||||
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
|
||||
give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
|
||||
The slot count is paid in decode concurrency — a fused-step lottery, not a
|
||||
fair share — which is exactly the axis the shape hunt was blind to (probe
|
||||
leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
|
||||
is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
|
||||
table, so these rows are the baseline for this corpus, not a replacement.
|
||||
- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
|
||||
tokens, per-stream + aggregate) so no future shape change can be priced on
|
||||
prefill alone.
|
||||
|
||||
|
||||
## 2026-10-10
|
||||
|
||||
Reference in New Issue
Block a user