docs: final 256k verdict — boots at 4.03-4.5 GiB KV, band 34.1-36.5 straddles target

The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB
(planner buckets; the trim went to margin). Final 256k config
MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative
MAXLEN=184224 at 3 GiB.
This commit is contained in:
Frank committed 2026-09-29 21:55:58 -07:00
1 parent 0ccb01708b
commit 57072a44df
2 files changed
+23 -12

No files matched your search

@@ -87,7 +87,7 @@ Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** | | MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single | | 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared | | ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−14%) costs ~10%; 2 of 3 depths just under 35 | | 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
\* The TunableOp numbers are invalid by construction: the tuned GEMMs \* The TunableOp numbers are invalid by construction: the tuned GEMMs
changed generated behavior (repetition-loop, the server's changed generated behavior (repetition-loop, the server's
@@ -203,17 +203,19 @@ The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
lands in the 60+ tok/s band before acceptance effects. Code work in lands in the 60+ tok/s band before acceptance effects. Code work in
`expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
copy-in path itself is compiled — see `/app/r4dhip/r4d.so`). copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
2. **256k ctx — measured**: boots at `MAXLEN=262144 KV_BYTES=4831838208` 2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
(a 262,144-token window needs **4.0 GiB** KV — hybrid attention + `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token; needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
at 3 GiB the engine estimates max len 184,224). Decode not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−14%) and ~10% estimates max len 184,224 at that pool; the exact-4.0 override loses
decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override ~20 MB to accounting and boots as 3.98 < 4.0). Decode band
(0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge; **34.1-36.5** across both KV sizes — the arena is **identical (5,648
the alternative is `MAXLEN=184224` at 3 GiB KV, which keeps the full slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV margin, not slots), so the window costs ~10% decode vs the 32k config
override (the earlier "before" reading was wrong) — the arena is what and no KV trim recovers it. The alternative that keeps the full
remains. 37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
sizes the arena AFTER the KV override (the earlier "before" reading was
wrong) — the arena is what remains.
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at 3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx, 2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
so slots are a capacity feature, not a throughput one. Two full 256k so slots are a capacity feature, not a throughput one. Two full 256k
+9
View File
@@ -96,6 +96,15 @@
under 35). KV-exact-4.0 arm in flight for the last +3.5% slots. under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
Correction recorded: the planner sizes the arena AFTER the KV override Correction recorded: the planner sizes the arena AFTER the KV override
(the earlier "before" reading was wrong) — the arena is what remains. (the earlier "before" reading was wrong) — the arena is what remains.
- **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
(5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
band, straddling the 35 target within noise. Final 256k config:
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
## 2026-09-28 ## 2026-09-28
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to