docs: final 256k verdict — boots at 4.03-4.5 GiB KV, band 34.1-36.5 straddles target

The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB
(planner buckets; the trim went to margin). Final 256k config
MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative
MAXLEN=184224 at 3 GiB.
This commit is contained in:
Frank
2026-09-29 21:55:58 -07:00
parent 0ccb01708b
commit 57072a44df
2 changed files with 23 additions and 12 deletions
@@ -87,7 +87,7 @@ Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−14%) costs ~10%; 2 of 3 depths just under 35 |
| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
\* The TunableOp numbers are invalid by construction: the tuned GEMMs
changed generated behavior (repetition-loop, the server's
@@ -203,17 +203,19 @@ The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
lands in the 60+ tok/s band before acceptance effects. Code work in
`expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
2. **256k ctx — measured**: boots at `MAXLEN=262144 KV_BYTES=4831838208`
(a 262,144-token window needs **4.0 GiB** KV — hybrid attention +
mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token;
at 3 GiB the engine estimates max len 184,224). Decode
36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−14%) and ~10%
decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override
(0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge;
the alternative is `MAXLEN=184224` at 3 GiB KV, which keeps the full
37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV
override (the earlier "before" reading was wrong) — the arena is what
remains.
2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
`KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
estimates max len 184,224 at that pool; the exact-4.0 override loses
~20 MB to accounting and boots as 3.98 < 4.0). Decode band
**34.1-36.5** across both KV sizes — the arena is **identical (5,648
slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
margin, not slots), so the window costs ~10% decode vs the 32k config
and no KV trim recovers it. The alternative that keeps the full
37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
sizes the arena AFTER the KV override (the earlier "before" reading was
wrong) — the arena is what remains.
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
so slots are a capacity feature, not a throughput one. Two full 256k
+9
View File
@@ -96,6 +96,15 @@
under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
Correction recorded: the planner sizes the arena AFTER the KV override
(the earlier "before" reading was wrong) — the arena is what remains.
- **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
(5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
band, straddling the 35 target within noise. Final 256k config:
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
## 2026-09-28
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to