docs: final 256k verdict — boots at 4.03-4.5 GiB KV, band 34.1-36.5 straddles target
The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB (planner buckets; the trim went to margin). Final 256k config MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative MAXLEN=184224 at 3 GiB.
This commit is contained in:
@@ -87,7 +87,7 @@ Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
|
||||
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
|
||||
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
|
||||
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
|
||||
| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−14%) costs ~10%; 2 of 3 depths just under 35 |
|
||||
| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
|
||||
|
||||
\* The TunableOp numbers are invalid by construction: the tuned GEMMs
|
||||
changed generated behavior (repetition-loop, the server's
|
||||
@@ -203,17 +203,19 @@ The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
|
||||
lands in the 60+ tok/s band before acceptance effects. Code work in
|
||||
`expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
|
||||
copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
|
||||
2. **256k ctx — measured**: boots at `MAXLEN=262144 KV_BYTES=4831838208`
|
||||
(a 262,144-token window needs **4.0 GiB** KV — hybrid attention +
|
||||
mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token;
|
||||
at 3 GiB the engine estimates max len 184,224). Decode
|
||||
36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−14%) and ~10%
|
||||
decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override
|
||||
(0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge;
|
||||
the alternative is `MAXLEN=184224` at 3 GiB KV, which keeps the full
|
||||
37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV
|
||||
override (the earlier "before" reading was wrong) — the arena is what
|
||||
remains.
|
||||
2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
|
||||
`KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
|
||||
needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
|
||||
not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
|
||||
estimates max len 184,224 at that pool; the exact-4.0 override loses
|
||||
~20 MB to accounting and boots as 3.98 < 4.0). Decode band
|
||||
**34.1-36.5** across both KV sizes — the arena is **identical (5,648
|
||||
slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
|
||||
margin, not slots), so the window costs ~10% decode vs the 32k config
|
||||
and no KV trim recovers it. The alternative that keeps the full
|
||||
37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
|
||||
sizes the arena AFTER the KV override (the earlier "before" reading was
|
||||
wrong) — the arena is what remains.
|
||||
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
|
||||
2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
|
||||
so slots are a capacity feature, not a throughput one. Two full 256k
|
||||
|
||||
@@ -96,6 +96,15 @@
|
||||
under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
|
||||
Correction recorded: the planner sizes the arena AFTER the KV override
|
||||
(the earlier "before" reading was wrong) — the arena is what remains.
|
||||
- **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
|
||||
boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
|
||||
engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
|
||||
4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
|
||||
(5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
|
||||
Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
|
||||
band, straddling the 35 target within noise. Final 256k config:
|
||||
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
|
||||
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
|
||||
|
||||
## 2026-09-28
|
||||
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
||||
|
||||
Reference in New Issue
Block a user