From 57072a44df3e8f35084137a2c19de96a8a7dc6ee Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 29 Sep 2026 21:55:58 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20final=20256k=20verdict=20=E2=80=94=20bo?= =?UTF-8?q?ots=20at=204.03-4.5=20GiB=20KV,=20band=2034.1-36.5=20straddles?= =?UTF-8?q?=20target?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB (planner buckets; the trim went to margin). Final 256k config MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative MAXLEN=184224 at 3 GiB. --- ...server-qwen38-fp8-expert-offload-tuning.md | 26 ++++++++++--------- serverAppConfigs-docs/log.md | 9 +++++++ 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index b529762..bbbb7d6 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -87,7 +87,7 @@ Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a | MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** | | 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single | | ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared | -| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−14%) costs ~10%; 2 of 3 depths just under 35 | +| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes | \* The TunableOp numbers are invalid by construction: the tuned GEMMs changed generated behavior (repetition-loop, the server's @@ -203,17 +203,19 @@ The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left: lands in the 60+ tok/s band before acceptance effects. Code work in `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the copy-in path itself is compiled — see `/app/r4dhip/r4d.so`). -2. **256k ctx — measured**: boots at `MAXLEN=262144 KV_BYTES=4831838208` - (a 262,144-token window needs **4.0 GiB** KV — hybrid attention + - mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token; - at 3 GiB the engine estimates max len 184,224). Decode - 36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−14%) and ~10% - decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override - (0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge; - the alternative is `MAXLEN=184224` at 3 GiB KV, which keeps the full - 37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV - override (the earlier "before" reading was wrong) — the arena is what - remains. +2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with + `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window + needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token, + not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine + estimates max len 184,224 at that pool; the exact-4.0 override loses + ~20 MB to accounting and boots as 3.98 < 4.0). Decode band + **34.1-36.5** across both KV sizes — the arena is **identical (5,648 + slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to + margin, not slots), so the window costs ~10% decode vs the 32k config + and no KV trim recovers it. The alternative that keeps the full + 37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner + sizes the arena AFTER the KV override (the earlier "before" reading was + wrong) — the arena is what remains. 3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at 2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx, so slots are a capacity feature, not a throughput one. Two full 256k diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 6726939..2a0151b 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -96,6 +96,15 @@ under 35). KV-exact-4.0 arm in flight for the last +3.5% slots. Correction recorded: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains. + - **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB + boot-refused by 20 MB of accounting (3.98 available < 4.0 needed; + engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES= + 4328521728`) boots — but the arena is **identical to the 4.5 GiB arm + (5,648 slots)**: the planner buckets, the 0.47 GiB went to margin. + Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5 + band, straddling the 35 target within noise. Final 256k config: + `MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed + alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band). ## 2026-09-28 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to