docs: qwen38-fp8 expert-offload tuning — prefetch off hits 37.4-40.6 tok/s (target met)

Measured campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4
fork, 2x R9700, TP=2): telemetry (with_stack=false) measured the
prefetcher at 36% precision — ~half of gather traffic wasted; deleting
--expert-prefetch: 26.8 -> 40.57/37.44/37.57 tok/s. Prefetch-on shape
survived every other challenger (gather geometry, spec depth, TunableOp
[corrupts numerics], MNBT, concurrency, ngram [PLE fork bug]). 256k ctx
boots at 4.5 GiB KV (36.65/34.21/34.77, arena -14%). New runbook +
index links + log entry.

serverAppConfigs-docs/log.md

@@ -45,6 +45,57 @@
     `=` keys the dict `ttft;dur` and every lookup silently returns nothing.
     Four configurations reported "no Server-Timing" while the header was
     present on all of them.
+- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s,
+  mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4
+  fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full
+  flag-level A/B battery at the 32k working point; every number is from the
+  server's usage counters (SSE chunks are invalid under spec decode). New
+  runbook: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+  - Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1,
+    TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx,
+    acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
+  - **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64
+    (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the
+    stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in
+    ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves
+    the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral);
+    prefetch depth 2 (neutral — the tier is 16 slots/layer against a
+    ~2,400-slab working set at 73% misses); 2-way concurrency (summed
+    25.49 < single 26.8 — under TP both streams' misses gather on each
+    rank, the links do not add capacity, and the shared arena thrashes).
+  - **TunableOp rejected:** it silently skips under expert offload unless
+    `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics —
+    repetition loops, the server's repetition-halt firing at ~140 of 256
+    tokens, "79.8 tok/s" measured on halted degenerate completions, and a
+    fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no
+    persistent cache in the container lifecycle.
+  - **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free
+    at worker init); `--max-num-seqs 2` (compile constraint violation at
+    CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not
+    prepared" — PLE prep is wired only for the MTP spec path).
+  - **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction;
+    the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91%
+    of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s =
+    42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait
+    (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the
+    fork's gather read path — fork-patch territory, not flags.
+  - **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm
+    (>10 min, two attempts, while the AsyncLLM wrapper logs success in
+    1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min
+    profile window's trace buffer ~40 GB → the kernel OOM-killed the engine
+    core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer
+    expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY
+    through that stop path, so the telemetry JSON is unreachable without a
+    fork patch.
+  - **256k ctx boots** (the last original-goal dimension): a 262,144-token
+    window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈
+    16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused
+    (`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate
+    at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77**
+    (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just
+    under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
+    Correction recorded: the planner sizes the arena AFTER the KV override
+    (the earlier "before" reading was wrong) — the arena is what remains.
 
 ## 2026-09-28
 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to