docs: qwen38-fp8 expert-offload tuning — prefetch off hits 37.4-40.6 tok/s (target met)

Measured campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4
fork, 2x R9700, TP=2): telemetry (with_stack=false) measured the
prefetcher at 36% precision — ~half of gather traffic wasted; deleting
--expert-prefetch: 26.8 -> 40.57/37.44/37.57 tok/s. Prefetch-on shape
survived every other challenger (gather geometry, spec depth, TunableOp
[corrupts numerics], MNBT, concurrency, ngram [PLE fork bug]). 256k ctx
boots at 4.5 GiB KV (36.65/34.21/34.77, arena -14%). New runbook +
index links + log entry.
This commit is contained in:
Frank
2026-09-29 21:22:21 -07:00
parent 67672dc794
commit 0ccb01708b
4 changed files with 316 additions and 0 deletions
+11
View File
@@ -158,6 +158,17 @@ The LLM inference fleet on the two hosts. Full index:
across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
assessment — reusable expert tiering, four architecture gaps, one a
silent-wrongness trap on the Engram key
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
36% precision, wasting ~half the gather traffic (the LRU itself runs at
82.45% hit); the prefetch-on incumbent 26.8 survived every other
challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
slab size)
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4