docs: qwen38-fp8 expert-offload tuning — prefetch off hits 37.4-40.6 tok/s (target met)
Measured campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4 fork, 2x R9700, TP=2): telemetry (with_stack=false) measured the prefetcher at 36% precision — ~half of gather traffic wasted; deleting --expert-prefetch: 26.8 -> 40.57/37.44/37.57 tok/s. Prefetch-on shape survived every other challenger (gather geometry, spec depth, TunableOp [corrupts numerics], MNBT, concurrency, ngram [PLE fork bug]). 256k ctx boots at 4.5 GiB KV (36.65/34.21/34.77, arena -14%). New runbook + index links + log entry.
This commit is contained in:
@@ -158,6 +158,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
|
||||
assessment — reusable expert tiering, four architecture gaps, one a
|
||||
silent-wrongness trap on the Engram key
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
|
||||
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
|
||||
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
|
||||
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
|
||||
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
|
||||
36% precision, wasting ~half the gather traffic (the LRU itself runs at
|
||||
82.45% hit); the prefetch-on incumbent 26.8 survived every other
|
||||
challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
|
||||
concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
|
||||
fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
|
||||
slab size)
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
|
||||
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
|
||||
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
|
||||
|
||||
Reference in New Issue
Block a user