@@ -0,0 +1,244 @@
---
type : research
title : Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning
description : VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700) : the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size).
tags :
- dankserver
- r9700
- gfx1201
- qwen3.8
- fp8
- expert-offload
- speculative-decoding
- tuning
timestamp : 2026-09-29
---
# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning
> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving
> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what
> runtime flags actually move decode at a 32k working point?
>
> **Answer in one line.** **37.44–
> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the
> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served):
> ~half of all gather traffic was useless copies, and the copies it "saved"
> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s)
> survived every other challenger measured: gather geometry, spec depth,
> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.
This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
image `docker.io/tcclaviger/vllm:latest` , engine
`v0.29.0.dev0+g2bdbbc8080` ), NOT the `frank-primary` systemd service —
the R9V production unit is untouched, so no stop-and-mask was required for
any of this. All numbers are decode tok/s measured from the server's own
`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
under speculative decoding because rejected drafts change the accounting.
## Deployment shape (the tuned incumbent)
Serve script on the host (`/home/dank/serve_qwen_fp8.sh` ), env hooks in
CAPS. **PREFETCH must be unset ** (omitting the flag; the fork's default is
off):
```
MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards)
IMAGE=docker.io/tcclaviger/vllm:latest
--tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98
--kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
```
Device notes carried from the bring-up: `--device /dev/dri` must be the
WHOLE directory (individual renderD nodes enumerate but refuse contexts —
"No CUDA GPUs are available" while device_count() reports 2); the image
entrypoint execs `vllm serve` itself.
The memory plan this produces (from the engine's own `expert_plan.py` log):
| region | size |
|---|---:|
| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB |
| VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) |
| swap region | 11.53 GiB (5,038 slots, 111-112/layer) |
| host-backed experts | 52.74 GiB per rank, 23,040 experts |
| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool |
| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2` ) |
| runtime overhead | 7.6 GiB (non-torch + buffers + activation) |
## Measured arms (2026-09-29)
Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
64-token warm pass, temperature 0, max_tokens 256:
| Arm | 166 | 1087 | 4064 | Acceptance | Verdict |
|---|---:|---:|---:|---:|---|
| **MTP4, stock 8x16, PREFETCH OFF ** | **40.57 ** | **37.44 ** | **37.57 ** | 54.0% | **NEW INCUMBENT — target met ** |
| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) |
| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) |
| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −
| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out |
| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −
| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out |
| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out |
| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out |
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts ** |
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed ** | — | no gain vs 26.8 single |
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−
\* The TunableOp numbers are invalid by construction: the tuned GEMMs
changed generated behavior (repetition-loop, the server's
"repetition detected" halt firing at ~140-146 tokens instead of the full
256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were
measured on those halted degenerate completions, and the text-only check on
a fresh prompt came back 20.01 tok/s — * below * baseline. GEMM autotuning
that changes output text is broken tuning.
The decode curve without spec decode is dead flat 12.11 tok/s from 304 to
16,095 tokens — decode is context-independent on this shape, bound per-step
by the expert gather, not by KV depth.
## Telemetry — the measurement that found the win
`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window
(`/start_profile` → traffic → `/stop_profile` ) makes the fork log and write
`expert_cache_rank<N>.json` beside the traces. * * `torch_profiler_with_stack=false`
is mandatory in the profiler config** — the default (true) grew a 6.5-min
window's trace buffer ~40 GB and the kernel OOM-killed the engine core
(anon-rss 95.2 GB, `oom-kill` in `journalctl -k` ). The short window at
166 ctx (~15 s of traffic) completed cleanly with `with_stack=false` .
The fork's own numbers (102 steps, both ranks identical):
```
hit rate 0.8245 (137,084/166,262) host installs 15,814
read-through 13,364 prefetched 36,720
served by prefetch (migrations) 13,369
```
- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at
100%; swap-region layers 78-81%. The "arena holds only 27% of experts
→ 73% miss" arithmetic was wrong — temporal reuse carries it.
- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369
(36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure
waste — roughly half of all gather traffic — and the 13,369 "savings"
would have been installs anyway. Depth-1 prefetch also cannot cover the
~2,400-slab per-step working set with its 16 slots/layer (768 total).
- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs,
2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.
## The wall, measured
- **PCIe link (pcie-link-probe):** GPUs `03:00.0` /`07:00.0` LnkCap/LnkSta
32GT/s x16; root ports `00:01.1` /`00:01.3` cap x8. Effective **Gen5 x8 =
31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle
downshift, not the loaded rate.
- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab)
**22.74 GB/s ** ; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈
pageable — staging is not the lever. Single-transfer latency at slab size
105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the
link delivers at the same size**. The bottleneck is the fork's read path
(chunk/lane geometry, per-transfer cost), not the wire.
- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each
step is ~100 ms of compute then hundreds of ms waiting on expert
streaming. rocm-smi reports * utilization * , which reads ~0 while weights
load during an arm's reload; VRAM is the honest panel there.
## What the remaining flags cannot do (all measured)
- **Gather geometry** — 16x32 and 32x64 measured no better despite the
fork's "tuned at the Gen4 ceiling" comment; the constants are already at
parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6
additionally proves the UVA read-through path (`step tokens >
R4D_LRU_UVA_TOKENS=5` ) is * slower * than copy-in.
- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the
tier is 3x too small for the working set and the predictor is 36%
precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but
the precision result makes tuning them pointless.
- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on
device cuda:1 (31.29/31.86)`); ` --max-num-seqs 2` fails CUDA-graph
capture (` Constraints violated (L['query_start_loc'].size()[0])`); and
the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral
in both directions.
- **Concurrency** — under TP=2 both streams' tokens are computed on both
cards, so each rank gathers the union of both streams' misses; the links
do not add capacity and the shared arena thrashes. Summed 2-way (25.49)
measured *below* single-stream (26.8) with prefetch on.
- **ngram-only drafting** — fails at engine init:
` RuntimeError: PLE inputs were not prepared`; the fork prepares this
model's PLE inputs only on the MTP spec path. Needs a fork patch, not a
config.
- **V-only q8 KV** — stock ` --kv-cache-dtype` offers fp8 variants only
(K+V together); a V-only q8 needs a fork patch.
## Fork bugs / traps found (for the engine authors or a future patch)
1. ` CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is
active unless ` CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set
(` tunableop_warmup.py`).
2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent
cache (container-local), and corrupt numerics (repetition loops,
early-EOS halts, invalid measurements). Rejected outright.
3. The torch-profiler stop path hangs the workers on ROCm when
` torch_profiler_with_stack` is true (default): >10 min hangs on two
attempts while the AsyncLLM wrapper reports "stopped successfully" in
1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB
and OOM-killed the engine core. With ` with_stack=false` the stop
completes and the telemetry report runs. The telemetry counters remain
reachable ONLY through that profiler stop — no direct RPC trigger
exists (` telemetry_report` has exactly one caller, ` gpu_worker.py`).
4. ngram spec method does not prepare PLE inputs (above).
## Where the headroom lives now
The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s
software gap at slab size. If the gather reached wire rate, decode
lands in the 60+ tok/s band before acceptance effects. Code work in
` expert_lru.py`'s gather (` R4D_LRU_GATHER_*` is read at import; the
copy-in path itself is compiled — see ` /app/r4dhip/r4d.so`).
2. **256k ctx — measured**: boots at ` MAXLEN=262144 KV_BYTES=4831838208`
(a 262,144-token window needs **4.0 GiB** KV — hybrid attention +
mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token;
at 3 GiB the engine estimates max len 184,224). Decode
36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−
decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override
(0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge;
the alternative is ` MAXLEN=184224` at 3 GiB KV, which keeps the full
37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV
override (the earlier "before" reading was wrong) — the arena is what
remains.
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
so slots are a capacity feature, not a throughput one. Two full 256k
streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting
the arena below usefulness; slots-at-256k is a fork-patch or
smaller-payload question.
## Reproducing
Host-side scratch scripts (2026-09-29): ` /home/dank/serve_qwen_fp8.sh` (the
serve script with all hooks), ` /home/dank/mtp_measure.py` (usage-counter
decode measure, 3 depths), ` /home/dank/mtp_measure_short.py` (single depth,
for profile windows), ` /home/dank/bench2way.py` (concurrency), plus the arm
scripts ` arm_*.sh` / ` sweep_*.sh` and the detached runner ` /home/dank/job.sh`
(outputs under ` /home/dank/jobs/NAME.out|.rc|.done`). The measure scripts
are ` podman cp`'d into the container after every recreate (the serve script
re-copies them; a recreate wipes container-local files).
## See Also
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
V4.1 scale)
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
(production is the R9V fork; this page's container is the ad hoc detour)
- [[x4-pair-experiment]] — the same residency law measured on a narrower
link: streaming over PCIe loses, computing in RAM does not care
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
Frank