--- type: research title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size). tags: - dankserver - r9700 - gfx1201 - qwen3.8 - fp8 - expert-offload - speculative-decoding - tuning timestamp: 2026-09-29 --- # Qwen3.8-Flash-Next FP8 — expert-offload decode tuning > **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving > `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what > runtime flags actually move decode at a 32k working point? > > **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by > deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the > prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served): > ~half of all gather traffic was useless copies, and the copies it "saved" > would have been installs anyway. The prefetch-on incumbent (26.8 tok/s) > survived every other challenger measured: gather geometry, spec depth, > TunableOp, scheduler batch knobs, concurrency, ngram-only drafting. This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman, image `docker.io/tcclaviger/vllm:latest`, engine `v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service — the R9V production unit is untouched, so no stop-and-mask was required for any of this. All numbers are decode tok/s measured from the server's own `usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid under speculative decoding because rejected drafts change the accounting. ## Deployment shape (the tuned incumbent) Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is off): ``` MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea # image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929; # the floating :latest tag moved once already — always launch by digest. # repo tag of this state: qwen-fp8-tuned-20260929 --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98 --kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":4}' # NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic ``` Published to the host LAN via the serve script's `PORTMAP` hook (`PORTMAP=7773:7770`): health and completions answer on `http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config, decode 36.49/34.55/35.98 through the published path). Host port 7773 was chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED — any future start would collide with a published 7770. Device notes carried from the bring-up: `--device /dev/dri` must be the WHOLE directory (individual renderD nodes enumerate but refuse contexts — "No CUDA GPUs are available" while device_count() reports 2); the image entrypoint execs `vllm serve` itself. The memory plan this produces (from the engine's own `expert_plan.py` log): | region | size | |---|---:| | expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB | |   VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) | |   swap region | 11.53 GiB (5,038 slots, 111-112/layer) | | host-backed experts | 52.74 GiB per rank, 23,040 experts | | KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool | | CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) | | runtime overhead | 7.6 GiB (non-torch + buffers + activation) | ## Measured arms (2026-09-29) Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a 64-token warm pass, temperature 0, max_tokens 256: | Arm | 166 | 1087 | 4064 | Acceptance | Verdict | |---|---:|---:|---:|---:|---| | **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** | | MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) | | MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) | | MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out | | MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out | | MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out | | MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out | | MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out | | MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out | | MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** | | 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single | | ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared | | 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes | \* The TunableOp numbers are invalid by construction: the tuned GEMMs changed generated behavior (repetition-loop, the server's "repetition detected" halt firing at ~140-146 tokens instead of the full 256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were measured on those halted degenerate completions, and the text-only check on a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning that changes output text is broken tuning. The decode curve without spec decode is dead flat 12.11 tok/s from 304 to 16,095 tokens — decode is context-independent on this shape, bound per-step by the expert gather, not by KV depth. ## Telemetry — the measurement that found the win `R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window (`/start_profile` → traffic → `/stop_profile`) makes the fork log and write `expert_cache_rank.json` beside the traces. **`torch_profiler_with_stack=false` is mandatory in the profiler config** — the default (true) grew a 6.5-min window's trace buffer ~40 GB and the kernel OOM-killed the engine core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at 166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`. The fork's own numbers (102 steps, both ranks identical): ``` hit rate 0.8245 (137,084/166,262) host installs 15,814 read-through 13,364 prefetched 36,720 served by prefetch (migrations) 13,369 ``` - **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at 100%; swap-region layers 78-81%. The "arena holds only 27% of experts → 73% miss" arithmetic was wrong — temporal reuse carries it. - **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369 (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure waste — roughly half of all gather traffic — and the 13,369 "savings" would have been installs anyway. Depth-1 prefetch also cannot cover the ~2,400-slab per-step working set with its 16 slots/layer (768 total). - Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs, 2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful. ## The wall, measured - **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta 32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 = 31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle downshift, not the loaded rate. - **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab) **22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈ pageable — staging is not the lever. Single-transfer latency at slab size 105.7 µs vs 78 µs ideal → ~21% per-transfer overhead. - **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the link delivers at the same size**. The bottleneck is the fork's read path (chunk/lane geometry, per-transfer cost), not the wire. - **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each step is ~100 ms of compute then hundreds of ms waiting on expert streaming. rocm-smi reports *utilization*, which reads ~0 while weights load during an arm's reload; VRAM is the honest panel there. ## What the remaining flags cannot do (all measured) - **Gather geometry** — 16x32 and 32x64 measured no better despite the fork's "tuned at the Gen4 ceiling" comment; the constants are already at parity on Gen5 x8. The read-path gap is structural, not a tuning knob. - **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6 additionally proves the UVA read-through path (`step tokens > R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in. - **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the tier is 3x too small for the working set and the predictor is 36% precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but the precision result makes tuning them pointless. - **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph capture (`Constraints violated (L['query_start_loc'].size()[0])`); and the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral in both directions. - **Concurrency** — under TP=2 both streams' tokens are computed on both cards, so each rank gathers the union of both streams' misses; the links do not add capacity and the shared arena thrashes. Summed 2-way (25.49) measured *below* single-stream (26.8) with prefetch on. - **ngram-only drafting** — fails at engine init: `RuntimeError: PLE inputs were not prepared`; the fork prepares this model's PLE inputs only on the MTP spec path. Needs a fork patch, not a config. - **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only (K+V together); a V-only q8 needs a fork patch. ## Fork bugs / traps found (for the engine authors or a future patch) 1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set (`tunableop_warmup.py`). 2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent cache (container-local), and corrupt numerics (repetition loops, early-EOS halts, invalid measurements). Rejected outright. 3. The torch-profiler stop path hangs the workers on ROCm when `torch_profiler_with_stack` is true (default): >10 min hangs on two attempts while the AsyncLLM wrapper reports "stopped successfully" in 1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB and OOM-killed the engine core. With `with_stack=false` the stop completes and the telemetry report runs. The telemetry counters remain reachable ONLY through that profiler stop — no direct RPC trigger exists (`telemetry_report` has exactly one caller, `gpu_worker.py`). 4. ngram spec method does not prepare PLE inputs (above). ## Where the headroom lives now The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left: 1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s software gap at slab size. If the gather reached wire rate, decode lands in the 60+ tok/s band before acceptance effects. Code work in `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the copy-in path itself is compiled — see `/app/r4dhip/r4d.so`). 2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token, not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine estimates max len 184,224 at that pool; the exact-4.0 override loses ~20 MB to accounting and boots as 3.98 < 4.0). Decode band **34.1-36.5** across both KV sizes — the arena is **identical (5,648 slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to margin, not slots), so the window costs ~10% decode vs the 32k config and no KV trim recovers it. The alternative that keeps the full 37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains. 3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at 2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx, so slots are a capacity feature, not a throughput one. Two full 256k streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting the arena below usefulness; slots-at-256k is a fork-patch or smaller-payload question. ## Reproducing Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth, for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh` (outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts are `podman cp`'d into the container after every recreate (the serve script re-copies them; a recreate wipes container-local files). ## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec) Two same-day cutovers moved this page's findings into production: 1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off — measured 37.4-40.6 tok/s at ~54% acceptance. 2. **The overnight redo campaign re-measured EVERY lever at matched geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline) and concluded the day: the incumbent cutover to **bf16 KV, no speculative decode** (Frank's call). ### What the redo campaign measured (all at shallow depth, 166/1087/4064) | lever | verdict | |---|---| | prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed | | gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak | | KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) | | spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed | | MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** | | UVA | 2048 POISON (-39%); 8192 ~parity | | arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below | | six slots | every SEQS-6 KV sizing refused at engine init | Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. ### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec) ``` FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract) FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token) FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized) FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead 5.25 GiB instead of the 7.60 GiB default, so ~1,025 more arena slots) FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) NO --speculative-config, NO --expert-prefetch ``` Boot-probed 2026-10-01 (transient systemd units at the production uid + memlock, full-log capture; the rootless bench probe had masked the production memory environment). The night searched two concurrency contracts: | rung | result | |---|---| | 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch | | 3 × 245760, bf16 | **OOM identically** | | 3 × 237568, bf16 | PASS — KV 718,606 tokens | | **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) | | 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit | | 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x | | 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x | | 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured | | **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots | Boot is NOT proof of runtime survivability: the five-slot shape passed every boot gate and still died on the first long-generation request. After any knife-edge shape change, send a request with a large `max_tokens` before calling it done. The deployed shape is **four contexts of the model's full native 262,144 window**. fp8 KV is what buys the capacity — 7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 % decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even at fp8. The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE reservation, not a VRAM wall. Failures are clean and diagnostic: the engine reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of which Y GiB is free`, and the fork's giant ERROR lines are split by journald — read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is only available while the container lives). Rollback to the 2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there). ## Parallelism options on this box (2026-10-01 survey) Surveyed from the running image's own CLI surface. Verdicts are specific to dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model with only 2 KV heads. | option | flag | verdict on this box | |---|---|---| | TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` | | DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below | | PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) | | DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) | | EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all | | multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards | ### DCP is structurally unavailable here — do not re-test on 2 GPUs `--decode-context-parallel-size 2` fails `VllmConfig` validation at boot: ``` Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be greater than the model's total number of KV heads (2). Increase `--tensor-parallel-size` or set `--decode-context-parallel-size 1`. ``` DCP shards the KV along the *sequence* dimension after KV heads have been spread across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not `TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}` (`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real — each rank would hold half the sequence's KV — but it is unreachable without two more cards. ### Measuring decode: the harness lives INSIDE the container `/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must run in the engine's namespace and the files must be copied in first: ``` podman cp /home/dank/mtp_measure.py frank-primary-qwen:/ podman cp /home/dank/bench_iter.py frank-primary-qwen:/ podman exec frank-primary-qwen python3 /mtp_measure.py ``` Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2] No such file or directory` — measured 2026-10-01, which silently cost a whole arm's numbers. A window script must do the copy, not assume the path exists. ### The deployed shape, measured again (2026-10-01 A/B window) A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT 2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is reproducible across boots. ## See Also - [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page (which frontier MoE fits this box; the same PCIe-expert-stream wall at V4.1 scale) - [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V fork era; production has been the tcclaviger fork since) - [[x4-pair-experiment]] — the same residency law measured on a narrower link: streaming over PCIe loses, computing in RAM does not care - [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here