diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md new file mode 100644 index 0000000..b529762 --- /dev/null +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -0,0 +1,244 @@ +--- +type: research +title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning +description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size). +tags: + - dankserver + - r9700 + - gfx1201 + - qwen3.8 + - fp8 + - expert-offload + - speculative-decoding + - tuning +timestamp: 2026-09-29 +--- + +# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning + +> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving +> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what +> runtime flags actually move decode at a 32k working point? +> +> **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by +> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the +> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served): +> ~half of all gather traffic was useless copies, and the copies it "saved" +> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s) +> survived every other challenger measured: gather geometry, spec depth, +> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting. + +This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman, +image `docker.io/tcclaviger/vllm:latest`, engine +`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service — +the R9V production unit is untouched, so no stop-and-mask was required for +any of this. All numbers are decode tok/s measured from the server's own +`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid +under speculative decoding because rejected drafts change the accounting. + +## Deployment shape (the tuned incumbent) + +Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in +CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is +off): + +``` +MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) +IMAGE=docker.io/tcclaviger/vllm:latest +--tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98 +--kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching +--speculative-config '{"method":"mtp","num_speculative_tokens":4}' +# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic +``` + +Device notes carried from the bring-up: `--device /dev/dri` must be the +WHOLE directory (individual renderD nodes enumerate but refuse contexts — +"No CUDA GPUs are available" while device_count() reports 2); the image +entrypoint execs `vllm serve` itself. + +The memory plan this produces (from the engine's own `expert_plan.py` log): + +| region | size | +|---|---:| +| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB | +|   VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) | +|   swap region | 11.53 GiB (5,038 slots, 111-112/layer) | +| host-backed experts | 52.74 GiB per rank, 23,040 experts | +| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool | +| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) | +| runtime overhead | 7.6 GiB (non-torch + buffers + activation) | + +## Measured arms (2026-09-29) + +Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a +64-token warm pass, temperature 0, max_tokens 256: + +| Arm | 166 | 1087 | 4064 | Acceptance | Verdict | +|---|---:|---:|---:|---:|---| +| **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** | +| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) | +| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) | +| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out | +| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out | +| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out | +| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out | +| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out | +| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out | +| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** | +| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single | +| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared | +| 256k ctx (MAXLEN=262144, KV 4.5 GiB, prefetch off) | 36.65 | 34.21 | 34.77 | 53.0% | boots; arena 6574→5648 (−14%) costs ~10%; 2 of 3 depths just under 35 | + +\* The TunableOp numbers are invalid by construction: the tuned GEMMs +changed generated behavior (repetition-loop, the server's +"repetition detected" halt firing at ~140-146 tokens instead of the full +256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were +measured on those halted degenerate completions, and the text-only check on +a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning +that changes output text is broken tuning. + +The decode curve without spec decode is dead flat 12.11 tok/s from 304 to +16,095 tokens — decode is context-independent on this shape, bound per-step +by the expert gather, not by KV depth. + +## Telemetry — the measurement that found the win + +`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window +(`/start_profile` → traffic → `/stop_profile`) makes the fork log and write +`expert_cache_rank.json` beside the traces. **`torch_profiler_with_stack=false` +is mandatory in the profiler config** — the default (true) grew a 6.5-min +window's trace buffer ~40 GB and the kernel OOM-killed the engine core +(anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at +166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`. + +The fork's own numbers (102 steps, both ranks identical): + +``` +hit rate 0.8245 (137,084/166,262) host installs 15,814 +read-through 13,364 prefetched 36,720 +served by prefetch (migrations) 13,369 +``` + +- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at + 100%; swap-region layers 78-81%. The "arena holds only 27% of experts + → 73% miss" arithmetic was wrong — temporal reuse carries it. +- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369 + (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure + waste — roughly half of all gather traffic — and the 13,369 "savings" + would have been installs anyway. Depth-1 prefetch also cannot cover the + ~2,400-slab per-step working set with its 16 slots/layer (768 total). +- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs, + 2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful. + +## The wall, measured + +- **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta + 32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 = + 31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle + downshift, not the loaded rate. +- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab) + **22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈ + pageable — staging is not the lever. Single-transfer latency at slab size + 105.7 µs vs 78 µs ideal → ~21% per-transfer overhead. +- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the + link delivers at the same size**. The bottleneck is the fork's read path + (chunk/lane geometry, per-transfer cost), not the wire. +- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each + step is ~100 ms of compute then hundreds of ms waiting on expert + streaming. rocm-smi reports *utilization*, which reads ~0 while weights + load during an arm's reload; VRAM is the honest panel there. + +## What the remaining flags cannot do (all measured) + +- **Gather geometry** — 16x32 and 32x64 measured no better despite the + fork's "tuned at the Gen4 ceiling" comment; the constants are already at + parity on Gen5 x8. The read-path gap is structural, not a tuning knob. +- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6 + additionally proves the UVA read-through path (`step tokens > + R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in. +- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the + tier is 3x too small for the working set and the predictor is 36% + precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but + the precision result makes tuning them pointless. +- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on + device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph + capture (`Constraints violated (L['query_start_loc'].size()[0])`); and + the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral + in both directions. +- **Concurrency** — under TP=2 both streams' tokens are computed on both + cards, so each rank gathers the union of both streams' misses; the links + do not add capacity and the shared arena thrashes. Summed 2-way (25.49) + measured *below* single-stream (26.8) with prefetch on. +- **ngram-only drafting** — fails at engine init: + `RuntimeError: PLE inputs were not prepared`; the fork prepares this + model's PLE inputs only on the MTP spec path. Needs a fork patch, not a + config. +- **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only + (K+V together); a V-only q8 needs a fork patch. + +## Fork bugs / traps found (for the engine authors or a future patch) + +1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is + active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set + (`tunableop_warmup.py`). +2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent + cache (container-local), and corrupt numerics (repetition loops, + early-EOS halts, invalid measurements). Rejected outright. +3. The torch-profiler stop path hangs the workers on ROCm when + `torch_profiler_with_stack` is true (default): >10 min hangs on two + attempts while the AsyncLLM wrapper reports "stopped successfully" in + 1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB + and OOM-killed the engine core. With `with_stack=false` the stop + completes and the telemetry report runs. The telemetry counters remain + reachable ONLY through that profiler stop — no direct RPC trigger + exists (`telemetry_report` has exactly one caller, `gpu_worker.py`). +4. ngram spec method does not prepare PLE inputs (above). + +## Where the headroom lives now + +The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left: + +1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s + software gap at slab size. If the gather reached wire rate, decode + lands in the 60+ tok/s band before acceptance effects. Code work in + `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the + copy-in path itself is compiled — see `/app/r4dhip/r4d.so`). +2. **256k ctx — measured**: boots at `MAXLEN=262144 KV_BYTES=4831838208` + (a 262,144-token window needs **4.0 GiB** KV — hybrid attention + + mamba pages cost ~16 KiB/token, not the 32k plan's 9,190 B/token; + at 3 GiB the engine estimates max len 184,224). Decode + 36.65/34.21/34.77 — the arena pays 6,574→5,648 slots (−14%) and ~10% + decode; 2 of 3 depths sit just under 35. The KV-exact 4.0 GiB override + (0.5 GiB back to the arena, ~+3.5% slots) is the remaining nudge; + the alternative is `MAXLEN=184224` at 3 GiB KV, which keeps the full + 37.4-40.6 band. Correction: the planner sizes the arena AFTER the KV + override (the earlier "before" reading was wrong) — the arena is what + remains. +3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at + 2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx, + so slots are a capacity feature, not a throughput one. Two full 256k + streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting + the arena below usefulness; slots-at-256k is a fork-patch or + smaller-payload question. + +## Reproducing + +Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the +serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter +decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth, +for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm +scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh` +(outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts +are `podman cp`'d into the container after every recreate (the serve script +re-copies them; a recreate wipes container-local files). + +## See Also + +- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page + (which frontier MoE fits this box; the same PCIe-expert-stream wall at + V4.1 scale) +- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary + (production is the R9V fork; this page's container is the ad hoc detour) +- [[x4-pair-experiment]] — the same residency law measured on a narrower + link: streaming over PCIe loses, computing in RAM does not care +- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 23a1f6e..ab80908 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -101,6 +101,16 @@ frank-small, port 7771] bound, ~23 tok/s ceiling at today's residency). Includes the Affinity V4-Flash-to-V4.1 port assessment — reusable expert tiering, four architecture gaps, one of them a silent-wrongness trap on the Engram key +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29: + the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container + (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, + TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with + `--expert-prefetch` DELETED** — telemetry measured the prefetcher at + 36% precision, wasting ~half the gather traffic (the LRU itself runs at + 82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger + (geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency, + ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather + read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size) - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by [[frank-primary-lucebox-ds4]], then [[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3 diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index d0e35c5..94c1eda 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -158,6 +158,17 @@ The LLM inference fleet on the two hosts. Full index: across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port assessment — reusable expert tiering, four architecture gaps, one a silent-wrongness trap on the Engram key +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29: + the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container + (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, + TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with + `--expert-prefetch` DELETED** — telemetry measured the prefetcher at + 36% precision, wasting ~half the gather traffic (the LRU itself runs at + 82.45% hit); the prefetch-on incumbent 26.8 survived every other + challenger (geometry, spec depth, TunableOp-corrupts, batch knobs, + concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the + fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same + slab size) - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 349caf6..6726939 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -45,6 +45,57 @@ `=` keys the dict `ttft;dur` and every lookup silently returns nothing. Four configurations reported "no Server-Timing" while the header was present on all of them. +- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s, + mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4 + fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full + flag-level A/B battery at the 32k working point; every number is from the + server's usage counters (SSE chunks are invalid under spec decode). New + runbook: [[dankserver-qwen38-fp8-expert-offload-tuning]]. + - Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1, + TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx, + acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63). + - **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64 + (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the + stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in + ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves + the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral); + prefetch depth 2 (neutral — the tier is 16 slots/layer against a + ~2,400-slab working set at 73% misses); 2-way concurrency (summed + 25.49 < single 26.8 — under TP both streams' misses gather on each + rank, the links do not add capacity, and the shared arena thrashes). + - **TunableOp rejected:** it silently skips under expert offload unless + `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics — + repetition loops, the server's repetition-halt firing at ~140 of 256 + tokens, "79.8 tok/s" measured on halted degenerate completions, and a + fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no + persistent cache in the container lifecycle. + - **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free + at worker init); `--max-num-seqs 2` (compile constraint violation at + CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not + prepared" — PLE prep is wired only for the MTP spec path). + - **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction; + the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91% + of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s = + 42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait + (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the + fork's gather read path — fork-patch territory, not flags. + - **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm + (>10 min, two attempts, while the AsyncLLM wrapper logs success in + 1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min + profile window's trace buffer ~40 GB → the kernel OOM-killed the engine + core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer + expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY + through that stop path, so the telemetry JSON is unreachable without a + fork patch. + - **256k ctx boots** (the last original-goal dimension): a 262,144-token + window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈ + 16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused + (`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate + at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77** + (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just + under 35). KV-exact-4.0 arm in flight for the last +3.5% slots. + Correction recorded: the planner sizes the arena AFTER the KV override + (the earlier "before" reading was wrong) — the arena is what remains. ## 2026-09-28 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to