serverAppConfigs/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

---

type: research title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size). tags:

• dankserver
• r9700
• gfx1201
• qwen3.8
• fp8
• expert-offload
• speculative-decoding
• tuning timestamp: 2026-09-29

---

Qwen3.8-Flash-Next FP8 — expert-offload decode tuning

Question this page answers. On the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both R9700s, what runtime flags actually move decode at a 32k working point?

Answer in one line. 37.44–40.57 tok/s — the >35 target is met — by deleting --expert-prefetch 1. Expert-cache telemetry measured the prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served): ~half of all gather traffic was useless copies, and the copies it "saved" would have been installs anyway. The prefetch-on incumbent (26.8 tok/s) survived every other challenger measured: gather geometry, spec depth, TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.

This is the ad hoc qwen-fp8 container on Dankserver (rootless podman, image docker.io/tcclaviger/vllm:latest, engine v0.29.0.dev0+g2bdbbc8080), NOT the frank-primary systemd service — the R9V production unit is untouched, so no stop-and-mask was required for any of this. All numbers are decode tok/s measured from the server's own usage.completion_tokens / (wall - ttft) — SSE chunk counting is invalid under speculative decoding because rejected drafts change the accounting.

Deployment shape (the tuned incumbent)

Serve script on the host (/home/dank/serve_qwen_fp8.sh), env hooks in CAPS. PREFETCH must be unset (omitting the flag; the fork's default is off):

MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8   (~172.8 GiB, 131 shards)
IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
# the floating :latest tag moved once already — always launch by digest.
# repo tag of this state: qwen-fp8-tuned-20260929
--tensor-parallel-size 2  --max-model-len 32768  --gpu-memory-utilization 0.98
--kv-cache-memory-bytes 2147483648  --max-num-seqs 4  --enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic

Published to the host LAN via the serve script's PORTMAP hook (PORTMAP=7773:7770): health and completions answer on http://192.168.0.235:7773 (verified host-side 2026-09-29; 256k config, decode 36.49/34.55/35.98 through the published path). Host port 7773 was chosen over 7770 because frank-primary (7770) is stopped but ENABLED — any future start would collide with a published 7770.

Device notes carried from the bring-up: --device /dev/dri must be the WHOLE directory (individual renderD nodes enumerate but refuse contexts — "No CUDA GPUs are available" while device_count() reports 2); the image entrypoint execs vllm serve itself.

The memory plan this produces (from the engine's own expert_plan.py log):

region	size
expert cache space	15.05 GiB = 6,574 slots x 2.344 MiB
VRAM-only resident layers	3.52 GiB (layers 0-2, 1,536 experts)
swap region	11.53 GiB (5,038 slots, 111-112/layer)
host-backed experts	52.74 GiB per rank, 23,040 experts
KV reserve (one max-len request +8%)	0.36 GiB of the 2 GiB pool
CUDA graphs	1.93 GiB (21 graphs; 1.32 GiB at --max-num-seqs 2)
runtime overhead	7.6 GiB (non-torch + buffers + activation)

Measured arms (2026-09-29)

Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a 64-token warm pass, temperature 0, max_tokens 256:

Arm	166	1087	4064	Acceptance	Verdict
MTP4, stock 8x16, PREFETCH OFF	40.57	37.44	37.57	54.0%	NEW INCUMBENT — target met
MTP4, stock 8x16, prefetch ON	26.80	25.15	26.81	60.7%	old incumbent (+51%/+40% worse)
MTP4, prefetch ON (re-check)	25.80	24.38	26.63	47.9%	reproduces (noise band ±1)
MTP4, prefetch ON, gather 16x32	24.30	22.77	24.11	50.3%	−9% — ruled out
MTP4, prefetch ON, gather 32x64	26.58	24.94	26.34	49.6%	parity — ruled out
MTP3, prefetch ON	24.47	24.41	24.44	66.7%	−9%, flat in ctx — ruled out
MTP6, prefetch ON	19.07	17.81	13.54	17.4%	collapses; crosses UVA threshold — ruled out
MTP4, prefetch ON, MNBT 8192	29.32	24.12	23.65	—	neutral — ruled out
MTP4, prefetch ON, PREFETCH depth 2	25.83	25.06	25.87	56.0%	neutral — ruled out
MTP4, prefetch ON, TunableOp w/ offload	23.59	(79.82*)	(79.89*)	(96.5%*)	rejected — corrupts
2-way concurrency, prefetch ON (2 streams)	—	—	15.06 + 23.11 = 25.49 summed	—	no gain vs 26.8 single
ngram-only drafter	—	—	—	—	fork bug: PLE inputs unprepared
256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off)	36.65 / 36.48	34.21 / 34.13	34.77 / 35.02	53.0 / 39.4%	boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes

* The TunableOp numbers are invalid by construction: the tuned GEMMs changed generated behavior (repetition-loop, the server's "repetition detected" halt firing at ~140-146 tokens instead of the full 256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were measured on those halted degenerate completions, and the text-only check on a fresh prompt came back 20.01 tok/s — below baseline. GEMM autotuning that changes output text is broken tuning.

The decode curve without spec decode is dead flat 12.11 tok/s from 304 to 16,095 tokens — decode is context-independent on this shape, bound per-step by the expert gather, not by KV depth.

Telemetry — the measurement that found the win

R4D_LRU_TELEMETRY=1 arms per-layer counters; a torch-profiler window (/start_profile → traffic → /stop_profile) makes the fork log and write expert_cache_rank<N>.json beside the traces. torch_profiler_with_stack=false is mandatory in the profiler config — the default (true) grew a 6.5-min window's trace buffer ~40 GB and the kernel OOM-killed the engine core (anon-rss 95.2 GB, oom-kill in journalctl -k). The short window at 166 ctx (~15 s of traffic) completed cleanly with with_stack=false.

The fork's own numbers (102 steps, both ranks identical):

hit rate 0.8245 (137,084/166,262)   host installs 15,814
read-through 13,364                 prefetched 36,720
served by prefetch (migrations) 13,369

• The LRU is good: 82.45% hit rate overall; layers 0-2 pinned at 100%; swap-region layers 78-81%. The "arena holds only 27% of experts → 73% miss" arithmetic was wrong — temporal reuse carries it.
• The prefetcher is the poison: 36,720 slabs moved to serve 13,369 (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure waste — roughly half of all gather traffic — and the 13,369 "savings" would have been installs anyway. Depth-1 prefetch also cannot cover the ~2,400-slab per-step working set with its 16 slots/layer (768 total).
• Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs, 2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.

The wall, measured

• PCIe link (pcie-link-probe): GPUs 03:00.0/07:00.0 LnkCap/LnkSta 32GT/s x16; root ports 00:01.1/00:01.3 cap x8. Effective Gen5 x8 = 31.5 GB/s/direction. rocm-smi pcie_link_speed: 25 is the ASPM idle downshift, not the loaded rate.
• H2D (HIP probe, pinned): 2.345 MiB (the engine's exact expert slab) 22.74 GB/s; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈ pageable — staging is not the lever. Single-transfer latency at slab size 105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
• Engine's effective gather rate: ~6.5-9.5 GB/s = 30-42% of what the link delivers at the same size. The bottleneck is the fork's read path (chunk/lane geometry, per-transfer cost), not the wire.
• Duty cycle: peak GPU util 100%, smooth ~29% with prefetch on — each step is ~100 ms of compute then hundreds of ms waiting on expert streaming. rocm-smi reports utilization, which reads ~0 while weights load during an arm's reload; VRAM is the honest panel there.

What the remaining flags cannot do (all measured)

• Gather geometry — 16x32 and 32x64 measured no better despite the fork's "tuned at the Gen4 ceiling" comment; the constants are already at parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
• Spec depth — 4 is the optimum both ways (with prefetch on). Depth 6 additionally proves the UVA read-through path (step tokens > R4D_LRU_UVA_TOKENS=5) is slower than copy-in.
• Prefetch anything — depth 2 neutral, depth 1 the new negative; the tier is 3x too small for the working set and the predictor is 36% precise. expert_prefetch_k/budget/slots exist (defaults 20/14/16) but the precision result makes tuning them pointless.
• Arena growth via shrink — UTIL=0.99 fails boot (Free memory on device cuda:1 (31.29/31.86)); --max-num-seqs 2 fails CUDA-graph capture (Constraints violated (L['query_start_loc'].size()[0])); and the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral in both directions.
• Concurrency — under TP=2 both streams' tokens are computed on both cards, so each rank gathers the union of both streams' misses; the links do not add capacity and the shared arena thrashes. Summed 2-way (25.49) measured below single-stream (26.8) with prefetch on.
• ngram-only drafting — fails at engine init: RuntimeError: PLE inputs were not prepared; the fork prepares this model's PLE inputs only on the MTP spec path. Needs a fork patch, not a config.
• V-only q8 KV — stock --kv-cache-dtype offers fp8 variants only (K+V together); a V-only q8 needs a fork patch.

Fork bugs / traps found (for the engine authors or a future patch)

1. CLAV_TUNABLEOP_SWEEP=1 is silently skipped whenever expert offload is active unless CLAV_TUNABLEOP_WITH_OFFLOAD=1 is also set (tunableop_warmup.py).
2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent cache (container-local), and corrupt numerics (repetition loops, early-EOS halts, invalid measurements). Rejected outright.
3. The torch-profiler stop path hangs the workers on ROCm when torch_profiler_with_stack is true (default): >10 min hangs on two attempts while the AsyncLLM wrapper reports "stopped successfully" in 1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB and OOM-killed the engine core. With with_stack=false the stop completes and the telemetry report runs. The telemetry counters remain reachable ONLY through that profiler stop — no direct RPC trigger exists (telemetry_report has exactly one caller, gpu_worker.py).
4. ngram spec method does not prepare PLE inputs (above).

Where the headroom lives now

The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:

1. Fork patch to the gather read path — the ~6.5-9.5 → 22.7 GB/s software gap at slab size. If the gather reached wire rate, decode lands in the 60+ tok/s band before acceptance effects. Code work in expert_lru.py's gather (R4D_LRU_GATHER_* is read at import; the copy-in path itself is compiled — see /app/r4dhip/r4d.so).
2. 256k ctx — measured, final: boots at MAXLEN=262144 with KV_BYTES=4831838208 or 4328521728 alike — a 262,144-token window needs 4.0 GiB KV (hybrid attention + mamba pages ≈ 16 KiB/token, not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine estimates max len 184,224 at that pool; the exact-4.0 override loses ~20 MB to accounting and boots as 3.98 < 4.0). Decode band 34.1-36.5 across both KV sizes — the arena is identical (5,648 slots) at 4.5 and 4.03 GiB (the planner buckets; the trim went to margin, not slots), so the window costs ~10% decode vs the 32k config and no KV trim recovers it. The alternative that keeps the full 37.4-40.6 band is MAXLEN=184224 at 3 GiB KV. Correction: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains.
3. 2+ slots at long ctx — the KV pool post-spec holds ~66k tokens at 2 GiB (~2x 32k streams); concurrency summed below parity at 4064 ctx, so slots are a capacity feature, not a throughput one. Two full 256k streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting the arena below usefulness; slots-at-256k is a fork-patch or smaller-payload question.

Reproducing

Host-side scratch scripts (2026-09-29): /home/dank/serve_qwen_fp8.sh (the serve script with all hooks), /home/dank/mtp_measure.py (usage-counter decode measure, 3 depths), /home/dank/mtp_measure_short.py (single depth, for profile windows), /home/dank/bench2way.py (concurrency), plus the arm scripts arm_*.sh / sweep_*.sh and the detached runner /home/dank/job.sh (outputs under /home/dank/jobs/NAME.out|.rc|.done). The measure scripts are podman cp'd into the container after every recreate (the serve script re-copies them; a recreate wipes container-local files).

2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)

Two same-day cutovers moved this page's findings into production:

1. Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment (the GLM window is documented in frank-primary-glm53-flash-w4a16; git tag frank-primary-fp8-mtp-20261001 is the last commit of that shape). The production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off — measured 37.4-40.6 tok/s at ~54% acceptance.
2. The overnight redo campaign re-measured EVERY lever at matched geometry (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline) and concluded the day: the incumbent cutover to bf16 KV, no speculative decode (Frank's call).

What the redo campaign measured (all at shallow depth, 166/1087/4064)

lever	verdict
prefetch (re-confirmed)	the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed
gather geometry	R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7% (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak
KV dtype	bf16 (auto) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected)
spec family	no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed
MNBT	512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; locked 1024
UVA	2048 POISON (-39%); 8192 ~parity
arena	flat ±1.4% from ~6,830 down to ~5,500 slots; knee below
six slots	every SEQS-6 KV sizing refused at engine init

Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.

The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)

FRANK_PRIMARY_MAXLEN=200704   200704-token window
FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3166500000   PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048       --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982      budget = total x util, the law (1.0 unbootable, see below)
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12  -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120  --expert-offload-mem (host-region cap, explicit)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
NO --speculative-config, NO --expert-prefetch
FRANK_PRIMARY_NO_PREFIX_CACHING=1  --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)

2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault

Symptom (Frank, 2026-10-04): output turned repetitive/corrupted at ~40k tokens into a long conversation; short sessions stayed clean.

Evidence (live podman logs frank-primary-qwen, the 2026-10-04 18:19 boot): one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one running request, and the engine's own 18:46:08 log line reading "Prefix cache hit rate: 48.1%". 48 gdn_hip faults, then the process-lifetime silence cap:

[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
  with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
[gdn_hip] ERROR: state fault report limit (32) reached; further faults
  are counted silently

One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23 GDN layers each, the state non-finite (NaN). The guard's own text in this build: "known Qwen3.8-Flash-Next (QFN) flaw under prefix caching and is being actively worked; output quality for this request may degrade. Mitigation: --no-enable-prefix-caching."

Mechanism. With prefix caching on, the fork defaults QFN to mamba_cache_mode=align (boot log: "Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration … when prefix caching is enabled"; config/cache.py: "none" is the no-prefix-cache mode, "all" is deprecated — align is the only sound mode WITH the flag). Align mode checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a mamba page per block and RE-SEEDS the state from that page when a request resumes from a cached prefix. The block size is raised to 3136 tokens for the page-size equality (boot log: "Setting attention block size to 3136 tokens to ensure that attention page size is >= mamba page size" + "Padding mamba page size by 0.13% … exactly equal"). The corrupt state landed at the 13th boundary: 13 × 3136 = 40,768 tokens — the reported "~40k" mark. The NaN persisted in the mamba page, so every later turn of that conversation re-seeded the same non-finite state for those two heads (silently — the fault-report cap was already spent) while fresh short sessions looked fine.

RAM ruled out the same day (the 2026-10-04 17:17 host reboot is when the current memory speed took effect): zero MCE/EDAC kernel records since boot; an 8 GiB in-band pattern check stable over 60 s under live load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an overclock); and a two-head, single-slot NaN state is a state-path fault, not scattered finite bit corruption.

Fix. FRANK_PRIMARY_NO_PREFIX_CACHING=1 → --no-enable-prefix-caching (the fork-named mitigation; the flag renders in both directions from the knob so a fork default change cannot silently flip the contract). Cost: no prefix reuse — every turn re-prefills the whole history instead of at most 3135 tail tokens. A silently corrupted conversation is worse than a slower first token.

Verified 2026-10-05 (deploy b820d34):

• Engine argv carries --no-enable-prefix-caching; health 200; warm boot init engine … took 76.57 s (the mounted compile cache is intact).
• Boot plan unchanged, as predicted: expert cache 17.77 GiB / 7,762 slots (the flag touches no plan term), CUDA-graph estimate 0.07 GiB vs 0.08 GiB captured (inside the 12 MiB/graph charge).
• GPU KV cache size: 424,678 tokens — up from 401,408 at the same 3,166,500,000 B ask: the pool now prices ~7,456 B/token (was 7,891.8), the mamba/side per-token term dropping once the per-block checkpoint pages are gone; the 3136 attention block-size raise nonetheless stays (the page-size invariant is enforced in mode "none" too). Frank's 400,700-token floor is cleared with ~24k of headroom.
• Zero gdn_hip lines in the boot-and-traffic window (only the two INFO module banners mention the name).
• Boundary probe (single request, no prefix reuse): prompt 39,280 + completion 1,491 = 40,771 total tokens — across the 40,768 line; the task (write 1–400, one per line) completed coherently: 399 integers parsed, 0.997 monotone, tail … 398 399 400. Pre-fix, this boundary is exactly where the conversation degraded (the 13th 3136-page re-seed).
• e2e: 32/33 pass; the one failure is the admin spec's 192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental, unrelated to this change; re-run tests/e2e/run.sh once the mini is up).

RESULT — REVERTED THE SAME DAY. With every mechanical check green (zero faults, coherent 40,771-token probe), live long conversations STILL degraded at ~40k on the running endpoint (Frank, direct observation). The container serving that window (--no-enable-prefix- caching, booted 2026-10-05 12:47 PDT) logged ZERO gdn_hip faults while the corruption reproduced — so this recurrence is fault-free: the NaN-state guard did not fire, which the re-seed theory requires. Open hypotheses, in order: (1) the poisoned transcript is being re-fed — corrupted assistant turns already in the history keep producing corrupted continuations even with perfect state, so the same conversation cannot distinguish fixed from unfixed; (2) a second long-context mechanism with no guard coverage — fp8 KV degradation at depth is the next suspect (a bf16-KV boot costs ~2× the pool per token and would settle it). Prefix caching is back ON (FRANK_PRIMARY_NO_PREFIX_CACHING commented out — the cost bought no fix). The knob and the serve branch stay for a future fork build with a fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if it still breaks at ~40k with zero faults, A/B the KV dtype.

Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost

rocm-smi showed 3.57 GiB free per R9700 at the deployed shape E (28.28 GiB used of 31.86 GiB, after hours of real traffic — so it is the high-water mark, not an idle artifact). Growing the expert arena into it is NOT a matter of raising --gpu-memory-utilization, because the arena is

arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
budget = device_total x gpu_memory_utilization          # expert_plan.py:567

and every term on the right is either fixed or a lie:

term	plan charges	measured reality	reachable from our side?
budget	31.22 GiB (0.98)	field is le=1 so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below)	yes — shipped 0.982
non-expert weights	6.28 GiB	checkpoint census	no
KV reserve	1.91 GiB (registry 9,190 B/token x 200704 +8%)	real pool bills 2.93 GiB at 7,892 B/token	no (registry constant)
CUDA graphs	0.55 GiB = 256 MiB + 52 MiB x 6	Graph capturing finished in 3 secs, took 0.31 GiB	yes — VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB; VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the whole term outright (checked first, pair ignored when set)
runtime overhead	5.25 GiB (--expert-precise-memory)	far lower at idle	no (registry constant)

Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982: arena 17.52 GiB / 7,653 slots measured, +131 slots. Shape G takes the graph term further (0.32 -> 0.07 GiB) at the same util.

Shape G takes the graph term to the measured floor, util to the law. The graph base pool is the actual over-charge: shape F's own boot reported Graph capturing finished in 4 secs, took -0.06 GiB — a negative capture delta, so the 256 MiB base was fiction. Charging 0 base + 12 MiB/graph (0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 / arena 17.77 GiB / 7,762 slots measured. The KV ask went to 3,166,500,000 B/rank, which the engine's own 7,891.8 B/token price lands at 401,408 tokens measured / 2.00x concurrency — the first shape holding two full-length conversations with real spare. (The ask/landed delta is block alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)

UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured 2026-10-03. One deploy took the field maximum to buy the last 0.57 GiB of arena; both workers refused init BEFORE loading a single weight, twice: ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...) — the worker demands free > budget and budget == total, so le=1 on the CLI field is a trap: the engine dies in a 2-restart crash loop with zero payload progress. So 0.982 is a LAW, not a preference, and the "why 0.982" answer now has two proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real 995-token request). Everything beyond ~7,912 slots needs the registry edited inside the image (the fork's log line says exactly that: "Update the registry entry if this drifts"), not a deploy.

The host region was the last arena-plan term still autofitting per boot — now it is a literal (2026-10-04). With no --expert-offload-mem the planner binds the host region to MemAvailable AT BOOT minus PLE 47.68 - 2 x 7.0 process - 5.0 api/engine - 5% margin: the 2026-10-05 boot read 237.97 GiB and offered 159.39 GiB usable, while the same host reads 75.4 GiB available while running — the bound tracks whatever the co-tenants hold at that moment. The demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a boot where the bound drops near it silently converts next-hottest experts to VRAM-only (the pinned_extra warning is the only trace). FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 pins it: measured demand + 14% headroom, on a host with 246.1 GiB total. Semantics verified in the shipped image (offloader/expert_plan.py): the cap binds as the bound, an exceedance only warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).

MNBT 1024 -> 2048 is free on every axis the plan can see. The registry overhead model is base + 0.22 GiB x max(0, (nbt - 4096)/1024) anchored at nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25 was itself fitted at nbt 4096, which means 2048 is covered by an activation figure measured at a larger batch. SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02 — the query_start_loc ConstraintViolationError is a SEQS property). The costs are a one-time recompile (the compile range is part of the cache key, so FRANK_PRIMARY_CACHE_DIR pays once) and a wider prefill chunk, which the 09-29 sweep found decode-interchangeable across 1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).

--expert-precise-memory is the arena lever that does not touch the window. It attacks the planner's overhead term (shape F above attacks the graph term — the third and last term this repo can reach), and the flag swaps the default registry figure for the fitted one. Measured 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate booted AS the service; numbers are the engine's own plan lines):

plan line	shape D	shape E
runtime overhead (non-torch + buffers + activation)	7.60 GiB	5.25 GiB (precise (gfx1201 TP2 nbt4096/seqs16/len262144))
expert cache space	14.87 GiB	17.22 GiB
slots	6,496	7,522 (+1,026, +16%)
swap region	-	13.7 GiB / 5,986 slots, 133-134 per layer
boot	-	healthy in 420 s, no OOM

'expert_precise_memory': True also appears in non-default args, so a silent argparse drop is visible in the boot log. The flag lands in the arena, not the pool - --kv-cache-memory-bytes is untouched at 3.15e9/rank (option A), because the pool is not an arena lever and cutting it further would only cost concurrency: a 2.2e9 pool was probed in the same window and reports 277,686 tokens / 1.38x, so it is the conservative fallback if free VRAM ever binds, not the shipped value.

The compile cache is a boot-cost fix, and it was worth 173 s per boot. Measured 2026-10-02: the image ships no /root/.cache at all (a fresh container: No such file or directory), and after a boot that directory holds vllm at 1,154,714,333 B in 4,665 files plus comgr at 229 MB, all inside the container's ephemeral layer. Every podman rm -f - i.e. every deploy, restart and reboot - discards it, and the engine re-pays:

Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
Dynamo bytecode transform time: 4.99 s
Compiling a graph for compile range (1, 1024) takes 173.19 s
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
torch.compile took 181.20 s in total
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)

Mounting a host directory at /root/.cache makes every later boot load the AOT artifact instead. The mount is deliberately the parent, not /root/.cache/vllm: VLLM_CACHE_ROOT defaults to the vllm child, and comgr is the HIP-kernel JIT cache, so one mount persists both — and it hides nothing, because the image ships no /root/.cache. The cache key includes the compile range, so the MNBT choice costs exactly one compilation. Entries are version- and config-hashed: a stale entry is a miss, never a wrong answer.

Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this exact shape:

boot	engine log
cold (deploy)	Compiling a graph for compile range (1, 1024) takes 179.24 s → torch.compile took 188.26 s in total
warm (restart)	Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s → torch.compile took 3.97 s in total → init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)

The host directory held 1.4 GB in 4,707 files after the cold boot. Time to /health is dominated by the 172.8 GiB payload load (the warm boot reached health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a proportional cut in boot time.

GPU_MAX_HW_QUEUES=2 caps the HSA hardware queues per GPU. Verified a real knob rather than a no-op - the literal is read out of /opt/rocm/core-10.0/lib/libhsa-runtime64.so - and shipped deliberately alongside the 32x64 gather geometry. It is not A/B measured.

The pool, not the slot count, sets how many conversations live at once. Measured live at this shape: GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x (3,146,129,408 B reserved). The per-token price is shape-dependent: the 262144 shape priced at 7,701 B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer, so a concurrency figure never extrapolates across geometries. The endpoint advertises 3 slots: a third LONG request queues rather than being refused, and shorter requests use the spare blocks. Verified: three concurrent 67-token requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a capacity contract, not a throughput one (2-way concurrency summed BELOW parity at 4064 ctx).

--max-num-seqs 2 is impossible on this model (measured 2026-10-02)

Four single-variable boot probes against the then-deployed shape, production stopped for each:

MAXLEN	SEQS	MNBT	result
262144	2	1024	Engine core initialization failed
262144	2	2048	Engine core initialization failed
262144	3	1024	healthy - 792,069 tokens, 3.02x, arena 6,248
262144	3	2048	healthy - the then-deployed incumbent

The failure is torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant (3), raised from AOT build_guards(strict_error=True) during determine_available_memory -> profile_run -> _dummy_run. model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT is not a lever here - it only moves the compile RANGE ((1,1024) vs (1,2048)), which is why the identical error appears at both rungs. --enforce-eager bypasses the path at the cost of CUDA graphs. The same failure class killed the 2026-10-01 bf16 2 x 262144 shape.

The arena lever is max_model_len, not --kv-cache-memory-bytes

The fork's planner sizes expert cache as budget - non-expert - KV reserve - graphs - overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len - not from the --kv-cache-memory-bytes value:

MAXLEN	KV reserve	arena
262144	2.48 GiB	14.30 GiB / 6,248 slots
200704	1.91 GiB	14.87 GiB / 6,496 slots

D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 pool cut (2.75 GiB) bought zero arena slots. It buys runtime HEADROOM, which is still worth having - the shape it replaced died with 0 bytes free on a real 995-token request. Grow the arena by lowering max_model_len, or by taking the overhead term with --expert-precise-memory (below); the pool is the headroom lever only.

The other arena lever: --expert-precise-memory (overhead 7.60 -> 5.25 GiB)

The flag is now ON. It was refused on 2026-10-01 for a reason that does not generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB, NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also did not stay up: 0 bytes free at runtime, and a real client request (995-token prompt, max_tokens=261149) killed the worker with Tried to allocate 50.00 MiB ... 0 bytes free.

At 200704 x 3 the flag fits with room and is worth +1,026 slots (+16%). What made it affordable is the window trim, not a pool cut: the arena is DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704 window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.

Boot-probed 2026-10-02 (transient systemd units at the production uid + memlock, full-log capture; the rootless bench probe had masked the production memory environment). The night searched two concurrency contracts:

rung	result
3 x 262144, bf16, KV 11,047,010,304	OOM at graph capture — Tried to allocate 76.00 MiB ... 0 bytes free, 30.43 GiB of 31.86 GiB already allocated by PyTorch
3 x 245760, bf16	OOM identically
3 x 237568, bf16	PASS — KV 718,606 tokens
2 x 262144, bf16 + --expert-precise-memory	DEPLOYED then crashed — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag)
2 x 262144, bf16 (no precise flag)	did not stay up either: 0 bytes free at runtime, worker killed by a real request
3 x 200704, fp8, pool sized for 2 contexts	DEPLOYED 2026-10-02 (shape D): 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom
3 x 200704, fp8, + --expert-precise-memory	shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s
3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982	shape F (2026-10-03): arena 17.52 GiB / 7,653 slots measured, budget 31.29, Estimated CUDA graph memory: 0.32 GiB, KV unchanged 398,658
3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV 3.1665e9	DID NOT BOOT (2026-10-03): ValueError: Free memory on device cuda:N (31.29/31.86 GiB) on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop
3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700	DEPLOYED 2026-10-03 (shape G): arena 17.77 GiB / 7,762 slots measured (+109 over F, +240 over E), KV 401,408 tokens / 2.00x concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> 0.89 GiB real free per card held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch

Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even at fp8.

The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE reservation, not a VRAM wall. Failures are clean and diagnostic: the engine reports torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of which Y GiB is free, and the fork's giant ERROR lines are split by journald — read them via journalctl -o json, not -o cat (and podman logs, which is only available while the container lives).

Rollback to the 2026-09-29 fp8+MTP4 shape: git tag frank-primary-fp8-mtp-20261001 (env trio + seam re-pinned there).

Parallelism options on this box (2026-10-01 survey)

Surveyed from the running image's own CLI surface. Verdicts are specific to dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model with only 2 KV heads.

option	flag	verdict on this box
TP	--tensor-parallel-size	DEPLOYED, =2. All-reduce dispatch shows R4D_AR then PYNCCL
DCP	--decode-context-parallel-size	REJECTED at TP=2 — see below
PP	--pipeline-parallel-size	supported; untried (stage handoff plus a TP-keyed expert plan)
DP	--data-parallel-size (+-rank, -start-rank)	impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card)
EP	--enable-expert-parallel + --all2all-backend (+--enable-eplb, --enable-dbo, --enable-elastic-ep)	supported (naive, allgather_reducescatter, deepep_high_throughput, deepep_low_latency, pplx, mori, nixl); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all
multi-node	--nnodes, --node-rank, --master-addr/port, --distributed-executor-backend	supported; the actual path to more cards

DCP is structurally unavailable here — do not re-test on 2 GPUs

--decode-context-parallel-size 2 fails VllmConfig validation at boot:

Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
greater than the model's total number of KV heads (2). Increase
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.

DCP shards the KV along the sequence dimension after KV heads have been spread across the TP ranks, so it requires TP > num_kv_heads. This model has 2 KV heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related fork facts: DCP reuses the TP ranks rather than multiplying them (it is not TP x DCP), and with PCP enabled the valid DCP sizes are {1, pcp, tp x pcp} (vllm/config/parallel.py). The KV memory a DCP win would have bought is real — each rank would hold half the sequence's KV — but it is unreachable without two more cards.

Measuring decode: the harness lives INSIDE the container

/home/dank/mtp_measure.py imports /bench_iter.py by absolute path, so it must run in the engine's namespace and the files must be copied in first:

podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
podman exec frank-primary-qwen python3 /mtp_measure.py

Without the copy the exec dies with can't open file '/mtp_measure.py': [Errno 2] No such file or directory — measured 2026-10-01, which silently cost a whole arm's numbers. A window script must do the copy, not assume the path exists.

The then-deployed shape, measured again (2026-10-01 A/B window)

A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT 2048 in that window) reported GPU KV cache size: 1,079,582 tokens, Maximum concurrency for 262,144 tokens per request: 4.12x — the same figures as the deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is reproducible across boots.

See Also

• dankserver-frontier-moe-feasibility — the sibling feasibility page (which frontier MoE fits this box; the same PCIe-expert-stream wall at V4.1 scale)
• frank-primary-r9v-radiance-qwen38-q8 — RETIRED 2026-09-30 (the R9V fork era; production has been the tcclaviger fork since)
• x4-pair-experiment — the same residency law measured on a narrower link: streaming over PCIe loses, computing in RAM does not care
• Dual-R9700-Cooperation — the board swap that made TP=2 legal here