serverAppConfigs/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

---

type: research title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size). tags:

• dankserver
• r9700
• gfx1201
• qwen3.8
• fp8
• expert-offload
• speculative-decoding
• tuning timestamp: 2026-09-29

---

Qwen3.8-Flash-Next FP8 — expert-offload decode tuning

Question this page answers. On the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both R9700s, what runtime flags actually move decode at a 32k working point?

Answer in one line. 37.44–40.57 tok/s — the >35 target is met — by deleting --expert-prefetch 1. Expert-cache telemetry measured the prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served): ~half of all gather traffic was useless copies, and the copies it "saved" would have been installs anyway. The prefetch-on incumbent (26.8 tok/s) survived every other challenger measured: gather geometry, spec depth, TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.

This is the ad hoc qwen-fp8 container on Dankserver (rootless podman, image docker.io/tcclaviger/vllm:latest, engine v0.29.0.dev0+g2bdbbc8080), NOT the frank-primary systemd service — the R9V production unit is untouched, so no stop-and-mask was required for any of this. All numbers are decode tok/s measured from the server's own usage.completion_tokens / (wall - ttft) — SSE chunk counting is invalid under speculative decoding because rejected drafts change the accounting.

Deployment shape (the tuned incumbent)

Serve script on the host (/home/dank/serve_qwen_fp8.sh), env hooks in CAPS. PREFETCH must be unset (omitting the flag; the fork's default is off):

MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8   (~172.8 GiB, 131 shards)
IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
# the floating :latest tag moved once already — always launch by digest.
# repo tag of this state: qwen-fp8-tuned-20260929
--tensor-parallel-size 2  --max-model-len 32768  --gpu-memory-utilization 0.98
--kv-cache-memory-bytes 2147483648  --max-num-seqs 4  --enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic

Published to the host LAN via the serve script's PORTMAP hook (PORTMAP=7773:7770): health and completions answer on http://192.168.0.235:7773 (verified host-side 2026-09-29; 256k config, decode 36.49/34.55/35.98 through the published path). Host port 7773 was chosen over 7770 because frank-primary (7770) is stopped but ENABLED — any future start would collide with a published 7770.

Device notes carried from the bring-up: --device /dev/dri must be the WHOLE directory (individual renderD nodes enumerate but refuse contexts — "No CUDA GPUs are available" while device_count() reports 2); the image entrypoint execs vllm serve itself.

The memory plan this produces (from the engine's own expert_plan.py log):

region	size
expert cache space	15.05 GiB = 6,574 slots x 2.344 MiB
VRAM-only resident layers	3.52 GiB (layers 0-2, 1,536 experts)
swap region	11.53 GiB (5,038 slots, 111-112/layer)
host-backed experts	52.74 GiB per rank, 23,040 experts
KV reserve (one max-len request +8%)	0.36 GiB of the 2 GiB pool
CUDA graphs	1.93 GiB (21 graphs; 1.32 GiB at --max-num-seqs 2)
runtime overhead	7.6 GiB (non-torch + buffers + activation)

Measured arms (2026-09-29)

Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a 64-token warm pass, temperature 0, max_tokens 256:

Arm	166	1087	4064	Acceptance	Verdict
MTP4, stock 8x16, PREFETCH OFF	40.57	37.44	37.57	54.0%	NEW INCUMBENT — target met
MTP4, stock 8x16, prefetch ON	26.80	25.15	26.81	60.7%	old incumbent (+51%/+40% worse)
MTP4, prefetch ON (re-check)	25.80	24.38	26.63	47.9%	reproduces (noise band ±1)
MTP4, prefetch ON, gather 16x32	24.30	22.77	24.11	50.3%	−9% — ruled out
MTP4, prefetch ON, gather 32x64	26.58	24.94	26.34	49.6%	parity — ruled out
MTP3, prefetch ON	24.47	24.41	24.44	66.7%	−9%, flat in ctx — ruled out
MTP6, prefetch ON	19.07	17.81	13.54	17.4%	collapses; crosses UVA threshold — ruled out
MTP4, prefetch ON, MNBT 8192	29.32	24.12	23.65	—	neutral — ruled out
MTP4, prefetch ON, PREFETCH depth 2	25.83	25.06	25.87	56.0%	neutral — ruled out
MTP4, prefetch ON, TunableOp w/ offload	23.59	(79.82*)	(79.89*)	(96.5%*)	rejected — corrupts
2-way concurrency, prefetch ON (2 streams)	—	—	15.06 + 23.11 = 25.49 summed	—	no gain vs 26.8 single
ngram-only drafter	—	—	—	—	fork bug: PLE inputs unprepared
256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off)	36.65 / 36.48	34.21 / 34.13	34.77 / 35.02	53.0 / 39.4%	boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes

* The TunableOp numbers are invalid by construction: the tuned GEMMs changed generated behavior (repetition-loop, the server's "repetition detected" halt firing at ~140-146 tokens instead of the full 256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were measured on those halted degenerate completions, and the text-only check on a fresh prompt came back 20.01 tok/s — below baseline. GEMM autotuning that changes output text is broken tuning.

The decode curve without spec decode is dead flat 12.11 tok/s from 304 to 16,095 tokens — decode is context-independent on this shape, bound per-step by the expert gather, not by KV depth.

Telemetry — the measurement that found the win

R4D_LRU_TELEMETRY=1 arms per-layer counters; a torch-profiler window (/start_profile → traffic → /stop_profile) makes the fork log and write expert_cache_rank<N>.json beside the traces. torch_profiler_with_stack=false is mandatory in the profiler config — the default (true) grew a 6.5-min window's trace buffer ~40 GB and the kernel OOM-killed the engine core (anon-rss 95.2 GB, oom-kill in journalctl -k). The short window at 166 ctx (~15 s of traffic) completed cleanly with with_stack=false.

The fork's own numbers (102 steps, both ranks identical):

hit rate 0.8245 (137,084/166,262)   host installs 15,814
read-through 13,364                 prefetched 36,720
served by prefetch (migrations) 13,369

• The LRU is good: 82.45% hit rate overall; layers 0-2 pinned at 100%; swap-region layers 78-81%. The "arena holds only 27% of experts → 73% miss" arithmetic was wrong — temporal reuse carries it.
• The prefetcher is the poison: 36,720 slabs moved to serve 13,369 (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure waste — roughly half of all gather traffic — and the 13,369 "savings" would have been installs anyway. Depth-1 prefetch also cannot cover the ~2,400-slab per-step working set with its 16 slots/layer (768 total).
• Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs, 2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.

The wall, measured

• PCIe link (pcie-link-probe): GPUs 03:00.0/07:00.0 LnkCap/LnkSta 32GT/s x16; root ports 00:01.1/00:01.3 cap x8. Effective Gen5 x8 = 31.5 GB/s/direction. rocm-smi pcie_link_speed: 25 is the ASPM idle downshift, not the loaded rate.
• H2D (HIP probe, pinned): 2.345 MiB (the engine's exact expert slab) 22.74 GB/s; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈ pageable — staging is not the lever. Single-transfer latency at slab size 105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
• Engine's effective gather rate: ~6.5-9.5 GB/s = 30-42% of what the link delivers at the same size. The bottleneck is the fork's read path (chunk/lane geometry, per-transfer cost), not the wire.
• Duty cycle: peak GPU util 100%, smooth ~29% with prefetch on — each step is ~100 ms of compute then hundreds of ms waiting on expert streaming. rocm-smi reports utilization, which reads ~0 while weights load during an arm's reload; VRAM is the honest panel there.

What the remaining flags cannot do (all measured)

• Gather geometry — 16x32 and 32x64 measured no better despite the fork's "tuned at the Gen4 ceiling" comment; the constants are already at parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
• Spec depth — 4 is the optimum both ways (with prefetch on). Depth 6 additionally proves the UVA read-through path (step tokens > R4D_LRU_UVA_TOKENS=5) is slower than copy-in.
• Prefetch anything — depth 2 neutral, depth 1 the new negative; the tier is 3x too small for the working set and the predictor is 36% precise. expert_prefetch_k/budget/slots exist (defaults 20/14/16) but the precision result makes tuning them pointless.
• Arena growth via shrink — UTIL=0.99 fails boot (Free memory on device cuda:1 (31.29/31.86)); --max-num-seqs 2 fails CUDA-graph capture (Constraints violated (L['query_start_loc'].size()[0])); and the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral in both directions.
• Concurrency — under TP=2 both streams' tokens are computed on both cards, so each rank gathers the union of both streams' misses; the links do not add capacity and the shared arena thrashes. Summed 2-way (25.49) measured below single-stream (26.8) with prefetch on.
• ngram-only drafting — fails at engine init: RuntimeError: PLE inputs were not prepared; the fork prepares this model's PLE inputs only on the MTP spec path. Needs a fork patch, not a config.
• V-only q8 KV — stock --kv-cache-dtype offers fp8 variants only (K+V together); a V-only q8 needs a fork patch.

Fork bugs / traps found (for the engine authors or a future patch)

1. CLAV_TUNABLEOP_SWEEP=1 is silently skipped whenever expert offload is active unless CLAV_TUNABLEOP_WITH_OFFLOAD=1 is also set (tunableop_warmup.py).
2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent cache (container-local), and corrupt numerics (repetition loops, early-EOS halts, invalid measurements). Rejected outright.
3. The torch-profiler stop path hangs the workers on ROCm when torch_profiler_with_stack is true (default): >10 min hangs on two attempts while the AsyncLLM wrapper reports "stopped successfully" in 1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB and OOM-killed the engine core. With with_stack=false the stop completes and the telemetry report runs. The telemetry counters remain reachable ONLY through that profiler stop — no direct RPC trigger exists (telemetry_report has exactly one caller, gpu_worker.py).
4. ngram spec method does not prepare PLE inputs (above).

Where the headroom lives now

The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:

1. Fork patch to the gather read path — the ~6.5-9.5 → 22.7 GB/s software gap at slab size. If the gather reached wire rate, decode lands in the 60+ tok/s band before acceptance effects. Code work in expert_lru.py's gather (R4D_LRU_GATHER_* is read at import; the copy-in path itself is compiled — see /app/r4dhip/r4d.so).
2. 256k ctx — measured, final: boots at MAXLEN=262144 with KV_BYTES=4831838208 or 4328521728 alike — a 262,144-token window needs 4.0 GiB KV (hybrid attention + mamba pages ≈ 16 KiB/token, not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine estimates max len 184,224 at that pool; the exact-4.0 override loses ~20 MB to accounting and boots as 3.98 < 4.0). Decode band 34.1-36.5 across both KV sizes — the arena is identical (5,648 slots) at 4.5 and 4.03 GiB (the planner buckets; the trim went to margin, not slots), so the window costs ~10% decode vs the 32k config and no KV trim recovers it. The alternative that keeps the full 37.4-40.6 band is MAXLEN=184224 at 3 GiB KV. Correction: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains.
3. 2+ slots at long ctx — the KV pool post-spec holds ~66k tokens at 2 GiB (~2x 32k streams); concurrency summed below parity at 4064 ctx, so slots are a capacity feature, not a throughput one. Two full 256k streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting the arena below usefulness; slots-at-256k is a fork-patch or smaller-payload question.

Reproducing

Host-side scratch scripts (2026-09-29): /home/dank/serve_qwen_fp8.sh (the serve script with all hooks), /home/dank/mtp_measure.py (usage-counter decode measure, 3 depths), /home/dank/mtp_measure_short.py (single depth, for profile windows), /home/dank/bench2way.py (concurrency), plus the arm scripts arm_*.sh / sweep_*.sh and the detached runner /home/dank/job.sh (outputs under /home/dank/jobs/NAME.out|.rc|.done). The measure scripts are podman cp'd into the container after every recreate (the serve script re-copies them; a recreate wipes container-local files).

2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)

Two same-day cutovers moved this page's findings into production:

1. Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment (the GLM window is documented in frank-primary-glm53-flash-w4a16; git tag frank-primary-fp8-mtp-20261001 is the last commit of that shape). The production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off — measured 37.4-40.6 tok/s at ~54% acceptance.
2. The overnight redo campaign re-measured EVERY lever at matched geometry (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline) and concluded the day: the incumbent cutover to bf16 KV, no speculative decode (Frank's call).

What the redo campaign measured (all at shallow depth, 166/1087/4064)

lever	verdict
prefetch (re-confirmed)	the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed
gather geometry	R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7% (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak
KV dtype	bf16 (auto) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected)
spec family	no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed
MNBT	512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; locked 1024
UVA	2048 POISON (-39%); 8192 ~parity
arena	flat ±1.4% from ~6,830 down to ~5,500 slots; knee below
six slots	every SEQS-6 KV sizing refused at engine init

Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.

The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)

FRANK_PRIMARY_MAXLEN=200704   200704-token window
FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
NO --speculative-config, NO --expert-prefetch

--expert-precise-memory is the arena lever that does not touch the window. The planner's overhead term is the only other place the arena can come from, and the flag swaps the default registry figure for the fitted one. Measured 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate booted AS the service; numbers are the engine's own plan lines):

plan line	shape D	shape E
runtime overhead (non-torch + buffers + activation)	7.60 GiB	5.25 GiB (precise (gfx1201 TP2 nbt4096/seqs16/len262144))
expert cache space	14.87 GiB	17.22 GiB
slots	6,496	7,522 (+1,026, +16%)
swap region	-	13.7 GiB / 5,986 slots, 133-134 per layer
boot	-	healthy in 420 s, no OOM

'expert_precise_memory': True also appears in non-default args, so a silent argparse drop is visible in the boot log. The flag lands in the arena, not the pool - --kv-cache-memory-bytes is untouched at 3.15e9/rank (option A), because the pool is not an arena lever and cutting it further would only cost concurrency: a 2.2e9 pool was probed in the same window and reports 277,686 tokens / 1.38x, so it is the conservative fallback if free VRAM ever binds, not the shipped value.

The compile cache is a boot-cost fix, and it was worth 173 s per boot. Measured 2026-10-02: the image ships no /root/.cache at all (a fresh container: No such file or directory), and after a boot that directory holds vllm at 1,154,714,333 B in 4,665 files plus comgr at 229 MB, all inside the container's ephemeral layer. Every podman rm -f - i.e. every deploy, restart and reboot - discards it, and the engine re-pays:

Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
Dynamo bytecode transform time: 4.99 s
Compiling a graph for compile range (1, 1024) takes 173.19 s
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
torch.compile took 181.20 s in total
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)

Mounting a host directory at /root/.cache makes every later boot load the AOT artifact instead. The mount is deliberately the parent, not /root/.cache/vllm: VLLM_CACHE_ROOT defaults to the vllm child, and comgr is the HIP-kernel JIT cache, so one mount persists both — and it hides nothing, because the image ships no /root/.cache. The cache key includes the compile range, so the MNBT choice costs exactly one compilation. Entries are version- and config-hashed: a stale entry is a miss, never a wrong answer.

Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this exact shape:

boot	engine log
cold (deploy)	Compiling a graph for compile range (1, 1024) takes 179.24 s → torch.compile took 188.26 s in total
warm (restart)	Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s → torch.compile took 3.97 s in total → init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)

The host directory held 1.4 GB in 4,707 files after the cold boot. Time to /health is dominated by the 172.8 GiB payload load (the warm boot reached health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a proportional cut in boot time.

GPU_MAX_HW_QUEUES=2 caps the HSA hardware queues per GPU. Verified a real knob rather than a no-op - the literal is read out of /opt/rocm/core-10.0/lib/libhsa-runtime64.so - and shipped deliberately alongside the 32x64 gather geometry. It is not A/B measured.

The pool, not the slot count, sets how many conversations live at once. Measured live at this shape: GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x (3,146,129,408 B reserved). The per-token price is shape-dependent: the 262144 shape priced at 7,701 B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer, so a concurrency figure never extrapolates across geometries. The endpoint advertises 3 slots: a third LONG request queues rather than being refused, and shorter requests use the spare blocks. Verified: three concurrent 67-token requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a capacity contract, not a throughput one (2-way concurrency summed BELOW parity at 4064 ctx).

--max-num-seqs 2 is impossible on this model (measured 2026-10-02)

Four single-variable boot probes against the then-deployed shape, production stopped for each:

MAXLEN	SEQS	MNBT	result
262144	2	1024	Engine core initialization failed
262144	2	2048	Engine core initialization failed
262144	3	1024	healthy - 792,069 tokens, 3.02x, arena 6,248
262144	3	2048	healthy - the then-deployed incumbent

The failure is torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant (3), raised from AOT build_guards(strict_error=True) during determine_available_memory -> profile_run -> _dummy_run. model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT is not a lever here - it only moves the compile RANGE ((1,1024) vs (1,2048)), which is why the identical error appears at both rungs. --enforce-eager bypasses the path at the cost of CUDA graphs. The same failure class killed the 2026-10-01 bf16 2 x 262144 shape.

The arena lever is max_model_len, not --kv-cache-memory-bytes

The fork's planner sizes expert cache as budget - non-expert - KV reserve - graphs - overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len - not from the --kv-cache-memory-bytes value:

MAXLEN	KV reserve	arena
262144	2.48 GiB	14.30 GiB / 6,248 slots
200704	1.91 GiB	14.87 GiB / 6,496 slots

D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 pool cut (2.75 GiB) bought zero arena slots. It buys runtime HEADROOM, which is still worth having - the shape it replaced died with 0 bytes free on a real 995-token request. Grow the arena by lowering max_model_len, or by taking the overhead term with --expert-precise-memory (below); the pool is the headroom lever only.

The other arena lever: --expert-precise-memory (overhead 7.60 -> 5.25 GiB)

The flag is now ON. It was refused on 2026-10-01 for a reason that does not generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB, NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also did not stay up: 0 bytes free at runtime, and a real client request (995-token prompt, max_tokens=261149) killed the worker with Tried to allocate 50.00 MiB ... 0 bytes free.

At 200704 x 3 the flag fits with room and is worth +1,026 slots (+16%). What made it affordable is the window trim, not a pool cut: the arena is DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704 window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.

Boot-probed 2026-10-02 (transient systemd units at the production uid + memlock, full-log capture; the rootless bench probe had masked the production memory environment). The night searched two concurrency contracts:

rung	result
3 x 262144, bf16, KV 11,047,010,304	OOM at graph capture — Tried to allocate 76.00 MiB ... 0 bytes free, 30.43 GiB of 31.86 GiB already allocated by PyTorch
3 x 245760, bf16	OOM identically
3 x 237568, bf16	PASS — KV 718,606 tokens
2 x 262144, bf16 + --expert-precise-memory	DEPLOYED then crashed — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag)
2 x 262144, bf16 (no precise flag)	did not stay up either: 0 bytes free at runtime, worker killed by a real request
3 x 200704, fp8, pool sized for 2 contexts	DEPLOYED 2026-10-02 (shape D): 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom
3 x 200704, fp8, + --expert-precise-memory	DEPLOYED 2026-10-02 (shape E): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s

Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even at fp8.

The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE reservation, not a VRAM wall. Failures are clean and diagnostic: the engine reports torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of which Y GiB is free, and the fork's giant ERROR lines are split by journald — read them via journalctl -o json, not -o cat (and podman logs, which is only available while the container lives).

Rollback to the 2026-09-29 fp8+MTP4 shape: git tag frank-primary-fp8-mtp-20261001 (env trio + seam re-pinned there).

Parallelism options on this box (2026-10-01 survey)

Surveyed from the running image's own CLI surface. Verdicts are specific to dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model with only 2 KV heads.

option	flag	verdict on this box
TP	--tensor-parallel-size	DEPLOYED, =2. All-reduce dispatch shows R4D_AR then PYNCCL
DCP	--decode-context-parallel-size	REJECTED at TP=2 — see below
PP	--pipeline-parallel-size	supported; untried (stage handoff plus a TP-keyed expert plan)
DP	--data-parallel-size (+-rank, -start-rank)	impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card)
EP	--enable-expert-parallel + --all2all-backend (+--enable-eplb, --enable-dbo, --enable-elastic-ep)	supported (naive, allgather_reducescatter, deepep_high_throughput, deepep_low_latency, pplx, mori, nixl); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all
multi-node	--nnodes, --node-rank, --master-addr/port, --distributed-executor-backend	supported; the actual path to more cards

DCP is structurally unavailable here — do not re-test on 2 GPUs

--decode-context-parallel-size 2 fails VllmConfig validation at boot:

Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
greater than the model's total number of KV heads (2). Increase
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.

DCP shards the KV along the sequence dimension after KV heads have been spread across the TP ranks, so it requires TP > num_kv_heads. This model has 2 KV heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related fork facts: DCP reuses the TP ranks rather than multiplying them (it is not TP x DCP), and with PCP enabled the valid DCP sizes are {1, pcp, tp x pcp} (vllm/config/parallel.py). The KV memory a DCP win would have bought is real — each rank would hold half the sequence's KV — but it is unreachable without two more cards.

Measuring decode: the harness lives INSIDE the container

/home/dank/mtp_measure.py imports /bench_iter.py by absolute path, so it must run in the engine's namespace and the files must be copied in first:

podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
podman exec frank-primary-qwen python3 /mtp_measure.py

Without the copy the exec dies with can't open file '/mtp_measure.py': [Errno 2] No such file or directory — measured 2026-10-01, which silently cost a whole arm's numbers. A window script must do the copy, not assume the path exists.

The then-deployed shape, measured again (2026-10-01 A/B window)

A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT 2048 in that window) reported GPU KV cache size: 1,079,582 tokens, Maximum concurrency for 262,144 tokens per request: 4.12x — the same figures as the deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is reproducible across boots.

See Also

• dankserver-frontier-moe-feasibility — the sibling feasibility page (which frontier MoE fits this box; the same PCIe-expert-stream wall at V4.1 scale)
• frank-primary-r9v-radiance-qwen38-q8 — RETIRED 2026-09-30 (the R9V fork era; production has been the tcclaviger fork since)
• x4-pair-experiment — the same residency law measured on a narrower link: streaming over PCIe loses, computing in RAM does not care
• Dual-R9700-Cooperation — the board swap that made TP=2 legal here