serverAppConfigs/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

---
type: research
title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning
description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size).
tags:
  - dankserver
  - r9700
  - gfx1201
  - qwen3.8
  - fp8
  - expert-offload
  - speculative-decoding
  - tuning
timestamp: 2026-09-29
---

# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning

> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving
> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what
> runtime flags actually move decode at a 32k working point?
>
> **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by
> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the
> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served):
> ~half of all gather traffic was useless copies, and the copies it "saved"
> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s)
> survived every other challenger measured: gather geometry, spec depth,
> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.

This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
image `docker.io/tcclaviger/vllm:latest`, engine
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service —
the R9V production unit is untouched, so no stop-and-mask was required for
any of this. All numbers are decode tok/s measured from the server's own
`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
under speculative decoding because rejected drafts change the accounting.

## Deployment shape (the tuned incumbent)

Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in
CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is
off):

```
MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8   (~172.8 GiB, 131 shards)
IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
# the floating :latest tag moved once already — always launch by digest.
# repo tag of this state: qwen-fp8-tuned-20260929
--tensor-parallel-size 2  --max-model-len 32768  --gpu-memory-utilization 0.98
--kv-cache-memory-bytes 2147483648  --max-num-seqs 4  --enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
```

Published to the host LAN via the serve script's `PORTMAP` hook
(`PORTMAP=7773:7770`): health and completions answer on
`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
decode 36.49/34.55/35.98 through the published path). Host port 7773 was
chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED —
any future start would collide with a published 7770.

Device notes carried from the bring-up: `--device /dev/dri` must be the
WHOLE directory (individual renderD nodes enumerate but refuse contexts —
"No CUDA GPUs are available" while device_count() reports 2); the image
entrypoint execs `vllm serve` itself.

The memory plan this produces (from the engine's own `expert_plan.py` log):

| region | size |
|---|---:|
| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB |
|   VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) |
|   swap region | 11.53 GiB (5,038 slots, 111-112/layer) |
| host-backed experts | 52.74 GiB per rank, 23,040 experts |
| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool |
| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) |
| runtime overhead | 7.6 GiB (non-torch + buffers + activation) |

## Measured arms (2026-09-29)

Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
64-token warm pass, temperature 0, max_tokens 256:

| Arm | 166 | 1087 | 4064 | Acceptance | Verdict |
|---|---:|---:|---:|---:|---|
| **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** |
| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) |
| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) |
| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out |
| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out |
| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out |
| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out |
| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out |
| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out |
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |

\* The TunableOp numbers are invalid by construction: the tuned GEMMs
changed generated behavior (repetition-loop, the server's
"repetition detected" halt firing at ~140-146 tokens instead of the full
256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were
measured on those halted degenerate completions, and the text-only check on
a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning
that changes output text is broken tuning.

The decode curve without spec decode is dead flat 12.11 tok/s from 304 to
16,095 tokens — decode is context-independent on this shape, bound per-step
by the expert gather, not by KV depth.

## Telemetry — the measurement that found the win

`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window
(`/start_profile` → traffic → `/stop_profile`) makes the fork log and write
`expert_cache_rank<N>.json` beside the traces. **`torch_profiler_with_stack=false`
is mandatory in the profiler config** — the default (true) grew a 6.5-min
window's trace buffer ~40 GB and the kernel OOM-killed the engine core
(anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at
166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`.

The fork's own numbers (102 steps, both ranks identical):

```
hit rate 0.8245 (137,084/166,262)   host installs 15,814
read-through 13,364                 prefetched 36,720
served by prefetch (migrations) 13,369
```

- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at
  100%; swap-region layers 78-81%. The "arena holds only 27% of experts
  → 73% miss" arithmetic was wrong — temporal reuse carries it.
- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369
  (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure
  waste — roughly half of all gather traffic — and the 13,369 "savings"
  would have been installs anyway. Depth-1 prefetch also cannot cover the
  ~2,400-slab per-step working set with its 16 slots/layer (768 total).
- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs,
  2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.

## The wall, measured

- **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta
  32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 =
  31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle
  downshift, not the loaded rate.
- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab)
  **22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈
  pageable — staging is not the lever. Single-transfer latency at slab size
  105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the
  link delivers at the same size**. The bottleneck is the fork's read path
  (chunk/lane geometry, per-transfer cost), not the wire.
- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each
  step is ~100 ms of compute then hundreds of ms waiting on expert
  streaming. rocm-smi reports *utilization*, which reads ~0 while weights
  load during an arm's reload; VRAM is the honest panel there.

## What the remaining flags cannot do (all measured)

- **Gather geometry** — 16x32 and 32x64 measured no better despite the
  fork's "tuned at the Gen4 ceiling" comment; the constants are already at
  parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6
  additionally proves the UVA read-through path (`step tokens >
  R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in.
- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the
  tier is 3x too small for the working set and the predictor is 36%
  precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but
  the precision result makes tuning them pointless.
- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on
  device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph
  capture (`Constraints violated (L['query_start_loc'].size()[0])`); and
  the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral
  in both directions.
- **Concurrency** — under TP=2 both streams' tokens are computed on both
  cards, so each rank gathers the union of both streams' misses; the links
  do not add capacity and the shared arena thrashes. Summed 2-way (25.49)
  measured *below* single-stream (26.8) with prefetch on.
- **ngram-only drafting** — fails at engine init:
  `RuntimeError: PLE inputs were not prepared`; the fork prepares this
  model's PLE inputs only on the MTP spec path. Needs a fork patch, not a
  config.
- **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only
  (K+V together); a V-only q8 needs a fork patch.

## Fork bugs / traps found (for the engine authors or a future patch)

1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is
   active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set
   (`tunableop_warmup.py`).
2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent
   cache (container-local), and corrupt numerics (repetition loops,
   early-EOS halts, invalid measurements). Rejected outright.
3. The torch-profiler stop path hangs the workers on ROCm when
   `torch_profiler_with_stack` is true (default): >10 min hangs on two
   attempts while the AsyncLLM wrapper reports "stopped successfully" in
   1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB
   and OOM-killed the engine core. With `with_stack=false` the stop
   completes and the telemetry report runs. The telemetry counters remain
   reachable ONLY through that profiler stop — no direct RPC trigger
   exists (`telemetry_report` has exactly one caller, `gpu_worker.py`).
4. ngram spec method does not prepare PLE inputs (above).

## Where the headroom lives now

The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:

1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s
   software gap at slab size. If the gather reached wire rate, decode
   lands in the 60+ tok/s band before acceptance effects. Code work in
   `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
   copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
   `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
   needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
   not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
   estimates max len 184,224 at that pool; the exact-4.0 override loses
   ~20 MB to accounting and boots as 3.98 < 4.0). Decode band
   **34.1-36.5** across both KV sizes — the arena is **identical (5,648
   slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
   margin, not slots), so the window costs ~10% decode vs the 32k config
   and no KV trim recovers it. The alternative that keeps the full
   37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
   sizes the arena AFTER the KV override (the earlier "before" reading was
   wrong) — the arena is what remains.
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
   2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
   so slots are a capacity feature, not a throughput one. Two full 256k
   streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting
   the arena below usefulness; slots-at-256k is a fork-patch or
   smaller-payload question.

## Reproducing

Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the
serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter
decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth,
for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm
scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
(outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts
are `podman cp`'d into the container after every recreate (the serve script
re-copies them; a recreate wipes container-local files).

## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)

Two same-day cutovers moved this page's findings into production:

1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
   window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
   `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
   production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
   measured 37.4-40.6 tok/s at ~54% acceptance.
2. **The overnight redo campaign re-measured EVERY lever at matched
   geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
   and concluded the day: the incumbent cutover to **bf16 KV, no
   speculative decode** (Frank's call).

### What the redo campaign measured (all at shallow depth, 166/1087/4064)

| lever | verdict |
|---|---|
| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
| UVA | 2048 POISON (-39%); 8192 ~parity |
| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
| six slots | every SEQS-6 KV sizing refused at engine init |

Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.

### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)

```
FRANK_PRIMARY_MAXLEN=200704   200704-token window
FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3166500000   PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048       --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=1.0        budget = total x util, forced to the le=1 ceiling
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12  -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
NO --speculative-config, NO --expert-prefetch
```

### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost

`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
not an idle artifact). Growing the expert arena into it is NOT a matter of
raising `--gpu-memory-utilization`, because the arena is

```
arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
budget = device_total x gpu_memory_utilization          # expert_plan.py:567
```

and every term on the right is either fixed or a lie:

| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |

Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
graph term further (0.32 -> 0.07 GiB) at the same util.

**Shape G takes the graph term to the measured floor, util to the law.** The
graph base pool is the actual over-charge: shape F's own boot reported
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
full-length conversations with real spare. (The ask/landed delta is block
alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)

**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
arena; both workers refused init BEFORE loading a single weight, twice:
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
demands free > budget and budget == total, so `le=1` on the CLI field is a
trap: the engine dies in a 2-restart crash loop with zero payload progress.
So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
995-token request). Everything beyond ~7,912 slots needs the registry edited
inside the image (the fork's log line says exactly that: *"Update the
registry entry if this drifts"*), not a deploy.

**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
was itself fitted at nbt 4096, which means 2048 is covered by an activation
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
SEQS property). The costs are a one-time recompile (the compile **range** is
part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
prefill chunk, which the 09-29 sweep found decode-interchangeable across
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).

**`--expert-precise-memory` is the arena lever that does not touch the window.**
It attacks the planner's overhead term (shape F above attacks the graph term —
the third and last term this repo can reach), and the flag swaps the default
registry figure for the fitted one. Measured
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
booted AS the service; numbers are the engine's own plan lines):

| plan line | shape D | **shape E** |
|---|---|---|
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
| slots | 6,496 | **7,522** (+1,026, +16%) |
| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
| boot | - | healthy in 420 s, no OOM |

`'expert_precise_memory': True` also appears in `non-default args`, so a silent
argparse drop is visible in the boot log. The flag lands **in the arena, not the
pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
the pool is not an arena lever and cutting it further would only cost
concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
not the shipped value.

**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
container: `No such file or directory`), and after a boot that directory holds
`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
the container's ephemeral layer. Every
`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
engine re-pays:

```
Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
Dynamo bytecode transform time: 4.99 s
Compiling a graph for compile range (1, 1024) takes 173.19 s
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
torch.compile took 181.20 s in total
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
```

Mounting a host directory at `/root/.cache` makes every later boot load the AOT
artifact instead. The mount is deliberately the **parent**, not
`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
nothing, because the image ships no `/root/.cache`. The cache key
includes the compile range, so the MNBT choice costs exactly one compilation.
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
answer.

**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
exact shape:**

| boot | engine log |
|---|---|
| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |

The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
proportional cut in boot time.

`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
rather than a no-op - the literal is read out of
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
alongside the 32x64 gather geometry. It is **not** A/B measured.

**The pool, not the slot count, sets how many conversations live at once.**
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
so a concurrency figure never extrapolates across geometries. The endpoint
advertises 3 slots: a third LONG request queues rather than being refused, and
shorter requests use the spare blocks. Verified: three concurrent 67-token
requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
capacity contract, not a throughput one (2-way concurrency summed BELOW parity
at 4064 ctx).

#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)

Four single-variable boot probes against the then-deployed shape, production
stopped for each:

| MAXLEN | SEQS | MNBT | result |
|---|---|---|---|
| 262144 | 2 | 1024 | **Engine core initialization failed** |
| 262144 | 2 | 2048 | **Engine core initialization failed** |
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |

The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
is why the identical error appears at both rungs. `--enforce-eager` bypasses
the path at the cost of CUDA graphs. The same failure class killed the
2026-10-01 bf16 2 x 262144 shape.

#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`

The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
max_model_len** - not from the `--kv-cache-memory-bytes` value:

| MAXLEN | KV reserve | arena |
|---|---|---|
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |

D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
which is still worth having - the shape it replaced died with `0 bytes free`
on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
taking the overhead term with `--expert-precise-memory` (below); the pool is the
headroom lever only.

#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)

The flag is now ON. It was refused on 2026-10-01 for a reason that does not
generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
did not stay up: 0 bytes free at runtime, and a real client request (995-token
prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.

At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
made it affordable is the window trim, not a pool cut: the arena is
DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.

Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
memory environment). The night searched two concurrency contracts:

| rung | result |
|---|---|
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
| 3 x 245760, bf16 | **OOM identically** |
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |

Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
at fp8.

The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
only available while the container lives).

Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).

## Parallelism options on this box (2026-10-01 survey)

Surveyed from the running image's own CLI surface. Verdicts are specific to
dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
with only 2 KV heads.

| option | flag | verdict on this box |
|---|---|---|
| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |

### DCP is structurally unavailable here — do not re-test on 2 GPUs
`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:

```
Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
greater than the model's total number of KV heads (2). Increase
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
```

DCP shards the KV along the *sequence* dimension after KV heads have been spread
across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
each rank would hold half the sequence's KV — but it is unreachable without two
more cards.

### Measuring decode: the harness lives INSIDE the container
`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
run in the engine's namespace and the files must be copied in first:

```
podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
podman exec frank-primary-qwen python3 /mtp_measure.py
```

Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
No such file or directory` — measured 2026-10-01, which silently cost a whole
arm's numbers. A window script must do the copy, not assume the path exists.

### The then-deployed shape, measured again (2026-10-01 A/B window)
A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
reproducible across boots.

## See Also

- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
  (which frontier MoE fits this box; the same PCIe-expert-stream wall at
  V4.1 scale)
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
  fork era; production has been the tcclaviger fork since)
- [[x4-pair-experiment]] — the same residency law measured on a narrower
  link: streaming over PCIe loses, computing in RAM does not care
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here