---
type: reference
title: RDNA4 MoE Expert Offload VRAM Budget
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
timestamp: 2026-10-05T00:00:00-07:00
---

# RDNA4 MoE Expert Offload VRAM Budget

For VRAM-bound vLLM on the tcclaviger RDNA4 fork. Every boot ledger, arena
figure and rate below is from [[frank-archon]]'s tcclaviger era (2026-09-29 →
10-05: Qwen3.8-Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload
to host RAM) — that page owns the endpoint's shape, tuning history and era
record; this page owns only the plan arithmetic, the ladder method and the audit
procedure. Boot-failure capture lives in
[[vLLM Boot Forensics and Config Interrogation]].

## 1. The plan law

```
arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
budget = device_total × gpu_memory_utilization          # expert_plan.py build_plan
slots  = arena // expert_bytes                          # 2.344 MiB/slot ⇒ 437 slots/GiB
```

**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.

Which terms a deploy can move:

| term | reachable? | how |
|---|---|---|
| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
| non-expert weights | no | checkpoint census |
| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |

Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).

**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
budget == device total and BOTH ranks raise
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
995-token request).

## 2. The CUDA-graph term: charge the measured capture

Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
engine's own post-capture line `Graph capturing finished … took X GiB` (per
rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
pages the profiler had already counted) — the analytic base pool is pure
over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.

**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
`podman exec … env`.

## 3. The pool is invisible to the plan; free VRAM is the estimator gap

`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
reserves the registry price, not the actual pool. Raising the pool consumes
real free VRAM without growing the arena; the registry over-prices the pool
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
block-alignment loss when converting a token target to bytes). The ask lands
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.

**Driver-reported free VRAM after hours of traffic is NOT allocatable
arena.** It is the estimator gap: the registry over-prices KV and charges
5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
slabs, so it reads as the high-water mark. The only reachable levers are util
(within the law) and the graph charge; everything else needs a registry edit
inside the image — the fork's own log line says as much.

**MNBT is arena-free up to the anchor:** registry overhead =
`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
arena. MNBT does not move the KV pool.

## 4. Registry "compile headroom" is not reclaimable slack

`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
graph-capture headroom** the registry documents as "transient VRAM the
profiler never sees"). It moves exactly one term — the plan's `room` — so the
arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
for the phase that allocates LAST**: reclaiming it requires paying elsewhere
(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.

## 5. Boot ≠ survival (knife-edge shapes)

A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
capture, `/health` 200 — and die on the first real request. Measured
2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
killed seconds later by a 995-token prompt with `max_tokens=261149`:
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
any knife-edge change: send a LARGE-`max_tokens` request before calling it
done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
— the survivor had ~3.6 GiB free; the corpse had 0.

**KV accounting — pin the pool BYTES, floor-sized:** set
`--kv-cache-memory-bytes` PER RANK to the contract floor
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
come out of the arena/headroom and buy nothing. Measured B/token on this
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
planner's KV reserve is for ONE max-model-len request.

**Parallelism ceilings you cannot flag past:**

- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
  *"requires TP (2) to be greater than the model's total number of KV heads
  (2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
  across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
  needs TP ≥ 4 = four GPUs.
- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
  the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
- DP requires a full model copy per replica — on a model larger than the
  card it is impossible, not slow.

## 6. The context ladder (find the max bootable window by measurement)

"Max possible" = the largest value that BOOTS, measured — never the model's
spec maximum, never hand arithmetic.

1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
   in lockstep with MAXLEN or the rung tests nothing:
   `KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
   off a booted reference rung. Verify the rung's capacity still covers
   `seqs × MAXLEN` from the engine's own line.
2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
   shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
   Transient unit:
   `sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
   pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
   / 000 not bound); break early when the unit goes inactive — that is the
   failure signal, not a timeout.
4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
   with `podman rm`; a `podman logs` issued too soon after `podman run -d`
   fails with *no container with that name* — record UNKNOWN, never FAIL).
5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
   box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
   the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
   0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
   `Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
   orphans by explicit PID
   (`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
   wait for `MemAvailable` to recover.

**The hazard that cost production:** a leftover ladder script whose teardown
`podman rm -f`s the production container fires against a freshly DEPLOYED
engine. Teardown runs ONLY on failure, never after a pass; kill the whole
scratch process tree and verify none remain BEFORE deploying. On PASS: print
the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
SERVING, and pin the number into the repo (env + seam + harness
contextWindow) before deploying.

Cutover order: ladder → winner passes and serves → repo env + seam + harness
→ tests → commit+push → verify no scratch scripts remain → stash foreign work
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
+ engine flags in the log.

**Measured ladder on this box (200704×3, fp8 KV, precise memory):**

| shape | util | graphs | arena | result |
|---|---|---|---|---|
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |

bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
~207k per context max.

## 7. Arena audit procedure (when asked for "more experts")

1. Read the live plan before proposing anything — `grep -oE` with a
   digit-anchored tail (plain grep/journalctl truncate these columns to
   `17.`):

   ```
   ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
     | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
   ```

2. Classify each term (§1 table); take only the two reachable levers.
3. Verify after deploy: plan lines show new budget/graphs/arena;
   `GPU KV cache size` matches the ask; graph-estimate vs post-capture;
   `rocm-smi` high-water per card = the real margin;
   `systemctl show frank-archon -p NRestarts --value` = 0; a real chat
   completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
   `localhost` inside an ssh session on the host — the latter returns `000`
   and looks like an outage).

## Related

- [[RDNA4 Container Serving]] — the bring-up path and container traps
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[frank-archon]] — measured history
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent