serverAppConfigs/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	RDNA4 MoE Expert Offload VRAM Budget	The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.	vllm expert-offload vram arena kv-cache ladder tcclaviger	2026-10-05T00:00:00-07:00

RDNA4 MoE Expert Offload VRAM Budget

For VRAM-bound vLLM on the tcclaviger RDNA4 fork. Every boot ledger, arena figure and rate below is from frank-archon's tcclaviger era (2026-09-29 → 10-05: Qwen3.8-Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM) — that page owns the endpoint's shape, tuning history and era record; this page owns only the plan arithmetic, the ladder method and the audit procedure. Boot-failure capture lives in vLLM Boot Forensics and Config Interrogation.

1. The plan law

arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
budget = device_total × gpu_memory_utilization          # expert_plan.py build_plan
slots  = arena // expert_bytes                          # 2.344 MiB/slot ⇒ 437 slots/GiB

The arena is DEVICE-resident, so the plan predicts reality: the fp8 3-window shape's plan totalled 28.27 GiB and rocm-smi measured exactly 28.27 GiB. weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57 driver reserve) vs 31.86 GiB predicts a boot failure before it happens.

Which terms a deploy can move:

term	reachable?	how
budget (util)	yes, law-capped	--gpu-memory-utilization; +0.01 = +0.32 GiB on 31.86 GiB cards
CUDA graphs	yes	container env VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB (or _ESTIMATE_MB for an explicit total)
non-expert weights	no	checkpoint census
KV reserve	no	registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages
runtime overhead	partly	--expert-precise-memory swaps default 7.60 GiB for the registry-fitted 5.25 GiB

Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag (verify with podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py).

UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03). The CLI field is Field(gt=0, le=1), but the worker init demands free > budget: at 1.0 budget == device total and BOTH ranks raise ValueError: Free memory on device cuda:N (31.29/31.86 GiB …) BEFORE a single weight loads — a crash loop (NRestarts ~2 min apart, zero payload progress). Not a knife-edge: unbootable. Even bootable near-ceiling shapes die under real load (0.13 GiB free bf16 boot; 0 bytes free on a real 995-token request).

2. The CUDA-graph term: charge the measured capture

Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the engine's own post-capture line Graph capturing finished … took X GiB (per rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17 GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside pages the profiler had already counted) — the analytic base pool is pure over-charge. Shipped: BASE_MB=0, PER_GRAPH_MB=12 = 0.07 GiB — per-graph, so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit _ESTIMATE_MB total does NOT scale with ladder changes; base+per-graph does.

Verify every deploy: Estimated CUDA graph memory line vs the Graph capturing finished line; if capture exceeds the charge, raise _PER_GRAPH_MB back toward 52. Check VLLM_CUDAGRAPH_* actually present in podman exec … env.

3. The pool is invisible to the plan; free VRAM is the estimator gap

--kv-cache-memory-bytes does NOT appear in the arena arithmetic — the plan reserves the registry price, not the actual pool. Raising the pool consumes real free VRAM without growing the arena; the registry over-prices the pool (9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144; never reuse a B/token ratio across geometries, and subtract ~3.87 MB block-alignment loss when converting a token target to bytes). The ask lands block-aligned: 3,166,500,000 B → 401,408 tokens; verify with GPU KV cache size: N tokens, Maximum concurrency … X.Xx.

Driver-reported free VRAM after hours of traffic is NOT allocatable arena. It is the estimator gap: the registry over-prices KV and charges 5.25 GiB overhead against ~2.5 measured; torch's allocator does not return slabs, so it reads as the high-water mark. The only reachable levers are util (within the law) and the graph charge; everything else needs a registry edit inside the image — the fork's own log line says as much.

MNBT is arena-free up to the anchor: registry overhead = base + 0.22 GiB × max(0, (nbt−4096)/1024) anchored at nbt 4096, so rungs 1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096). MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus arena. MNBT does not move the KV pool.

4. Registry "compile headroom" is not reclaimable slack

--expert-precise-memory selects the fitted overhead (5.25 GiB, this arch at TP=2) over the default (7.60 GiB = fitted + 2.35 GiB of compile and graph-capture headroom the registry documents as "transient VRAM the profiler never sees"). It moves exactly one term — the plan's room — so the arena grows by that 2.35 GiB (~1,025 slots). That headroom is the budget for the phase that allocates LAST: reclaiming it requires paying elsewhere (smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01: bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.

5. Boot ≠ survival (knife-edge shapes)

A shape can pass EVERY boot gate — weights, profile, KV allocation, graph capture, /health 200 — and die on the first real request. Measured 2026-10-01: 5 × 262144 @ fp8 booted (KV 1,350,182 tokens / 5.15x) and was killed seconds later by a 995-token prompt with max_tokens=261149: torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free. After any knife-edge change: send a LARGE-max_tokens request before calling it done, and read post-boot free VRAM per rank (rocm-smi --showmeminfo vram) — the survivor had ~3.6 GiB free; the corpse had 0.

KV accounting — pin the pool BYTES, floor-sized: set --kv-cache-memory-bytes PER RANK to the contract floor (seqs × MAXLEN × measured B/token), not the native window — extra pool bytes come out of the arena/headroom and buy nothing. Measured B/token on this hybrid: 7,701 fp8 / 13,839 bf16 (decomposed: attention 6,138/12,276 + dtype-independent mamba/side 1,563). Measured rates are shape-dependent — 7,891.8 at the 200704×3 shape. Capacity does NOT scale with SEQS: the planner's KV reserve is for ONE max-model-len request.

Parallelism ceilings you cannot flag past:

• --decode-context-parallel-size refused for GQA when TP ≤ num_kv_heads: "requires TP (2) to be greater than the model's total number of KV heads (2)" — DCP shards KV along the sequence dimension AFTER KV heads spread across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model needs TP ≥ 4 = four GPUs.
• EP (--enable-expert-parallel + --all2all-backend) exists but REPLACES the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
• DP requires a full model copy per replica — on a model larger than the card it is impossible, not slow.

6. The context ladder (find the max bootable window by measurement)

"Max possible" = the largest value that BOOTS, measured — never the model's spec maximum, never hand arithmetic.

1. Scale the pool with the window. If KV_BYTES is pinned, it must shrink in lockstep with MAXLEN or the rung tests nothing: KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref)) off a booted reference rung. Verify the rung's capacity still covers seqs × MAXLEN from the engine's own line.
2. Boot under the PRODUCTION environment, not a shell probe — a rootless shell lacks LimitMEMLOCK=infinity and can pass where production OOMs. Transient unit: sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script> pointing the serve script at a scratch env file via its *_ENV_FILE hook.
3. Judge on health + unit liveness: poll /health (200 up / 503 loading / 000 not bound); break early when the unit goes inactive — that is the failure signal, not a timeout.
4. Capture the engine log per rung BEFORE teardown (podman logs dies with podman rm; a podman logs issued too soon after podman run -d fails with no container with that name — record UNKNOWN, never FAIL).
5. Ladder down on FAIL, one bonus step UP on PASS. Measured steps on this box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean the next rung down almost certainly passes: Tried to allocate 76.00 MiB … 0 bytes free … 30.43 GiB allocated by PyTorch died in graph CAPTION; Tried to allocate 10.28 GiB … 9.46 GiB free died in KV ALLOCATION.
6. Clean between rungs: stop the transient unit, podman rm -f, kill orphans by explicit PID (ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'), wait for MemAvailable to recover.

The hazard that cost production: a leftover ladder script whose teardown podman rm -fs the production container fires against a freshly DEPLOYED engine. Teardown runs ONLY on failure, never after a pass; kill the whole scratch process tree and verify none remain BEFORE deploying. On PASS: print the measured GPU KV cache size / Maximum concurrency line, LEAVE IT SERVING, and pin the number into the repo (env + seam + harness contextWindow) before deploying.

Cutover order: ladder → winner passes and serves → repo env + seam + harness → tests → commit+push → verify no scratch scripts remain → stash foreign work (git stash -u) if the tree is dirty → deploy (one restart) → verify health

• engine flags in the log.

Measured ladder on this box (200704×3, fp8 KV, precise memory):

shape	util	graphs	arena	result
E	0.98	0.55	17.22 GiB / 7,522 slots	boot ok
F	0.982	0.32	17.52 GiB / 7,653 slots	boot ok
1.0	1.00	0.07	—	UNBOOTABLE (worker-init ValueError, both ranks)
G (deployed)	0.982	0.07	17.77 GiB / 7,762 slots	KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s

bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) — ~207k per context max.

7. Arena audit procedure (when asked for "more experts")

1. Read the live plan before proposing anything — grep -oE with a digit-anchored tail (plain grep/journalctl truncate these columns to 17.):

   ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
     | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
   

2. Classify each term (§1 table); take only the two reachable levers.

3. Verify after deploy: plan lines show new budget/graphs/arena; GPU KV cache size matches the ask; graph-estimate vs post-capture; rocm-smi high-water per card = the real margin; systemctl show frank-archon -p NRestarts --value = 0; a real chat completion from a PEER (health-probe via 192.168.0.235:7770/health, not localhost inside an ssh session on the host — the latter returns 000 and looks like an outage).

Related

• RDNA4 Container Serving — the bring-up path and container traps
• vLLM Boot Forensics and Config Interrogation — capture and contracts
• frank-archon — measured history
• Llama Cpp Serving Sizing — the llama.cpp equivalent