Frank
04f4e2c6d4
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
---
|
||
type: reference
|
||
title: RDNA4 MoE Expert Offload VRAM Budget
|
||
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
|
||
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# RDNA4 MoE Expert Offload VRAM Budget
|
||
|
||
For VRAM-bound vLLM on the tcclaviger RDNA4 fork. Every boot ledger, arena
|
||
figure and rate below is from [[frank-archon]]'s tcclaviger era (2026-09-29 →
|
||
10-05: Qwen3.8-Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload
|
||
to host RAM) — that page owns the endpoint's shape, tuning history and era
|
||
record; this page owns only the plan arithmetic, the ladder method and the audit
|
||
procedure. Boot-failure capture lives in
|
||
[[vLLM Boot Forensics and Config Interrogation]].
|
||
|
||
## 1. The plan law
|
||
|
||
```
|
||
arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
|
||
budget = device_total × gpu_memory_utilization # expert_plan.py build_plan
|
||
slots = arena // expert_bytes # 2.344 MiB/slot ⇒ 437 slots/GiB
|
||
```
|
||
|
||
**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
|
||
3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
|
||
28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
|
||
driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
|
||
|
||
Which terms a deploy can move:
|
||
|
||
| term | reachable? | how |
|
||
|---|---|---|
|
||
| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
|
||
| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
|
||
| non-expert weights | no | checkpoint census |
|
||
| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
|
||
| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
|
||
|
||
Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
|
||
(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
|
||
|
||
**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
|
||
`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
|
||
budget == device total and BOTH ranks raise
|
||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
|
||
single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
|
||
progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
|
||
shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
|
||
995-token request).
|
||
|
||
## 2. The CUDA-graph term: charge the measured capture
|
||
|
||
Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
|
||
engine's own post-capture line `Graph capturing finished … took X GiB` (per
|
||
rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
|
||
GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
|
||
pages the profiler had already counted) — the analytic base pool is pure
|
||
over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
|
||
so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
|
||
`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
|
||
|
||
**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
|
||
capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
|
||
back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
||
`podman exec … env`.
|
||
|
||
## 3. The pool is invisible to the plan; free VRAM is the estimator gap
|
||
|
||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||
reserves the registry price, not the actual pool. Raising the pool consumes
|
||
real free VRAM without growing the arena; the registry over-prices the pool
|
||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
||
block-alignment loss when converting a token target to bytes). The ask lands
|
||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
||
|
||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||
5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
|
||
slabs, so it reads as the high-water mark. The only reachable levers are util
|
||
(within the law) and the graph charge; everything else needs a registry edit
|
||
inside the image — the fork's own log line says as much.
|
||
|
||
**MNBT is arena-free up to the anchor:** registry overhead =
|
||
`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
|
||
1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
|
||
MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
|
||
the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
|
||
arena. MNBT does not move the KV pool.
|
||
|
||
## 4. Registry "compile headroom" is not reclaimable slack
|
||
|
||
`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
|
||
TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
|
||
graph-capture headroom** the registry documents as "transient VRAM the
|
||
profiler never sees"). It moves exactly one term — the plan's `room` — so the
|
||
arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
|
||
for the phase that allocates LAST**: reclaiming it requires paying elsewhere
|
||
(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
|
||
bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
|
||
crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
|
||
|
||
## 5. Boot ≠ survival (knife-edge shapes)
|
||
|
||
A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
|
||
capture, `/health` 200 — and die on the first real request. Measured
|
||
2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
|
||
killed seconds later by a 995-token prompt with `max_tokens=261149`:
|
||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
|
||
any knife-edge change: send a LARGE-`max_tokens` request before calling it
|
||
done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
||
— the survivor had ~3.6 GiB free; the corpse had 0.
|
||
|
||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
||
planner's KV reserve is for ONE max-model-len request.
|
||
|
||
**Parallelism ceilings you cannot flag past:**
|
||
|
||
- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
|
||
*"requires TP (2) to be greater than the model's total number of KV heads
|
||
(2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
|
||
across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
|
||
needs TP ≥ 4 = four GPUs.
|
||
- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
|
||
the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
|
||
- DP requires a full model copy per replica — on a model larger than the
|
||
card it is impossible, not slow.
|
||
|
||
## 6. The context ladder (find the max bootable window by measurement)
|
||
|
||
"Max possible" = the largest value that BOOTS, measured — never the model's
|
||
spec maximum, never hand arithmetic.
|
||
|
||
1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
|
||
in lockstep with MAXLEN or the rung tests nothing:
|
||
`KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
|
||
off a booted reference rung. Verify the rung's capacity still covers
|
||
`seqs × MAXLEN` from the engine's own line.
|
||
2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
|
||
shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
|
||
Transient unit:
|
||
`sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
|
||
pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
|
||
3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
|
||
/ 000 not bound); break early when the unit goes inactive — that is the
|
||
failure signal, not a timeout.
|
||
4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
|
||
with `podman rm`; a `podman logs` issued too soon after `podman run -d`
|
||
fails with *no container with that name* — record UNKNOWN, never FAIL).
|
||
5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
|
||
box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
|
||
the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
|
||
0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
|
||
`Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
|
||
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
|
||
orphans by explicit PID
|
||
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
|
||
wait for `MemAvailable` to recover.
|
||
|
||
**The hazard that cost production:** a leftover ladder script whose teardown
|
||
`podman rm -f`s the production container fires against a freshly DEPLOYED
|
||
engine. Teardown runs ONLY on failure, never after a pass; kill the whole
|
||
scratch process tree and verify none remain BEFORE deploying. On PASS: print
|
||
the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
|
||
SERVING, and pin the number into the repo (env + seam + harness
|
||
contextWindow) before deploying.
|
||
|
||
Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
||
→ tests → commit+push → verify no scratch scripts remain → stash foreign work
|
||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||
+ engine flags in the log.
|
||
|
||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
||
|
||
| shape | util | graphs | arena | result |
|
||
|---|---|---|---|---|
|
||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
||
|
||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||
~207k per context max.
|
||
|
||
## 7. Arena audit procedure (when asked for "more experts")
|
||
|
||
1. Read the live plan before proposing anything — `grep -oE` with a
|
||
digit-anchored tail (plain grep/journalctl truncate these columns to
|
||
`17.`):
|
||
|
||
```
|
||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
|
||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||
```
|
||
|
||
2. Classify each term (§1 table); take only the two reachable levers.
|
||
3. Verify after deploy: plan lines show new budget/graphs/arena;
|
||
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
|
||
`rocm-smi` high-water per card = the real margin;
|
||
`systemctl show frank-archon -p NRestarts --value` = 0; a real chat
|
||
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
|
||
`localhost` inside an ssh session on the host — the latter returns `000`
|
||
and looks like an outage).
|
||
|
||
## Related
|
||
|
||
- [[RDNA4 Container Serving]] — the bring-up path and container traps
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
|
||
- [[frank-archon]] — measured history
|
||
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
|