Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
220 lines
12 KiB
Markdown
220 lines
12 KiB
Markdown
---
|
||
type: reference
|
||
title: RDNA4 MoE Expert Offload VRAM Budget
|
||
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
|
||
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# RDNA4 MoE Expert Offload VRAM Budget
|
||
|
||
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-archon: Qwen3.8-
|
||
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
|
||
Tuning history and the deployed shape live in
|
||
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
|
||
[[vLLM Boot Forensics and Config Interrogation]].
|
||
|
||
## 1. The plan law
|
||
|
||
```
|
||
arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
|
||
budget = device_total × gpu_memory_utilization # expert_plan.py build_plan
|
||
slots = arena // expert_bytes # 2.344 MiB/slot ⇒ 437 slots/GiB
|
||
```
|
||
|
||
**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
|
||
3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
|
||
28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
|
||
driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
|
||
|
||
Which terms a deploy can move:
|
||
|
||
| term | reachable? | how |
|
||
|---|---|---|
|
||
| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
|
||
| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
|
||
| non-expert weights | no | checkpoint census |
|
||
| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
|
||
| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
|
||
|
||
Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
|
||
(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
|
||
|
||
**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
|
||
`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
|
||
budget == device total and BOTH ranks raise
|
||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
|
||
single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
|
||
progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
|
||
shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
|
||
995-token request).
|
||
|
||
## 2. The CUDA-graph term: charge the measured capture
|
||
|
||
Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
|
||
engine's own post-capture line `Graph capturing finished … took X GiB` (per
|
||
rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
|
||
GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
|
||
pages the profiler had already counted) — the analytic base pool is pure
|
||
over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
|
||
so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
|
||
`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
|
||
|
||
**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
|
||
capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
|
||
back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
||
`podman exec … env`.
|
||
|
||
## 3. The pool is invisible to the plan; free VRAM is the estimator gap
|
||
|
||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||
reserves the registry price, not the actual pool. Raising the pool consumes
|
||
real free VRAM without growing the arena; the registry over-prices the pool
|
||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
||
block-alignment loss when converting a token target to bytes). The ask lands
|
||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
||
|
||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||
5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
|
||
slabs, so it reads as the high-water mark. The only reachable levers are util
|
||
(within the law) and the graph charge; everything else needs a registry edit
|
||
inside the image — the fork's own log line says as much.
|
||
|
||
**MNBT is arena-free up to the anchor:** registry overhead =
|
||
`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
|
||
1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
|
||
MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
|
||
the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
|
||
arena. MNBT does not move the KV pool.
|
||
|
||
## 4. Registry "compile headroom" is not reclaimable slack
|
||
|
||
`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
|
||
TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
|
||
graph-capture headroom** the registry documents as "transient VRAM the
|
||
profiler never sees"). It moves exactly one term — the plan's `room` — so the
|
||
arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
|
||
for the phase that allocates LAST**: reclaiming it requires paying elsewhere
|
||
(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
|
||
bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
|
||
crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
|
||
|
||
## 5. Boot ≠ survival (knife-edge shapes)
|
||
|
||
A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
|
||
capture, `/health` 200 — and die on the first real request. Measured
|
||
2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
|
||
killed seconds later by a 995-token prompt with `max_tokens=261149`:
|
||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
|
||
any knife-edge change: send a LARGE-`max_tokens` request before calling it
|
||
done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
||
— the survivor had ~3.6 GiB free; the corpse had 0.
|
||
|
||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
||
planner's KV reserve is for ONE max-model-len request.
|
||
|
||
**Parallelism ceilings you cannot flag past:**
|
||
|
||
- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
|
||
*"requires TP (2) to be greater than the model's total number of KV heads
|
||
(2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
|
||
across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
|
||
needs TP ≥ 4 = four GPUs.
|
||
- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
|
||
the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
|
||
- DP requires a full model copy per replica — on a model larger than the
|
||
card it is impossible, not slow.
|
||
|
||
## 6. The context ladder (find the max bootable window by measurement)
|
||
|
||
"Max possible" = the largest value that BOOTS, measured — never the model's
|
||
spec maximum, never hand arithmetic.
|
||
|
||
1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
|
||
in lockstep with MAXLEN or the rung tests nothing:
|
||
`KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
|
||
off a booted reference rung. Verify the rung's capacity still covers
|
||
`seqs × MAXLEN` from the engine's own line.
|
||
2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
|
||
shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
|
||
Transient unit:
|
||
`sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
|
||
pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
|
||
3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
|
||
/ 000 not bound); break early when the unit goes inactive — that is the
|
||
failure signal, not a timeout.
|
||
4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
|
||
with `podman rm`; a `podman logs` issued too soon after `podman run -d`
|
||
fails with *no container with that name* — record UNKNOWN, never FAIL).
|
||
5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
|
||
box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
|
||
the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
|
||
0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
|
||
`Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
|
||
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
|
||
orphans by explicit PID
|
||
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
|
||
wait for `MemAvailable` to recover.
|
||
|
||
**The hazard that cost production:** a leftover ladder script whose teardown
|
||
`podman rm -f`s the production container fires against a freshly DEPLOYED
|
||
engine. Teardown runs ONLY on failure, never after a pass; kill the whole
|
||
scratch process tree and verify none remain BEFORE deploying. On PASS: print
|
||
the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
|
||
SERVING, and pin the number into the repo (env + seam + harness
|
||
contextWindow) before deploying.
|
||
|
||
Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
||
→ tests → commit+push → verify no scratch scripts remain → stash foreign work
|
||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||
+ engine flags in the log.
|
||
|
||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
||
|
||
| shape | util | graphs | arena | result |
|
||
|---|---|---|---|---|
|
||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
||
|
||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||
~207k per context max.
|
||
|
||
## 7. Arena audit procedure (when asked for "more experts")
|
||
|
||
1. Read the live plan before proposing anything — `grep -oE` with a
|
||
digit-anchored tail (plain grep/journalctl truncate these columns to
|
||
`17.`):
|
||
|
||
```
|
||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
|
||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||
```
|
||
|
||
2. Classify each term (§1 table); take only the two reachable levers.
|
||
3. Verify after deploy: plan lines show new budget/graphs/arena;
|
||
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
|
||
`rocm-smi` high-water per card = the real margin;
|
||
`systemctl show frank-archon -p NRestarts --value` = 0; a real chat
|
||
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
|
||
`localhost` inside an ssh session on the host — the latter returns `000`
|
||
and looks like an outage).
|
||
|
||
## Related
|
||
|
||
- [[RDNA4 Container Serving]] — the bring-up path and container traps
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
|
||
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
|