docs: vault link integrity restored after rogue-session doc WIP landed
The parallel session's stash pop left GGUF Compatibility, Multi-Device Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving, vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9 unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link, and two multi-line-wrapped link tokens). All repointed to real pages; six rows added to the vault index; seam green.
This commit is contained in:
Frank committed
1 parent
380aafa1cc
commit
a6ad9af708
10 files changed
+2013
-1
No files matched your search
@@ -0,0 +1,219 @@
|
||||
---
|
||||
type: reference
|
||||
title: RDNA4 MoE Expert Offload VRAM Budget
|
||||
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
|
||||
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# RDNA4 MoE Expert Offload VRAM Budget
|
||||
|
||||
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
|
||||
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
|
||||
Tuning history and the deployed shape live in
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
|
||||
[[vLLM Boot Forensics and Config Interrogation]].
|
||||
|
||||
## 1. The plan law
|
||||
|
||||
```
|
||||
arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
|
||||
budget = device_total × gpu_memory_utilization # expert_plan.py build_plan
|
||||
slots = arena // expert_bytes # 2.344 MiB/slot ⇒ 437 slots/GiB
|
||||
```
|
||||
|
||||
**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
|
||||
3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
|
||||
28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
|
||||
driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
|
||||
|
||||
Which terms a deploy can move:
|
||||
|
||||
| term | reachable? | how |
|
||||
|---|---|---|
|
||||
| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
|
||||
| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
|
||||
| non-expert weights | no | checkpoint census |
|
||||
| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
|
||||
| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
|
||||
|
||||
Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
|
||||
(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
|
||||
|
||||
**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
|
||||
`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
|
||||
budget == device total and BOTH ranks raise
|
||||
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
|
||||
single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
|
||||
progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
|
||||
shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
|
||||
995-token request).
|
||||
|
||||
## 2. The CUDA-graph term: charge the measured capture
|
||||
|
||||
Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
|
||||
engine's own post-capture line `Graph capturing finished … took X GiB` (per
|
||||
rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
|
||||
GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
|
||||
pages the profiler had already counted) — the analytic base pool is pure
|
||||
over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
|
||||
so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
|
||||
`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
|
||||
|
||||
**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
|
||||
capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
|
||||
back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
||||
`podman exec … env`.
|
||||
|
||||
## 3. The pool is invisible to the plan; free VRAM is the estimator gap
|
||||
|
||||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||||
reserves the registry price, not the actual pool. Raising the pool consumes
|
||||
real free VRAM without growing the arena; the registry over-prices the pool
|
||||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
||||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
||||
block-alignment loss when converting a token target to bytes). The ask lands
|
||||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
||||
|
||||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||||
5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
|
||||
slabs, so it reads as the high-water mark. The only reachable levers are util
|
||||
(within the law) and the graph charge; everything else needs a registry edit
|
||||
inside the image — the fork's own log line says as much.
|
||||
|
||||
**MNBT is arena-free up to the anchor:** registry overhead =
|
||||
`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
|
||||
1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
|
||||
MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
|
||||
the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
|
||||
arena. MNBT does not move the KV pool.
|
||||
|
||||
## 4. Registry "compile headroom" is not reclaimable slack
|
||||
|
||||
`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
|
||||
TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
|
||||
graph-capture headroom** the registry documents as "transient VRAM the
|
||||
profiler never sees"). It moves exactly one term — the plan's `room` — so the
|
||||
arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
|
||||
for the phase that allocates LAST**: reclaiming it requires paying elsewhere
|
||||
(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
|
||||
bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
|
||||
crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
|
||||
|
||||
## 5. Boot ≠ survival (knife-edge shapes)
|
||||
|
||||
A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
|
||||
capture, `/health` 200 — and die on the first real request. Measured
|
||||
2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
|
||||
killed seconds later by a 995-token prompt with `max_tokens=261149`:
|
||||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
|
||||
any knife-edge change: send a LARGE-`max_tokens` request before calling it
|
||||
done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
||||
— the survivor had ~3.6 GiB free; the corpse had 0.
|
||||
|
||||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
||||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
||||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
||||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
||||
planner's KV reserve is for ONE max-model-len request.
|
||||
|
||||
**Parallelism ceilings you cannot flag past:**
|
||||
|
||||
- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
|
||||
*"requires TP (2) to be greater than the model's total number of KV heads
|
||||
(2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
|
||||
across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
|
||||
needs TP ≥ 4 = four GPUs.
|
||||
- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
|
||||
the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
|
||||
- DP requires a full model copy per replica — on a model larger than the
|
||||
card it is impossible, not slow.
|
||||
|
||||
## 6. The context ladder (find the max bootable window by measurement)
|
||||
|
||||
"Max possible" = the largest value that BOOTS, measured — never the model's
|
||||
spec maximum, never hand arithmetic.
|
||||
|
||||
1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
|
||||
in lockstep with MAXLEN or the rung tests nothing:
|
||||
`KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
|
||||
off a booted reference rung. Verify the rung's capacity still covers
|
||||
`seqs × MAXLEN` from the engine's own line.
|
||||
2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
|
||||
shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
|
||||
Transient unit:
|
||||
`sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
|
||||
pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
|
||||
3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
|
||||
/ 000 not bound); break early when the unit goes inactive — that is the
|
||||
failure signal, not a timeout.
|
||||
4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
|
||||
with `podman rm`; a `podman logs` issued too soon after `podman run -d`
|
||||
fails with *no container with that name* — record UNKNOWN, never FAIL).
|
||||
5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
|
||||
box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
|
||||
the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
|
||||
0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
|
||||
`Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
|
||||
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
|
||||
orphans by explicit PID
|
||||
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
|
||||
wait for `MemAvailable` to recover.
|
||||
|
||||
**The hazard that cost production:** a leftover ladder script whose teardown
|
||||
`podman rm -f`s the production container fires against a freshly DEPLOYED
|
||||
engine. Teardown runs ONLY on failure, never after a pass; kill the whole
|
||||
scratch process tree and verify none remain BEFORE deploying. On PASS: print
|
||||
the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
|
||||
SERVING, and pin the number into the repo (env + seam + harness
|
||||
contextWindow) before deploying.
|
||||
|
||||
Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
||||
→ tests → commit+push → verify no scratch scripts remain → stash foreign work
|
||||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||||
+ engine flags in the log.
|
||||
|
||||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
||||
|
||||
| shape | util | graphs | arena | result |
|
||||
|---|---|---|---|---|
|
||||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
||||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
||||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
||||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
||||
|
||||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
||||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||||
~207k per context max.
|
||||
|
||||
## 7. Arena audit procedure (when asked for "more experts")
|
||||
|
||||
1. Read the live plan before proposing anything — `grep -oE` with a
|
||||
digit-anchored tail (plain grep/journalctl truncate these columns to
|
||||
`17.`):
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
2. Classify each term (§1 table); take only the two reachable levers.
|
||||
3. Verify after deploy: plan lines show new budget/graphs/arena;
|
||||
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
|
||||
`rocm-smi` high-water per card = the real margin;
|
||||
`systemctl show frank-primary -p NRestarts --value` = 0; a real chat
|
||||
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
|
||||
`localhost` inside an ssh session on the host — the latter returns `000`
|
||||
and looks like an outage).
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 Container Serving]] — the bring-up path and container traps
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
|
||||
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
|
||||
Reference in new issue
Block a user