Frank
a6ad9af708
The parallel session's stash pop left GGUF Compatibility, Multi-Device Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving, vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9 unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link, and two multi-line-wrapped link tokens). All repointed to real pages; six rows added to the vault index; seam green.
251 lines
13 KiB
Markdown
251 lines
13 KiB
Markdown
---
|
||
type: reference
|
||
title: RDNA4 Container Serving
|
||
description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
|
||
tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# RDNA4 Container Serving
|
||
|
||
Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
|
||
**63.7 GiB real VRAM**, 246 GiB RAM, 7700X) with
|
||
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
|
||
with native MoE quant kernels and a first-class expert offload planner). The
|
||
tuning history and current deployed shape live in
|
||
[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
|
||
|
||
## 0. Know the box — and its phantom third GPU
|
||
|
||
`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
|
||
the box as 186.7 GiB. The third device is the 7700X's integrated Vega
|
||
(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
|
||
allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
|
||
dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
|
||
ships no code for that arch. **Never sum device `total_memory`; never count
|
||
the iGPU as headroom or a workload.**
|
||
|
||
## 1. Preflight: is the quant class EXECUTABLE, not merely present?
|
||
|
||
Expert offload is mandatory when weights exceed VRAM, and it requires the
|
||
checkpoint's experts class to have BOTH a compiled kernel AND an
|
||
`offload_recipe` ("a pinned expert has no host copy, so there is no
|
||
read-through fallback" — startup aborts otherwise). Check BEFORE downloading
|
||
anything:
|
||
|
||
```bash
|
||
podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
|
||
-e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
|
||
nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
|
||
grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
|
||
```
|
||
|
||
| quant | kernel in `r4d.so` | experts class | verdict |
|
||
|---|---|---|---|
|
||
| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
|
||
| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
|
||
| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
|
||
| RFA | none | no class at all | out |
|
||
|
||
Also confirm the architecture routes natively with NO overlay
|
||
(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
|
||
natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
|
||
apply vendor overlays** against a fast-moving pre-release image — the breakage
|
||
misleads (a missing symbol in the overlay's OWN copy of a module masks the
|
||
real cause, then moves to another symbol each patch;
|
||
`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
|
||
`transformers_utils/config.py`). The image ships quantizers for
|
||
`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
|
||
(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
|
||
can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
|
||
|
||
## 2. Fetch with verification, always
|
||
|
||
Gated repos: token never in argv (`pgrep -a curl` prints command lines).
|
||
Build a mode-600 curl config once:
|
||
|
||
```bash
|
||
{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
|
||
> ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
|
||
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
|
||
```
|
||
|
||
`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
|
||
the error body as a "successful artifact" (a gated repo once yielded three
|
||
153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
|
||
resumes torn shards; HTTP/2 cancels leave silently truncated files, so
|
||
re-verify EVERY shard's byte count against the Hub manifest (and the LFS
|
||
sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
|
||
law; [[frank-medium-model-swap]] has the same detail for GGUFs and
|
||
`.partNofM` fragments.
|
||
|
||
## 3. Serve — the known-good invocation and its traps
|
||
|
||
```bash
|
||
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
|
||
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
|
||
--cap-add SYS_PTRACE \
|
||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||
-v /home/dank/models/<model>:/models:ro \
|
||
$IMAGE /models --served-model-name <name> --port 7770 \
|
||
--tensor-parallel-size 2 --max-model-len 65536 \
|
||
--kv-cache-memory-bytes 2147483648 \
|
||
--max-num-seqs 4 --gpu-memory-utilization 0.95 \
|
||
--enable-prefix-caching --trust-remote-code
|
||
```
|
||
|
||
- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
|
||
first argument and do NOT pass `serve` (podman then tries to pull `serve`
|
||
as an image name). `-e VAR=1` must come BEFORE the image name.
|
||
- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
|
||
- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
|
||
- `podman run -d` without `-a` sends container stdout nowhere — read
|
||
`podman logs <ctr>`; and vLLM writes `Application startup complete` to
|
||
**stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
|
||
fires.
|
||
- Start at 65536, not the native ceiling; prove the plan, then scale.
|
||
- Load takes ~5 min with `/health` refusing the whole time — do not kill the
|
||
process (deploy.sh waits 20 min).
|
||
|
||
## 4. Container GPU-init triage (ordered)
|
||
|
||
1. **Enumerate-vs-context trap (most common):** individual
|
||
`renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
|
||
nodes — it enumerates agents (`device_count()` returns N) and then refuses
|
||
every context: `RuntimeError: No CUDA GPUs are available` on the first
|
||
real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
|
||
/dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
|
||
never with device nodes.
|
||
2. **`device_count()` is not evidence.** Prove usability with a real op
|
||
(`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
|
||
`mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
|
||
cross-check `rocm-smi --showmeminfo vram`.
|
||
3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
|
||
larger than the limit fails as
|
||
`torch.AcceleratorError: CUDA error: out of memory` raised from
|
||
`pin_memory()` — it names the accelerator but the fault is host pinned
|
||
memory. It cannot be raised from inside a container or even from root's
|
||
own shell (`Operation not permitted`); only PAM applies it at login, so
|
||
fixes only affect NEW sessions:
|
||
`/etc/security/limits.d/99-memlock.conf` with
|
||
`* - memlock unlimited` + `root - memlock unlimited`; the container needs
|
||
`--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
|
||
the container reports `unlimited` AND small pins succeed; if pins still
|
||
fail, memlock was never the cause.
|
||
4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
|
||
`grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
|
||
EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
|
||
host with hundreds of GiB free, which invites aborting something healthy.
|
||
Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
|
||
exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
|
||
Use `scripts/dankstatus.sh` instead of hand-rolling.
|
||
5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
|
||
(grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
|
||
`| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
|
||
`rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
|
||
Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
|
||
yields 0.00; filter to `^card[0-9]+$`.
|
||
6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
|
||
before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
|
||
7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
|
||
and confirm the drop was actually in effect for the run before treating a
|
||
failure as a capacity wall — a stale "it doesn't fit" from a contaminated
|
||
run is expensive to unwind.
|
||
|
||
## 5. Memory budgeting: the levers are not where they look
|
||
|
||
The planner builds `budget = total × util`, subtracts a KV reserve, and gives
|
||
the REMAINDER to the expert arena. Consequences, all measured:
|
||
|
||
- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
|
||
0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
|
||
- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
|
||
context shrinks the reserve, handing the space to the arena — measured 65536
|
||
→ 0.64 GiB KV, 32768 → 0.34 GiB KV.
|
||
- **Hybrid-attention undercount:** the planner's per-token KV figure (9190
|
||
B/token for Qwen4Exp) misses the large fixed per-sequence Mamba state
|
||
(~2x), so the reserve is always short and the arena silently covers it.
|
||
Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (2 GiB →
|
||
113,595 tokens / 3.47x concurrency at 32k). For long context,
|
||
`--kv-offloading-size` (GiB) spills KV to host.
|
||
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
|
||
(`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
|
||
it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
|
||
util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
|
||
util=1.0 unbootability law).
|
||
|
||
**Read the plan — it is the real acceptance test:**
|
||
|
||
```
|
||
expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
|
||
budget (total x util) 30.27 GiB
|
||
non-expert weights 6.28 GiB
|
||
KV reserve 0.36 GiB <- sanity-check against the model's own per-token figure
|
||
CUDA graphs 0.66 GiB
|
||
runtime overhead 7.60 GiB
|
||
expert cache space 15.36 GiB 6712 slots of 2.344 MiB
|
||
host-backed experts/rank 52.74 GiB 23040 experts
|
||
host: 105.48 GiB across 2 ranks of 160.89 GiB usable
|
||
```
|
||
|
||
Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
|
||
SHARED across TP ranks.
|
||
|
||
**Loading mechanism** (for reasoning about peak RAM): weights never materialise
|
||
in VRAM — expert Parameters start as CPU staging tensors; each layer's
|
||
`weight_loader` counts shard events and, on layer completion,
|
||
`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
|
||
and compacts the rest into exact-size pinned host tensors behind UVA. Peak
|
||
host RAM = `host-backed set + layers still loading`, not the whole model. At
|
||
runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
|
||
rest straight over PCIe via UVA views — no staging copy in the critical path.
|
||
|
||
**Emulation check:** a dequantize-to-compute path doubles real host RAM (118
|
||
GiB offload request → ~238 GiB) and is invisible in the weight count. Read the
|
||
log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
|
||
|
||
## 6. Performance profile and honest measurement
|
||
|
||
Decode on this pair is **PCIe-gather-bound, context-independent**: 12.11
|
||
tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost.
|
||
With ~29 % of experts arena-resident, ~0.84 GB crosses PCIe per token at the
|
||
engine's effective read rate; cold-vs-warm differing ~10 % confirms the LRU
|
||
works; growth is linear in hit rate, and only speculative decode cuts
|
||
sequential STEPS rather than making each cheaper.
|
||
|
||
**Benchmark traps that produced fake numbers here:**
|
||
|
||
- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
|
||
work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
|
||
non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
|
||
length). Correct TTFT: stream and time the FIRST content-bearing chunk.
|
||
- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
|
||
43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
|
||
a large vocabulary in shuffled order and print `cached_tokens`.
|
||
- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
|
||
prefill at 30k+ only.
|
||
- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
|
||
pollutes the metric); guessing words-per-token silently overran the window
|
||
(HTTP 400).
|
||
- Always run a canary: plant a unique token at the END of a long prompt and
|
||
ask the model to quote it — proves prompts reach the model at all.
|
||
- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
|
||
reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
|
||
concurrent streams scaled to 47 aggregate.
|
||
|
||
**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
|
||
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
|
||
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
|
||
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
|
||
see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
|
||
`--expert-precise-memory` swaps the fitted overhead figure in.
|
||
|
||
## Related
|
||
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
|
||
measured campaigns
|
||
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
|
||
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
|