serverAppConfigs/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	RDNA4 Container Serving	Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.	rdna4 gfx1201 r9700 vllm expert-offload podman rocm	2026-10-05T00:00:00-07:00

RDNA4 Container Serving

Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB RAM, 7700X) with docker.io/tcclaviger/vllm (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build with native MoE quant kernels and a first-class expert offload planner). The tuning history and current deployed shape live in dankserver-qwen38-fp8-expert-offload-tuning; this page is the bring-up law.

0. Know the box — and its phantom third GPU

torch.cuda.device_count() returns 3; summing total_memory overstates the box as 186.7 GiB. The third device is the 7700X's integrated Vega (gfx1036): torch reports ~123 GiB for it, rocm-smi reports 0.50 GiB, allocating on it moves system MemAvailable 1:1 (it is GTT over DDR5, not dedicated VRAM), and it dies with hipErrorInvalidKernelFile when the image ships no code for that arch. Never sum device total_memory; never count the iGPU as headroom or a workload.

1. Preflight: is the quant class EXECUTABLE, not merely present?

Expert offload is mandatory when weights exceed VRAM, and it requires the checkpoint's experts class to have BOTH a compiled kernel AND an offload_recipe ("a pinned expert has no host copy, so there is no read-through fallback" — startup aborts otherwise). Check BEFORE downloading anything:

podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
  nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
  grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'

quant	kernel in r4d.so	experts class	verdict
FP8	yes	Fp8HipMoEExperts	serves
MXFP4 / DSFP4	yes (gemm_moe_mxfp4a8_* / dsfp4a8_*)	R4dMxfp4MoEExperts / R4dDsfp4MoEExperts	serves
W4A16	no gemm_moe_w4a16_*	R4dW4a16MoEExperts exists	silently falls back to generic TritonWNA16Experts, no recipe → startup halts
RFA	none	no class at all	out

Also confirm the architecture routes natively with NO overlay (ModelRegistry.get_supported_archs() for Qwen4Exp/Glm5). The stock image natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. Do not apply vendor overlays against a fast-moving pre-release image — the breakage misleads (a missing symbol in the overlay's OWN copy of a module masks the real cause, then moves to another symbol each patch; cannot import name 'checkpoint_has_lm_head' = overlay clobbered transformers_utils/config.py). The image ships quantizers for rfa|rfi|mxfp4|mxfp4ocp|paro5 under /app/tools/quantize.py (--fp8-attn --quant-only --dry-run first) if a re-quant is needed — note it can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.

2. Fetch with verification, always

Gated repos: token never in argv (pgrep -a curl prints command lines). Build a mode-600 curl config once:

{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
  > ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"

--fail is MANDATORY — curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body as a "successful artifact" (a gated repo once yielded three 153-byte shards with identical sha256s and a cheerful DONE banner). -C - resumes torn shards; HTTP/2 cancels leave silently truncated files, so re-verify EVERY shard's byte count against the Hub manifest (and the LFS sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene law; frank-medium-model-swap has the same detail for GGUFs and .partNofM fragments.

3. Serve — the known-good invocation and its traps

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
  -v /home/dank/models/<model>:/models:ro \
  $IMAGE /models --served-model-name <name> --port 7770 \
  --tensor-parallel-size 2 --max-model-len 65536 \
  --kv-cache-memory-bytes 2147483648 \
  --max-num-seqs 4 --gpu-memory-utilization 0.95 \
  --enable-prefix-caching --trust-remote-code

• Entrypoint already execs vllm serve "$@" — pass the model path as the first argument and do NOT pass serve (podman then tries to pull serve as an image name). -e VAR=1 must come BEFORE the image name.
• Mount the whole /dev/dri, never individual render nodes (§4).
• --ipc=host and --shm-size are mutually exclusive in podman.
• podman run -d without -a sends container stdout nowhere — read podman logs <ctr>; and vLLM writes Application startup complete to stderr, so a podman logs X 2>/dev/null | grep -q readiness gate never fires.
• Start at 65536, not the native ceiling; prove the plan, then scale.
• Load takes ~5 min with /health refusing the whole time — do not kill the process (deploy.sh waits 20 min).

4. Container GPU-init triage (ordered)

1. Enumerate-vs-context trap (most common): individual renderD* nodes give HIP the render nodes WITHOUT the primary card* nodes — it enumerates agents (device_count() returns N) and then refuses every context: RuntimeError: No CUDA GPUs are available on the first real op. A/B-confirmed 2026-09-19. Rule: --device /dev/kfd --device /dev/dri --device /dev/mem; restrict GPUs with HIP_VISIBLE_DEVICES, never with device nodes.
2. device_count() is not evidence. Prove usability with a real op ((a@a).sum().item() on a fresh torch.randn), then repeat via mp.spawn, mp.fork, and subprocess.run([sys.executable, ...]), and cross-check rocm-smi --showmeminfo vram.
3. memlock rlimit → a MISLEADING CUDA OOM. Host default 8192 KB; a pin larger than the limit fails as torch.AcceleratorError: CUDA error: out of memory raised from pin_memory() — it names the accelerator but the fault is host pinned memory. It cannot be raised from inside a container or even from root's own shell (Operation not permitted); only PAM applies it at login, so fixes only affect NEW sessions: /etc/security/limits.d/99-memlock.conf with * - memlock unlimited + root - memlock unlimited; the container needs --ulimit memlock=-1:-1 to inherit. Discriminating test after the fix: the container reports unlimited AND small pins succeed; if pins still fail, memlock was never the cause.
4. /proc/meminfo TAB trap: the key/value separator is a TAB, so grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 returns EMPTY and a following awk prints 0.0 — a false "host out of memory" on a host with hundreds of GiB free, which invites aborting something healthy. Tab-safe: awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576; exit}' /proc/meminfo, cross-check <= MemTotal, refuse to print a zero. Use scripts/dankstatus.sh instead of hand-rolling.
5. Silent-parse traps: grep -c PATTERN file || echo 0 yields TWO lines (grep prints 0 AND exits 1) — [: 0\n0: integer expected; use | head -1 or || true. systemctl is-enabled can emit multiple lines. rocm-smi --showmeminfo vram --csv column order is device, VRAM Total Memory (B), VRAM Total Used Memory (B) — reading $3 as total/$4 as used yields 0.00; filter to ^card[0-9]+$.
6. Guard the box: long loads with co-tenants can drive RAM to 0 — abort before the OOM killer (MemAvailable < 4 GiB → podman rm -f, exit 1).
7. Drop page cache before a large load (echo 3 > /proc/sys/vm/drop_caches) and confirm the drop was actually in effect for the run before treating a failure as a capacity wall — a stale "it doesn't fit" from a contaminated run is expensive to unwind.

5. Memory budgeting: the levers are not where they look

The planner builds budget = total × util, subtracts a KV reserve, and gives the REMAINDER to the expert arena. Consequences, all measured:

• --gpu-memory-utilization grows the ARENA, not the KV pool (KV stayed 0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
• Lowering --max-model-len is the WRONG lever and backfires: a smaller context shrinks the reserve, handing the space to the arena — measured 65536 → 0.64 GiB KV, 32768 → 0.34 GiB KV.
• Hybrid-attention undercount: the planner's per-token KV figure (9190 B/token for Qwen4Exp) misses the large fixed per-sequence Mamba state (~2x), so the reserve is always short and the arena silently covers it. Fix by pinning the pool: --kv-cache-memory-bytes <BYTES> (2 GiB → 113,595 tokens / 3.47x concurrency at 32k). For long context, --kv-offloading-size (GiB) spills KV to host.
• --expert-offload-mem caps the HOST side, not the VRAM arena (cap = int(x * GIB) against host_memory_available()). Do not reach for it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise util (see dankserver-qwen38-fp8-expert-offload-tuning for the util=1.0 unbootability law).

Read the plan — it is the real acceptance test:

expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
  budget (total x util)      30.27 GiB
  non-expert weights          6.28 GiB
  KV reserve                  0.36 GiB   <- sanity-check against the model's own per-token figure
  CUDA graphs                 0.66 GiB
  runtime overhead            7.60 GiB
  expert cache space         15.36 GiB   6712 slots of 2.344 MiB
  host-backed experts/rank   52.74 GiB   23040 experts
  host: 105.48 GiB across 2 ranks of 160.89 GiB usable

Host fits when host_total < usable. The PLE n-gram table rides memfd and is SHARED across TP ranks.

Loading mechanism (for reasoning about peak RAM): weights never materialise in VRAM — expert Parameters start as CPU staging tensors; each layer's weight_loader counts shard events and, on layer completion, ExpertTopology.split_layer moves that layer's experts into the VRAM arena and compacts the rest into exact-size pinned host tensors behind UVA. Peak host RAM = host-backed set + layers still loading, not the whole model. At runtime apply_hot_cold sends arena-resident experts down a hot GEMM and the rest straight over PCIe via UVA views — no staging copy in the critical path.

Emulation check: a dequantize-to-compute path doubles real host RAM (118 GiB offload request → ~238 GiB) and is invisible in the weight count. Read the log for Using 'EMULATION' Mxfp4 MoE backend before concluding "doesn't fit".

6. Performance profile and honest measurement

Decode on this pair is PCIe-gather-bound, context-independent: 12.11 tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost. With ~29 % of experts arena-resident, ~0.84 GB crosses PCIe per token at the engine's effective read rate; cold-vs-warm differing ~10 % confirms the LRU works; growth is linear in hit rate, and only speculative decode cuts sequential STEPS rather than making each cheaper.

Benchmark traps that produced fake numbers here:

• max_tokens=1 is not TTFT (the server can satisfy it without doing the work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every length). Correct TTFT: stream and time the FIRST content-bearing chunk.
• A small repeating word loop is prefix-cacheable AND n-gram-predictable — a 43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use a large vocabulary in shuffled order and print cached_tokens.
• ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare prefill at 30k+ only.
• Size prompts with /v1/messages/count_tokens (a chat call for counting pollutes the metric); guessing words-per-token silently overran the window (HTTP 400).
• Always run a canary: plant a unique token at the END of a long prompt and ask the model to quote it — proves prompts reach the model at all.
• Measure warm AND long (podman exec + urllib at 127.0.0.1:7770): a 33-token reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two concurrent streams scaled to 47 aggregate.

Levers (all opt-in): --expert-prefetch N takes an INT depth, not a boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache already works); CLAV_TUNABLEOP_SWEEP=1 autotunes GEMM shapes (silently skipped under expert offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1 too — see dankserver-qwen38-fp8-expert-offload-tuning for why it was rejected); --expert-precise-memory swaps the fitted overhead figure in.

Related

• dankserver-qwen38-fp8-expert-offload-tuning — deployed shape, plan law, measured campaigns
• PCIe Link and Atomics — why decode is gather-bound and what cannot fix it
• RDNA4 MoE Expert Offload VRAM Budget — arena/plan arithmetic deep-dive
• frank-large-benchmark and frank-small-benchmark — window discipline on live hosts