serverAppConfigs/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

---
type: reference
title: RDNA4 Container Serving
description: "Bringing a model larger than VRAM onto the two-card box: quant gating, fetch verification, container GPU-init triage."
tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
timestamp: 2026-10-05T00:00:00-07:00
---

# RDNA4 Container Serving

Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201 — card count,
per-card VRAM and host RAM are [[Dankserver System Specs]]'s) with
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
with native MoE quant kernels and a first-class expert offload planner). The
tuning history and current deployed shape live in
[[frank-archon]]; this page is the bring-up law.

## 0. Know the box — and its phantom third GPU

`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
the box as 186.7 GiB. The third device is the 7700X's integrated Vega
(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
ships no code for that arch. **Never sum device `total_memory`; never count
the iGPU as headroom or a workload.**

## 1. Preflight: is the quant class EXECUTABLE, not merely present?

Expert offload is mandatory when weights exceed VRAM, and it requires the
checkpoint's experts class to have BOTH a compiled kernel AND an
`offload_recipe` ("a pinned expert has no host copy, so there is no
read-through fallback" — startup aborts otherwise). Check BEFORE downloading
anything:

```bash
podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
  nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
  grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
```

| quant | kernel in `r4d.so` | experts class | verdict |
|---|---|---|---|
| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
| RFA | none | no class at all | out |

Also confirm the architecture routes natively with NO overlay
(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
apply vendor overlays** against a fast-moving pre-release image — the breakage
misleads (a missing symbol in the overlay's OWN copy of a module masks the
real cause, then moves to another symbol each patch;
`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
`transformers_utils/config.py`). The image ships quantizers for
`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.

## 2. Fetch with verification, always

Gated repos: token never in argv (`pgrep -a curl` prints command lines).
Build a mode-600 curl config once:

```bash
{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
  > ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
```

`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
the error body as a "successful artifact" (a gated repo once yielded three
153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
resumes torn shards; HTTP/2 cancels leave silently truncated files, so
re-verify EVERY shard's byte count against the Hub manifest (and the LFS
sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
law; [[frank-medium-model-swap]] has the same detail for GGUFs and
`.partNofM` fragments.

## 3. Serve — the known-good invocation and its traps

```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
  -v /home/dank/models/<model>:/models:ro \
  $IMAGE /models --served-model-name <name> --port 7770 \
  --tensor-parallel-size 2 --max-model-len 65536 \
  --kv-cache-memory-bytes 2147483648 \
  --max-num-seqs 4 --gpu-memory-utilization 0.95 \
  --enable-prefix-caching --trust-remote-code
```

- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
  first argument and do NOT pass `serve` (podman then tries to pull `serve`
  as an image name). `-e VAR=1` must come BEFORE the image name.
- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
- `podman run -d` without `-a` sends container stdout nowhere — read
  `podman logs <ctr>`; and vLLM writes `Application startup complete` to
  **stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
  fires.
- Start at 65536, not the native ceiling; prove the plan, then scale.
- Load takes ~5 min with `/health` refusing the whole time — do not kill the
  process (deploy.sh waits 20 min).

## 4. Container GPU-init triage (ordered)

1. **Enumerate-vs-context trap (most common):** individual
   `renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
   nodes — it enumerates agents (`device_count()` returns N) and then refuses
   every context: `RuntimeError: No CUDA GPUs are available` on the first
   real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
   /dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
   never with device nodes.
2. **`device_count()` is not evidence.** Prove usability with a real op
   (`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
   `mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
   cross-check `rocm-smi --showmeminfo vram`.
3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
   larger than the limit fails as
   `torch.AcceleratorError: CUDA error: out of memory` raised from
   `pin_memory()` — it names the accelerator but the fault is host pinned
   memory. It cannot be raised from inside a container or even from root's
   own shell (`Operation not permitted`); only PAM applies it at login, so
   fixes only affect NEW sessions:
   `/etc/security/limits.d/99-memlock.conf` with
   `* - memlock unlimited` + `root - memlock unlimited`; the container needs
   `--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
   the container reports `unlimited` AND small pins succeed; if pins still
   fail, memlock was never the cause.
4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
   `grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
   EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
   host with hundreds of GiB free, which invites aborting something healthy.
   Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
   exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
   Use `scripts/dankstatus.sh` instead of hand-rolling.
5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
   (grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
   `| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
   `rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
   Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
   yields 0.00; filter to `^card[0-9]+$`.
6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
   before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
   and confirm the drop was actually in effect for the run before treating a
   failure as a capacity wall — a stale "it doesn't fit" from a contaminated
   run is expensive to unwind.

## 5. Memory budgeting: the levers are not where they look

Every figure in this section is a tcclaviger-era boot on the port that is
[[frank-archon]]'s today; the laws they teach are general. The planner builds
`budget = total × util`, subtracts a KV reserve, and gives
the REMAINDER to the expert arena. Consequences, all measured:

- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
  0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
  context shrinks the reserve, handing the space to the arena — measured 65536
  → 0.64 GiB KV, 32768 → 0.34 GiB KV.
- **Hybrid-attention undercount:** the planner's per-token KV figure for
  Qwen4Exp is an undercount — the printed figure and the true one are both on
  [[frank-archon]] — because it misses the large fixed per-sequence Mamba state
  (~2x), so the reserve is always short and the arena silently covers it.
  Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (era reading: a
  2 GiB pin = 113,595 tokens ≈ 3.47x concurrency at 32k — that pair is this
  page's record of the tcclaviger boots, not [[frank-archon]]'s). For long
  context,
  `--kv-offloading-size` (GiB) spills KV to host.
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
  (`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
  it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
  util (see [[frank-archon]] for the
  util=1.0 unbootability law).

**Read the plan — it is the real acceptance test:**

```
expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
  budget (total x util)      30.27 GiB
  non-expert weights          6.28 GiB
  KV reserve                  0.36 GiB   <- sanity-check against the model's own per-token figure
  CUDA graphs                 0.66 GiB
  runtime overhead            7.60 GiB
  expert cache space         15.36 GiB   6712 slots of 2.344 MiB
  host-backed experts/rank   52.74 GiB   23040 experts
  host: 105.48 GiB across 2 ranks of 160.89 GiB usable
```

Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
SHARED across TP ranks.

**Loading mechanism** (for reasoning about peak RAM): weights never materialise
in VRAM — expert Parameters start as CPU staging tensors; each layer's
`weight_loader` counts shard events and, on layer completion,
`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
and compacts the rest into exact-size pinned host tensors behind UVA. Peak
host RAM = `host-backed set + layers still loading`, not the whole model. At
runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
rest straight over PCIe via UVA views — no staging copy in the critical path.

**Emulation check:** a dequantize-to-compute path doubles real host RAM — the
measured pair (offload request vs host footprint) is on [[frank-archon]] — and
the doubling is invisible in the weight count. Read the log for `Using
'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".

## 6. Performance profile and honest measurement

Decode on this pair is **PCIe-gather-bound, context-independent**: the rate is
flat from a few hundred to sixteen thousand context tokens, so KV attention is
not the cost (the flat figure and its MTP/gather deltas are
[[Tcclaviger Fork Ops]]'s profile of the same shape). With ~29 % of experts
arena-resident, under a gigabyte crosses PCIe per token at the engine's
effective read rate; cold-vs-warm differing ~10 % confirms the LRU works;
growth is linear in hit rate, and only speculative decode cuts sequential STEPS
rather than making each cheaper.

**Benchmark traps that produced fake numbers here:**

- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
  work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
  non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
  length). Correct TTFT: stream and time the FIRST content-bearing chunk.
- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
  43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
  a large vocabulary in shuffled order and print `cached_tokens`.
- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
  prefill at 30k+ only.
- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
  pollutes the metric); guessing words-per-token silently overran the window
  (HTTP 400).
- Always run a canary: plant a unique token at the END of a long prompt and
  ask the model to quote it — proves prompts reach the model at all.
- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
  reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
  concurrent streams scaled to 47 aggregate.

**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
see [[frank-archon]] for why it was rejected);
`--expert-precise-memory` swaps the fitted overhead figure in.

## Related

- [[frank-archon]] — deployed shape, plan law,
  measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-dragoon]] and [[frank-zealot]] — window discipline on live hosts