serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-medium vllm-radiance MXFP4 Runbook	vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind an adaptive-routing proxy — and the PCIe-atomics reason TP=2 cannot run on this host.	frank-medium dankserver vllm vllm-radiance mxfp4 gfx1201 r9700 rccl pcie-atomics quadlet	2026-09-16

frank-medium vllm-radiance MXFP4 Runbook

The ACTIVE frank-medium deployment (since 2026-09-15/16): the stilldeadcode/vllm-radiance:0.9.3 image plus the ggz14/radiance-vllm-mxfp4 patches, serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the two Radeon AI PRO R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp deployment (frank-medium-rocmfpx).

Deployed shape: two single-GPU replicas behind an adaptive-routing proxy. The intended shape — one tensor-parallel process across both cards (TP=2) — cannot launch a collective on this host. The reason is a hardware property of the motherboard's PCIe topology, not the image, the RCCL build, or the kernel version. The evidence is in Why TP=2 cannot run here.

Architecture

graph LR
    C[clients: Hermes, harness] --> P["frank-medium-proxy<br/>:7770 adaptive routing"]
    P --> A["frank-medium-a<br/>R9700 #1, TP=1, :7771"]
    P --> B["frank-medium-b<br/>R9700 #2, TP=1, :7772"]

Three rootless Quadlet units, all WantedBy=default.target + Restart=on-failure:

Unit	Container	GPU	Port	Role
frank-medium-a	frank-medium-a	HIP_VISIBLE_DEVICES=0	7771	replica
frank-medium-b	frank-medium-b	HIP_VISIBLE_DEVICES=1	7772	replica
frank-medium-proxy	frank-medium-proxy	—	7770	routing

The fleet contract is unchanged: one endpoint on :7770, served model name frank-medium, OpenAI /v1, streaming included. The vLLM OpenAI API is stateless per request, so no session affinity is needed for correctness. Routing is described in Routing.

Only HIP_VISIBLE_DEVICES is set per replica, never a matching ROCR_VISIBLE_DEVICES: ROCR filters first and HIP then indexes into the filtered list, so the single-valued pair (ROCR=1,HIP=1) leaves an empty device set and the engine dies with hipErrorNoDevice.

frank-medium.container (TP=2, KV_MEM=18563072000, MAXLEN=262144) stays in the repo as the target shape for a host whose second card sits on an atomic-capable root port. deploy-remote.sh deliberately does not install it and moves any stale copy into ~/.AGENTTRASH/, because a TP=2 unit on this host would restart forever without ever becoming healthy.

Why TP=2 cannot run here

Symptom

Any two-rank collective dies at launch:

[FATAL ERROR]: HIP failure: 'the operation cannot be performed in the present state'
  group.cc:337 -> group.cc:656 -> group.cc:801 -> enqueue.cc:3111

With the runtime's own API log (AMD_LOG_LEVEL=3) the failing call is explicit:

ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>)
:1:rocvirtual.cpp :4208:  Pcie atomics not enabled, hostcall not supported
:1:rocvirtual.cpp :4636:  AQL dispatch failed!
:3:hip_module.cpp :893 :  hipLaunchKernel: Returned hipErrorIllegalState

ROCclr refuses to hand the kernel a hostcall buffer when the device has no PCIe atomics (rocclr/device/rocm/rocvirtual.cpp:3963), and the launch aborts. hipErrorIllegalState is that refusal, not a driver fault.

Chain of causation

1. RCCL's collective kernels carry a hidden hostcall buffer parameter.
2. ROCr passes that buffer only when pcie_atomics_ is true for the device; otherwise it returns "not supported" and the dispatch fails.
3. pcie_atomics_ is set once, at device init, from the HSA link info between the CPU agent and the GPU (rocclr/device/rocm/rocdevice.cpp:186, Device::checkAtomicSupport). It is not re-read per process.
4. The KFD driver populates that link info from pci_enable_atomic_ops_to_root(), which — per the PCIe spec and the kernel's implementation — requires every upstream bridge between the device and its root port to advertise AtomicOp Routing capability (drivers/pci/pci.c).

The hardware difference between the two cards

Card	PCIe chain (endpoint → root port)	AtomicOpsCap on every hop	AtomicOpsCtl
R9700 #1 (03:00.0)	02:00.0 → 01:00.0 → 00:01.1	Routing+ / Routing+ / Routing+	ReqEn+ (atomics ON)
R9700 #2 (0b:00.0)	0a:00.0 → 09:00.0 → 06:09.0 → 05:00.0 → 00:02.1	Routing+ / Routing+ / Routing− / Routing− / Routing+	ReqEn− (atomics OFF)

Card #1 is on a CPU root port; card #2 is behind the chipset bridges 05:00.0/06:09.0, which advertise AtomicOpsCap: Routing-. That capability bit is read-only — no software can enable atomic routing through those bridges — so the driver's atomic enablement call fails for card #2 and the KFD marks the CPU↔GPU2 link NO_ATOMICS.

Writing the AtomicOp Requester bit into the endpoint's Device Control 2 by hand (setpci -s 0b:00.0 CAP_EXP+28.w) does not help: the driver reads the chain at probe time, and ROCr consumes the driver's cached answer. Only a re-probe (a boot with the write already in place) would change it — and even then the two Routing- bridges would still make pci_enable_atomic_ops_to_root() fail.

What was ruled out

Suspect	Result
Kernel version	7.0.0-28, -30 and -31 fail identically (full apt full-upgrade to the newest available kernel changed nothing)
IOMMU	identical before and after iommu=pt
P2P hardware	fine: 45.3 GB/s bidirectional copy between the two cards; both cards pass rocm-bandwidth-test
The shipped librccl	a from-source rebuild of ROCm/rccl develop (with local fixes for that branch's dangling references) reproduces the same failure at the same call path — the library is not at fault
vLLM/pynccl	a bare two-rank torch.distributed allreduce with no vLLM involved fails the same way; single-rank collectives succeed on both cards (NCCL short-circuits at world_size=1, which is why a one-card test proves nothing)
Transport knobs	NCCL_PROTO=Simple, NCCL_P2P_DISABLE, NCCL_SHM_DISABLE, NCCL_CUMEM_ENABLE=0, MSCCL off, channel counts, launch modes, HSA_ENABLE_SDMA=0 — all identical
Image/support matrix	the same image on a different host with the same kernel (Proxmox VM, Codeberg issue #22) fails byte-identically

What would fix it

• Move card #2 to a slot whose path is atomic-capable (a CPU root port, as card #1 has). This is the only software-free fix and the one that restores true TP=2. On this AM5 platform the CPU provides the x16 slot plus x4 lanes, so a second x16-class card necessarily lands on the chipset path — a platform with more CPU lanes (Threadripper/EPYC) is the structural answer.
• Until then, two replicas are how both cards serve.

Routing

The proxy decides per request, from proxy/routing.py (policy) and proxy/affinity-proxy.py (HTTP plumbing):

• A continuation — a request carrying more than one message, i.e. an agent loop re-sending its context — is pinned to the replica that served the prefix before, because --enable-prefix-caching has that prefix warm there.
• Anything else — a one-shot question, a batch member, a burst of parallel subagents — spreads across the cards by counter, so both prefill in parallel.

Measured on this host, 90k-token contexts and 15k-token prompts, vllm bench plus a small sequential-turn client:

strategy	cold 16-request burst	warm-turn TTFT (4 turns × 2 families)
roundrobin (counter)	18.6 s	2676 ms — every other turn pays a cold prefill
affinity (hash of prefix)	30.1 s — one card does all the prefilling	1096 ms
adaptive (deployed)	19.97 s	1095 ms

Two failures found by measuring rather than reasoning, both now covered by tests/frank_medium_proxy.bats:

• Hashing a fresh prefix to pick a card sends every member of a same-prefix burst to one card (29.8 s for the burst above). Only continuations may be pinned; the spread has to come from a counter.
• The identity of a conversation is its first 4 KiB (AFFINITY_BYTES). A prefix shorter than that window is indistinguishable from its own tail, so turns of the same conversation hash differently and never pin.

ROUTING=roundrobin and ROUTING=affinity remain available to re-measure the choice; the default is adaptive.

Deployed configuration

Both replicas, uniform except GPU index and port:

Knob	Value	Why
TP	1	one card per replica
MAXLEN	114688	the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.14 available, and 131072 needs 5.58 against 5.22
KV_MEM	0	passed through literally as --kv-cache-memory 0, which the image's patched KV sizing reads as "profile this shape"; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here. Do not blank the variable instead of passing 0: with the flag omitted vLLM sizes the cache from GPU_UTIL alone and the KV allocation dies with torch.OutOfMemoryError (measured: 2.16 GiB requested, 116 MiB free, 29.34 GiB already held)
MAXSEQS	4	matches the per-replica share of the fleet's concurrency
CHUNK	8192	prefill chunk (upstream default), coupled to the context: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: CHUNK=16384 leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)). Raising CHUNK therefore means cutting MAXLEN; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking
GPU_UTIL	0.98	upstream default; the card is dedicated
SPEC_METHOD / SPEC	dflash / 7	the DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see Speculative depth); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive
SNAP / DRAFTER	/models/Qwen3.8-27B-Uncensored-MXFP4-awq / /models/Qwen3.8-27B-DFlash2-FP8	the served checkpoint and its drafter

Mounts are the TP=2 unit's minus nothing: /models, the HF cache, the shared compile cache, /patches (the pinned radiance clone; its patches are applied at every container start), /r4d (the patched libr4d — the image's shipped r4d.so predates the gated-delta-net overflow fix and NaNs this model), and the entrypoint/healthcheck/proxy scripts mounted read-only out of /home/dank/serverAppConfigs.

The healthcheck reads PORT, so one probe serves the replicas (:7771/:7772), the proxy (:7770) and the TP=2 target.

Deploy and verify

./tests/run.sh                       # unit/config seams
./scripts/deploy.sh dankserver --service frank-medium

The handler re-tags docker.io/stilldeadcode/vllm-radiance:0.9.3 as localhost/radiance-vllm:0.9.3 (this host's quadlet path cannot resolve registry-prefixed refs from local storage at create time), installs the three units, retires any stale TP=2 unit into the trash, restarts the trio and waits for http://127.0.0.1:7770/health.

Verification that means something:

curl -s http://192.168.0.235:7770/health        # proxy: names any replica that is down
curl -s http://192.168.0.235:7770/v1/models     # served name must be frank-medium
# and a real completion through the proxy:
curl -s http://192.168.0.235:7770/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"frank-medium","messages":[{"role":"user","content":"Say OK"}],"max_tokens":400}'

The same call works against either replica directly (:7771, :7772) — when one of them is down the other still serves, which is why the proxy reports the individual replica states instead of a single boolean.

A replica that dies mid-flight does not cost a client a request: a failed connection demotes that replica immediately and the proxy retries on the live one (vLLM's endpoints are stateless, so re-sending a request that never reached an engine is safe) and logs the failover. Verify with card 1 stopped:

systemctl --user stop frank-medium-a.service
for i in 1 2 3 4 5 6; do
  curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
    -d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
    http://127.0.0.1:7770/v1/chat/completions
done; echo
podman logs --since 60s frank-medium-proxy | grep -i failover
systemctl --user start frank-medium-a.service

Six 200s and one proxy: failover — … unreachable line is the pass condition. Restarting a stopped replica takes about 150 s (warm compile cache).

Benchmarks

vllm bench serve against the proxy (both cards serving), random dataset, --ignore-eos, 2026-09-16:

Case	Output t/s	TTFT p50	TPOT p50	Acceptance
conc 1, in 1k / out 512	42.8	382 ms	23.6 ms	43.1 %
conc 4, in 1k / out 512	147.6	471 ms	25.3 ms	34.2 %
conc 8, in 1k / out 512	230.8	913 ms	28.8 ms	39.0 %
conc 1, in 32k / out 32	2.5	11837 ms	29.6 ms	25.0 %

The 32k row is a prefill measurement: 32768 prompt tokens in 11.8 s is ≈2.8k tok/s of prefill on a single card. The replicas split the fleet's traffic, they do not pool capacity for one request, against the upstream reference's 4585 t/s at 32k on TP=2 — one card doing the whole prompt lands where you would expect.

Read these against the upstream repo's own reference table (2x R9700, TP=2, BetterBench corpus): step time 22.8-23.2 ms per update, combined decode 224 t/s, prefill 4585 t/s at 32k. Step time (TPOT) is the comparable number — tokens/s swings with draft acceptance, and synthetic random prompts accept far worse than the code/prose mix upstream measures (43 % here against 5.8 tokens/update there), so a lower token rate at the same step time is the dataset, not the deployment.

Bench caveat: the client must be given --tokenizer /models/Qwen3.8-27B-Uncensored-MXFP4-awq with /models mounted. Passing only --model frank-medium makes the client try to resolve that alias on HuggingFace and die before sending a request — a green-looking run that produced no numbers at all.

Speculative depth

Upstream ships depth 7 for dflash and notes the choice is content-dependent; it was measured for TP=2 on their corpus. Re-measured here on one card against a code corpus built from this repository's own sources (16 prompts, 512 output tokens each, two runs per setting):

Depth	conc 1	conc 4
SPEC=7 (deployed)	46.2 tok/s, TPOT 20.8 ms, 3.86 tok/update	118.7 / 127.0 tok/s, TPOT 25.1-29.3 ms
SPEC=5	41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update	136.1 / 129.4 tok/s, TPOT 25.3-26.6 ms

The step time is nearly identical at both depths (≈79 ms against ≈80 ms at conc 1) — depth 7 wins single-stream because it lands more tokens per step (3.86 against 3.41), and depth 5 wins at conc 4 by ≈8 % because more, narrower verify steps pack better. Depth 7 stays: this fleet serves about three people in interactive agent loops, where per-turn latency is the number that is felt, and it is also the better default on code at low concurrency. Switch a replica to SPEC=5 if batch throughput ever matters more than turn latency — it is one environment line in the unit, at the cost of a recompile (the spec depth is part of the graph cache key).

Citations

• ROCm/clr rocclr/device/rocm/rocvirtual.cpp:3963 — the hostcall-buffer refusal; rocclr/device/rocm/rocdevice.cpp:186 — checkAtomicSupport.
• Linux drivers/pci/pci.c — pci_enable_atomic_ops_to_root() and its PCI_EXP_DEVCAP2_ATOMIC_ROUTE requirements on upstream bridges.
• Codeberg StillDeadcode/vllm-radiance issue #22 — an independent host with the same image and kernel 7.0.0-30 failing identically.
• ggz14/radiance-vllm-mxfp4 — the patch set and the reference benchmark table this page compares against.