frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics

The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.

Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).

Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.

Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.

Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.

AGENTS.md

@@ -23,7 +23,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 |
+| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind a round-robin proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 |
 | Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
 
 Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
@@ -248,23 +248,61 @@ Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
 Rollback: [[halogen-flash-server]] (quadlet archived in
 `~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
 
-### frank-medium — ROCmFPX llama.cpp (Dankserver)
+### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)
 
-charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for gfx1201 with the fork's
-self-contained ROCm 7.14 toolchain (RPATH-bundled — host ROCm packages never
-participate). Serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF`
-(ftype 115; stock llama.cpp cannot load it) with `--split-mode layer` across
-both R9700s, 131072 ctx / 1 slot, f16 KV, flash-attn on, MTP draft head
-pinned to ROCm1, reasoning on. Env `scripts/frank-medium.env`, serve
-`scripts/serve-frank-medium.sh`.
+`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4`
+patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its
+patches are applied at every container start), serving
+`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the
+`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`).
 
-Runbook: `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md`.
+The deployment is **two single-GPU replicas behind a round-robin proxy** —
+`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and
+`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets
+under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at
+114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`).
+
+**TP=2 across both cards cannot run on this host.** RCCL's collective kernels
+carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on
+a device without PCIe atomics; the second R9700 sits behind the chipset
+bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a
+read-only capability, so no software can enable atomics on that path. Not the
+kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL
+build (a from-source rebuild fails identically). A slot on a CPU root port —
+there is none free on this AM5 platform for a second x16 card — or a platform
+with more CPU lanes restores TP=2. `frank-medium.container` stays in the repo
+as the TP=2 target; the deploy handler does not install it.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
+Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day).
 
 ## Key Lessons Learned
-
 Durable operational lessons from the fleet's history. Full detail in the
 linked runbooks.
 
+**PCIe / multi-GPU**
+
+- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
+  kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
+  them when the device has no atomics, and the failure surfaces as
+  `hipErrorIllegalState` ("the operation cannot be performed in the present
+  state") from `hipLaunchKernel`, not as anything that names atomics. The
+  value is consumed at driver probe time, so writing the AtomicOp Requester
+  bit with `setpci` at runtime changes nothing. Check the *chain*:
+  `lspci -vv -s <gpu>` and every upstream bridge must show
+  `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
+  endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
+  6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
+  and the rest behind the chipset (usually not) — that asymmetry is what
+  blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
+  short-circuits at `world_size=1`.
+- **When a GPU collective dies, read the runtime, not the framework.** The
+  message that identified the cause came from `AMD_LOG_LEVEL=3`
+  (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
+  failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
+  prints the failing HIP call with its return code.
+
+
 **ROCm / GPU**
 
 - **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor
@@ -359,7 +397,7 @@ Active:
 | Service | Runbook |
 |---------|---------|
 | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
-| frank-medium (ROCmFPX, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` |
+| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |