serverAppConfigs/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	Dual R9700 Cooperation	Why frank-medium's two Radeon AI PRO R9700s run as independent replicas instead of tensor-parallel, which options are ruled out with evidence, and the two hardware paths that restore true TP=2.	frank-medium rocm rccl pcie-atomics tensor-parallelism dankserver	2026-09-16

Dual R9700 Cooperation

Dankserver's frank-medium runs as two independent single-GPU replicas (:7771 on card #1, :7772 on card #2) behind an adaptive-routing proxy on :7770. This page answers why, and what would let the two cards work on one model instead. See frank-medium-radiance-mxfp4 for the deployment itself.

The short answer

Card #2 sits behind the B650 chipset, whose bridges cannot route PCIe atomic operations. RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to supply it to a device without atomics — so no RCCL collective or P2P operation can be dispatched on card #2. Two paths restore cooperation:

1. Move card #2 to the free, CPU-attached M.2_2 slot (~$30 M.2→PCIe x4 riser). Cheapest, and bandwidth is not a constraint (below).
2. A board whose CPU x16 bifurcates to x8/x8 (X670E/X870E class, ~$250-400). Best performance. This board has no x8/x8 mode.

The M.2_2 half of path 1 rests on a vendor claim, not on this host: ASUS lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as chipset, while an empty slot does not enumerate — so nothing on a running system can confirm it until a card is in it. And CPU-attached does not imply Routing+: 00:08.3 is CPU-side and reads Routing-. The precedent worth trusting is 00:01.2, the CPU M.2_1 port, which is Routing+. Note too what the riser buys: tensor parallelism (~40 % decode), not "both cards on one model" — the layer split already delivers that, so this is a performance project, not a capability one.

The gate, in source

void Device::checkAtomicSupport() {                    // rocclr/device/rocm/rocdevice.cpp:181
  std::vector<amd::Device::LinkAttrType> link_attrs;
  link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0));
  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) {
    if (link_attrs[0].second == 1) {
      info_.pcie_atomics_ = true;                      // no else branch
    }
  }
}

The flag is a queried hardware property, consumed at rocvirtual.cpp:3954 (the hostcall/AQL dispatch path), in rocblit.cpp (SDMA copy paths) and exported as hipDeviceProp.hostNativeAtomicSupported. There is no environment variable, module parameter, or sysfs toggle around it — verified by grepping ROCclr for getenv/HSA_*/ROCR_*/AMD_* in the gate's neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand (setpci -s 0b:00.0 CAP_EXP+28.w=0440) does nothing because the value is consumed at driver probe time.

What has to be true upstream of it: every PCIe bridge on the GPU's path to the CPU must advertise AtomicOpsCap: Routing+. On this board card #2's path includes 05:00.0 and 06:09.0, both Routing- — a read-only capability bit, so no software can enable it.

The same gate, as this board reports it (2026-09-16)

lspci -vv on dankserver. Card 0's chain is root port 00:01.1 → 01:00.0 → 02:00.0 → 03:00.0; card 1's is 00:02.1 → 05:00.0 → 06:00.0 → 06:09.0 → 09:00.0 → 0a:00.0 → 0b:00.0:

Device	Role	AtomicOpsCap	AtomicOpsCtl	Link
00:01.1	CPU root port → PCIEX16_1	Routing+	—	x16
03:00.0	card 0	32bit+ 64bit+	ReqEn+	x16
00:02.1	CPU root port → chipset	Routing+	—	x4
05:00.0, 06:00.0, 06:09.0	B650 bridges	Routing−	—	x1 at 06:09.0
09:00.0, 0a:00.0	PCIe switch ports	Routing+	—	x1 (downgraded)
0b:00.0	card 1	32bit+ 64bit+	ReqEn−	x16

Three things fall out of that table, none of them in the source alone:

• The CPU side is fine — 00:02.1 is Routing+. The wall is inside the chipset, not the platform, which is why a slot on CPU lanes is the fix.
• The endpoint's own bit differs between the cards, and that is the kernel's doing. pci_enable_atomic_ops_to_root() walks the bridge chain and refuses to set Requester Enable when any bridge cannot route atomics. ROCclr's pcie_atomics_ and the endpoint's ReqEn are two faces of one cause, and it makes lspci -vv -s <bdf> | grep AtomicOpsCtl a one-command diagnosis: on this host card 0 reads ReqEn+ and card 1 ReqEn−.
• Card 1's effective link is x1, not the x16 its own capability block advertises: 06:09.0 trained at x1, so ~2 GB/s is the real ceiling. That is a bandwidth fact rather than an atomics one, and it only matters for collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at x4. The layer split's boundary copy does not care.

Measured matrix

Bare 2-rank torch.distributed (backend=nccl) on the two R9700s, no vLLM, no model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset):

Operation	rank 0 (card #1)	rank 1 (card #2)
broadcast	OK	FAIL DistBackendError
all_reduce	OK	FAIL
barrier	OK	FAIL
send / recv (P2P)	blocked, then timeout	FAIL
all_gather (after failure)	group unusable	—

The AMD_LOG_LEVEL=3 trace names the cause: Pcie atomics not enabled, hostcall not supported → AQL dispatch failed! → hipLaunchKernel returns hipErrorIllegalState for ncclDevKernel_Generic_4(...). Nothing in vLLM or RCCL mentions atomics; the runtime log is the only place it appears.

send/recv failing is the decisive result: it kills pipeline parallelism as an option, because vLLM's PP stage handoff is exactly that primitive.

Software escapes: closed, and one still open

Idea	Finding
vLLM's IPC custom allreduce	Gated off by a table keyed to CUDA sm versions — {9.0, 10.0, 10.3, 10.7}; ROCm reports capability 12.0 for gfx1201, so the gate is False. custom_ar itself is compiled into the image.
...and if that table were patched	Still insufficient: broadcast, barrier, and the TP logits all_gather have no non-RCCL path.
The radiance fork's own allreduce	Already present and on by default (RADIANCE_USE_R4D_AR, size gate RADIANCE_AR_MAX_KB): a one-shot P2P push allreduce wrapping CudaCommunicator.all_reduce, exchanging IPC handles over the gloo group. With the gate raised it covers prefill and decode, and ncclDevKernel disappears from the profile. But it does not intercept PyNcclCommunicator.__init__'s warmup allreduce, nor pynccl_comm.all_gather — both direct RCCL calls.
NCCL symmetric memory	Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no ncclDevComm* API, so torch's symmetric-memory path cannot run.
Prefill/decode disaggregation	Needs a KV-transfer connector plus coordination collectives; both out.
gloo for GPU work	CPU/TCP collectives — orders of magnitude too slow for per-layer TP.
Rebuild RCCL so its kernels do not declare a hostcall buffer	OPEN — unverified. The claim is that the hostcall requirement is triggered by device-side printf/assert, and that -mprintf-kind=buffered avoids it. Counter-evidence: RCCL's device source at src/device/ contains no live printf, assert, or __assertfail (checked), so what makes its kernels declare HiddenHostcallBuffer is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with -mprintf-kind=buffered, dump a kernel's declared arguments (roc-obj-extract + llvm-readelf --notes) and look for hidden_hostcall_buffer; if it is gone, a 2-rank probe should dispatch. Stop and mask the service first — see the note below.

Bandwidth: why x4 is enough

A 27B TP=2 allreduce at batch 4 moves roughly 2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB} per layer per forward pass; across ~62 layers that is ≈10 MB per token-step, or ~1.4 ms at PCIe 4.0 x4 (~7.9 GB/s). TP is compute-bound here, not interconnect-bound. That is the whole reason the M.2 riser is the interesting option rather than the board swap.

Ruled out empirically (do not re-derive)

• Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure.
• iommu=pt, and amd_iommu=off is unrelated (that was a Dankclaw prefill knob).
• P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain memory traffic is not the problem; atomic routing is.
• A from-source librccl rebuild: identical failure at the identical call path.
• Every NCCL transport/algorithm knob.
• A one-card test proves nothing: NCCL short-circuits at world_size=1, which is exactly why the two replicas work at all.
• Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 — different host, same image, kernel 7.0.0-30, same enqueue.cc failure.

Test plan (when the hardware changes)

The verdict to look for is one line of lspci, before touching any model:

sudo lspci -vv -s <card2> | grep AtomicOpsCtl     # want: ReqEn+

If that reads ReqEn+, the rest is mechanical: --tensor-parallel-size 2 with HIP_VISIBLE_DEVICES=0,1, then the same benchmark set (scripts/bench-turn-latency.py, and the endpoint's own timings block) so the two architectures are compared on identical workloads. If it still reads ReqEn-, the card is still on a non-atomic path — check lspci -tv for the bridge chain rather than assuming the move worked.

Rule for any future multi-GPU collective experiment: stop and mask the production service first (sudo systemctl stop frank-medium && sudo systemctl mask frank-medium, per the benchmarking policy), or run it on a host that is not serving users. On 2026-09-16 a bare RCCL P2P probe was run against the live service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the GPU driver is a plausible cause and this host has precedent for going unreachable until a power cycle.

The third path, and the one in production: llama.cpp's layer split

Everything above is about collectives, and vLLM cannot proceed without them in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage handoff is a p2p send/recv of the hidden state, the primitive this host also fails. llama.cpp's --split-mode layer needs no collective at all: layers are distributed across the cards and the boundary activation is a plain per-backend copy, once per forward pass. No RCCL kernel is ever launched on the second card, so the atomics gate never comes up.

That is what frank-medium runs today (frank-medium-llamacpp-dflash2): one process, one model, both cards, 262144 context. Measured against tensor split on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a build with a working RCCL — which is exactly what the M.2 riser or a board swap in the short answer would buy.

So the ranking changed on 2026-09-16: software already gets both cards onto one model, and the hardware fix is no longer a means to that end — it is a means to tensor parallelism, which buys ~40 % decode and would let this runtime drop layer-split's serial stages entirely.

What would change the answer

• An RCCL build whose kernels stop declaring a hostcall buffer — the one software route that is not closed. See the last row of the escape table for the hypothesis, its counter-evidence, and the two-command test.
• The symmetric-kernel path inside RCCL, if a release ever ships the device API (ncclDevComm*) that torch's symmetric-memory path needs. The shipped release does not.
• Firmware or an AGESA update that makes the chipset bridges advertise atomic routing — a capability bit, so this would show up in lspci on either side.
• A GPU-side workaround: none found in ROCclr, and the gate is per-device, queried once at probe.

See Also

• Pipeline Parallelism/index — two-machine research (the PP variant of this problem)
• Dankserver System Specs — hardware inventory
• Observability — the scrape/dashboard/alert stack around these endpoints