command-center/agent/managed-skills/rccl-pcie-atomics-collective-failure/SKILL.md

name, description

name	description
rccl-pcie-atomics-collective-failure	Diagnose AMD multi-GPU RCCL/collective launch failures (hipErrorIllegalState, hostcall) by checking PCIe atomic routing on every upstream bridge, decide whether any software path exists, and rank the fix (CPU-attached M.2 riser vs x8/x8 board vs layer split); includes the bare 2-rank probe matrix and the stop-and-mask safety rule.

RCCL / PCIe atomics collective failure

Use when a multi-GPU AMD host fails to run tensor/pipeline parallelism, or when asked "can these GPUs work together?".

1. Identify the cause from the runtime, not the framework

The failure surfaces as hipErrorIllegalState from hipLaunchKernel for ncclDevKernel_*. Nothing in vLLM/RCCL/PyTorch names atomics. Get the real message with:

AMD_LOG_LEVEL=3 <the failing process>
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"

2. Check the whole chain, not just the card

sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap"   # want ReqEn+
lspci -tv                                                        # every bridge upstream
sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+

A Routing- bridge is a read-only hardware capability — no software can enable it. setpci -s <gpu> CAP_EXP+28.w=0440 (DCTL2 bit 6) does nothing because the value is consumed at driver probe time. Typical AM5 layout: one x16 slot on CPU root ports (atomics fine), the rest behind the chipset (usually not).

The gate in source, for the record:

void Device::checkAtomicSupport() {              // rocclr/.../rocdevice.cpp:181
  link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
    if (link_attrs[0].second == 1) info_.pcie_atomics_ = true;   // no else
}

Consumed at rocvirtual.cpp case HiddenHostcallBuffer: — the kernel declares a hostcall buffer and ROCr refuses to fill it without atomics. No env var, module param, or sysfs toggle exists (grep getenv|HSA_|ROCR_|AMD_ around the gate). Do not promise a software override without checking this.

3. Run the probe matrix — one probe answers several design questions

Bare 2-rank torch.distributed (no vLLM, no model), one process per GPU, init_process_group("nccl", timeout=timedelta(seconds=25)):

Op	CPU-lane card	chipset card	What it decides
broadcast/all_reduce/barrier	OK	FAIL	TP is dead
send/recv (P2P)	hang	FAIL	PP is dead too — vLLM's PP handoff is this primitive

Test send/recv in isolation (a prior failed op leaves the group broken and the result meaningless — it just hangs until the timeout). Set an explicit PG timeout so a blocked pair raises instead of hanging.

SAFETY — non-negotiable. Never run this on a host that is serving: sudo systemctl stop <svc> && sudo systemctl mask <svc> first, or use a host that serves nobody. A probe run against a live GPU service preceded a hard host down (no ARP, no ICMP from two subnets, power cycle required). See the benchmarking policy: it covers ANY GPU experiment, not just benchmarks.

Telling "off" from "wedged" afterwards: ip neigh show <ip> from a peer — FAILED/INCOMPLETE = off or NIC dead; a merely wedged kernel still answers ARP.

4. Rank the fixes

1. Move the card to a CPU-attached M.2 slot (M.2→PCIe x4 riser). Cheap and reversible. Get the board's own spec page: it lists which M.2 slots come from the CPU vs the chipset, and the chipset x16 slot's real width (often x1 mode). M.2 slots can be empty even when the x16 slots are full.
2. Board whose CPU x16 bifurcates x8/x8 — both cards on the CPU root port.
3. llama.cpp / ik_llama.cpp --split-mode layer — no collectives at all, so it sidesteps the wall entirely; costs the GGUF conversion plus paged attention, continuous batching, and spec decode.
4. Two TP=1 replicas + a routing proxy — what you do when you cannot fix the topology; keeps throughput but the cards never share one model.

Verify before benchmarking anything: AtomicOpsCtl: ReqEn+ on the moved card. X4 is usually enough for TP (≈10 MB per token-step for a 27B at batch 4 ≈1.4 ms at Gen4 x4) — compute-bound, not interconnect-bound.

5. Closed software routes (don't re-litigate)

• vLLM's IPC custom allreduce is gated by a CUDA-sm-version-keyed table ({9.0, 10.0, 10.3, 10.7}); ROCm reports 12.0 on gfx1201 → False. The ops are compiled, the gate is not. Patching the table is still not enough — broadcast/barrier/TP-logits all_gather have no non-RCCL path.
• A fork that wraps .all_reduce (e.g. radiance's RADIANCE_USE_R4D_AR one-shot P2P allreduce, gloo for IPC-handle exchange) does not intercept PyNcclCommunicator.__init__'s warmup allreduce or pynccl_comm.all_gather.
• NCCL symmetric memory: needs an RCCL exposing ncclDevComm*; many builds ship the NCCL version string without the device API.
• P/D disaggregation: needs a KV connector and coordination collectives.
• gloo: CPU/TCP, unreachable for per-layer TP.

Open, unverified: recompiling RCCL so its kernels stop declaring a hostcall buffer (-mprintf-kind=buffered is the usual claim). Counter-evidence: RCCL's device source has no live printf/assert/__assertfail, so the trigger is unexplained. Test: rebuild, then dump a kernel's declared args (roc-obj-extract + llvm-readelf --notes) and look for hidden_hostcall_buffer; if gone, retry the 2-rank probe (service stopped).