command-center/agent/managed-skills/rccl-pcie-atomics-collective-failure/SKILL.md

---
name: rccl-pcie-atomics-collective-failure
description: "Diagnose AMD multi-GPU RCCL/collective launch failures (hipErrorIllegalState, hostcall) by checking PCIe atomic routing on every upstream bridge, decide whether any software path exists, and rank the fix (CPU-attached M.2 riser vs x8/x8 board vs layer split); includes the bare 2-rank probe matrix and the stop-and-mask safety rule."
---

# RCCL / PCIe atomics collective failure

Use when a multi-GPU AMD host fails to run tensor/pipeline parallelism, or when
asked "can these GPUs work together?".

## 1. Identify the cause from the runtime, not the framework

The failure surfaces as `hipErrorIllegalState` from `hipLaunchKernel` for
`ncclDevKernel_*`. Nothing in vLLM/RCCL/PyTorch names atomics. Get the real
message with:

```bash
AMD_LOG_LEVEL=3 <the failing process>
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
```

## 2. Check the whole chain, not just the card

```bash
sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap"   # want ReqEn+
lspci -tv                                                        # every bridge upstream
sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+
```

A `Routing-` bridge is a **read-only hardware capability** — no software can
enable it. `setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) does nothing
because the value is consumed at driver probe time. Typical AM5 layout: one x16
slot on CPU root ports (atomics fine), the rest behind the chipset (usually not).

The gate in source, for the record:

```cpp
void Device::checkAtomicSupport() {              // rocclr/.../rocdevice.cpp:181
  link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
    if (link_attrs[0].second == 1) info_.pcie_atomics_ = true;   // no else
}
```

Consumed at `rocvirtual.cpp` `case HiddenHostcallBuffer:` — the kernel *declares*
a hostcall buffer and ROCr refuses to fill it without atomics. **No env var,
module param, or sysfs toggle exists** (grep `getenv|HSA_|ROCR_|AMD_` around the
gate). Do not promise a software override without checking this.

## 3. Run the probe matrix — one probe answers several design questions

Bare 2-rank `torch.distributed` (no vLLM, no model), one process per GPU,
`init_process_group("nccl", timeout=timedelta(seconds=25))`:

| Op | CPU-lane card | chipset card | What it decides |
|---|---|---|---|
| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
| `send`/`recv` (P2P) | hang | FAIL | **PP is dead too** — vLLM's PP handoff is this primitive |

Test `send`/`recv` **in isolation** (a prior failed op leaves the group broken
and the result meaningless — it just hangs until the timeout). Set an explicit
PG timeout so a blocked pair raises instead of hanging.

**SAFETY — non-negotiable.** Never run this on a host that is serving:
`sudo systemctl stop <svc> && sudo systemctl mask <svc>` first, or use a host
that serves nobody. A probe run against a live GPU service preceded a hard host
down (no ARP, no ICMP from two subnets, power cycle required). See the
benchmarking policy: it covers ANY GPU experiment, not just benchmarks.

Telling "off" from "wedged" afterwards: `ip neigh show <ip>` from a peer —
`FAILED`/`INCOMPLETE` = off or NIC dead; a merely wedged kernel still answers ARP.

## 4. Rank the fixes

1. **Move the card to a CPU-attached M.2 slot** (M.2→PCIe x4 riser). Cheap and
   reversible. Get the board's *own spec page*: it lists which M.2 slots come
   from the CPU vs the chipset, and the chipset x16 slot's real width (often
   x1 mode). M.2 slots can be empty even when the x16 slots are full.
2. **Board whose CPU x16 bifurcates x8/x8** — both cards on the CPU root port.
3. **llama.cpp / ik_llama.cpp `--split-mode layer`** — no collectives at all, so
   it sidesteps the wall entirely; costs the GGUF conversion plus paged
   attention, continuous batching, and spec decode.
4. Two TP=1 replicas + a routing proxy — what you do when you cannot fix the
   topology; keeps throughput but the cards never share one model.

**Verify before benchmarking anything:** `AtomicOpsCtl: ReqEn+` on the moved
card. X4 is usually enough for TP (≈10 MB per token-step for a 27B at batch 4
≈1.4 ms at Gen4 x4) — compute-bound, not interconnect-bound.

## 5. Closed software routes (don't re-litigate)

- vLLM's IPC custom allreduce is gated by a **CUDA-sm-version-keyed table**
  (`{9.0, 10.0, 10.3, 10.7}`); ROCm reports `12.0` on gfx1201 → `False`. The ops
  are compiled, the gate is not. Patching the table is still not enough —
  `broadcast`/`barrier`/TP-logits `all_gather` have no non-RCCL path.
- A fork that wraps `.all_reduce` (e.g. radiance's `RADIANCE_USE_R4D_AR` one-shot
  P2P allreduce, gloo for IPC-handle exchange) does not intercept
  `PyNcclCommunicator.__init__`'s warmup allreduce or `pynccl_comm.all_gather`.
- NCCL symmetric memory: needs an RCCL exposing `ncclDevComm*`; many builds ship
  the NCCL version string without the device API.
- P/D disaggregation: needs a KV connector *and* coordination collectives.
- gloo: CPU/TCP, unreachable for per-layer TP.

**Open, unverified**: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the usual claim). Counter-evidence: RCCL's
device source has no live `printf`/`assert`/`__assertfail`, so the trigger is
unexplained. Test: rebuild, then dump a kernel's declared args
(`roc-obj-extract` + `llvm-readelf --notes`) and look for
`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).