---
name: amd-multigpu-cooperation-triage
description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives."
---

# AMD Multi-GPU Cooperation Triage

When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix".

## 1. Check atomic-op routing on the whole path

RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*:

```cpp
void Device::checkAtomicSupport() {                      // rocclr/device/rocm/rocdevice.cpp
  link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
    if (link_attrs[0].second == 1) info_.pcie_atomics_ = true;   // no else, no override
}
```

```bash
# Endpoint capability + negotiated link, and EVERY upstream bridge
for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
lspci -tv            # is the card on a CPU root port or behind the chipset?
```

- Endpoint must show `AtomicOpsCtl: ReqEn+`.
- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe.

Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`.

## 2. Map the board before recommending anything

Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.

## 3. Prove what the runtime actually needs (do it OFF the production host)

Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:

```python
torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
    world_size=2, rank=r, timeout=timedelta(seconds=25))
# probe: broadcast, all_reduce, barrier, send/recv, all_gather
```

- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
- `world_size=1` always succeeds; one-card success proves nothing.
- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/<stdin>'`).
- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.

## 4. Don't be fooled by "collective-free" claims

Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.

## 5. The option set, ranked

| Option | Works when | Cost | Trade |
|---|---|---|---|
| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient |
| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser |
| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode |
| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today |
| Patch the custom-AR table | never | — | other collectives still die |
| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host |

## Pitfalls

- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`).
- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show <ip>` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP.
- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`).