command-center/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md

name, description

name	description
amd-multigpu-cooperation-triage	Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives.

AMD Multi-GPU Cooperation Triage

When someone wants N GPUs "working together" on one model (TP/PP), decide feasibility first — the answer on consumer AMD platforms is often "no, and here is the $30 fix".

1. Check atomic-op routing on the whole path

RCCL collective and P2P kernels carry a hidden hostcall buffer parameter. ROCr only supplies it when the device reports PCIe atomics, which is read once at driver probe from a queried hardware property:

void Device::checkAtomicSupport() {                      // rocclr/device/rocm/rocdevice.cpp
  link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
    if (link_attrs[0].second == 1) info_.pcie_atomics_ = true;   // no else, no override
}

# Endpoint capability + negotiated link, and EVERY upstream bridge
for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
lspci -tv            # is the card on a CPU root port or behind the chipset?

• Endpoint must show AtomicOpsCtl: ReqEn+.
• Every hop to the CPU must show AtomicOpsCap: Routing+. Routing- is a read-only capability — no software can enable it, and setpci ... CAP_EXP+28.w=0440 does not help because the driver already consumed the value at probe.

Failure signature (never names atomics): AMD_LOG_LEVEL=3 → Pcie atomics not enabled, hostcall not supported → AQL dispatch failed! → hipLaunchKernel: hipErrorIllegalState on ncclDevKernel_Generic_4.

2. Map the board before recommending anything

Read the vendor's techspec page, not a review: which slots are CPU-attached vs chipset, each slot's electrical width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.

3. Prove what the runtime actually needs (do it OFF the production host)

Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:

torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
    world_size=2, rank=r, timeout=timedelta(seconds=25))
# probe: broadcast, all_reduce, barrier, send/recv, all_gather

• Expect the chipset-attached rank to fail every op — including send/recv, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
• world_size=1 always succeeds; one-card success proves nothing.
• Run the probe with a real file (torch.multiprocessing.spawn re-executes it — piping a script into python3 - fails with FileNotFoundError: '/<stdin>').
• Stop and mask the serving unit first (systemctl stop X && systemctl mask X; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.

4. Don't be fooled by "collective-free" claims

Check the gate, not the feature list. vLLM's IPC custom allreduce (custom_ar=True when the ops are compiled) is gated by a CUDA-sm-keyed table (CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}) while ROCm reports e.g. DeviceCapability(major=12, minor=0) — so it is off on RDNA4 by a code limit, and current_platform.is_rocm() exclusions appear nearby. Even patching it is insufficient: TP init's broadcast/barrier/all_gather have no non-RCCL path and die the same way. No RCCL kernel involving the atomics-less card can launch ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.

5. The option set, ranked

Option	Works when	Cost	Trade
CPU-attached M.2 (x4) riser	a CPU-lane M.2 slot is free	~$30	TP=2 with x4 bandwidth — sufficient
CPU x8/x8 board	board supports CPU bifurcation	~$300+	best bandwidth, no riser
llama.cpp/ik_llama.cpp --split-mode layer	always (no collectives)	$0	GGUF model; loses paged attention, continuous batching, spec decode
Independent TP=1 replicas + router	always	$0	cards never share one model; best throughput today
Patch the custom-AR table	never	—	other collectives still die
P/D disaggregation	ROCm KV-transfer connector + collectives	—	out on this class of host

Pitfalls

• A one-card collective test proves nothing (NCCL short-circuits at world_size=1).
• Validation must come from the peer/host doing the work: probe reachability with ip neigh show <ip> — FAILED/INCOMPLETE means off or NIC dead, whereas a wedged kernel usually still answers ARP.
• Validate your own probe's naming before reporting a service outage (e.g. systemctl --user is-active gitea vs the real unit gitea-server).