Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
72 lines
5.1 KiB
Markdown
72 lines
5.1 KiB
Markdown
---
|
|
name: amd-multigpu-cooperation-triage
|
|
description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives."
|
|
---
|
|
|
|
# AMD Multi-GPU Cooperation Triage
|
|
|
|
When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix".
|
|
|
|
## 1. Check atomic-op routing on the whole path
|
|
|
|
RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*:
|
|
|
|
```cpp
|
|
void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp
|
|
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
|
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
|
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else, no override
|
|
}
|
|
```
|
|
|
|
```bash
|
|
# Endpoint capability + negotiated link, and EVERY upstream bridge
|
|
for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
|
|
lspci -tv # is the card on a CPU root port or behind the chipset?
|
|
```
|
|
|
|
- Endpoint must show `AtomicOpsCtl: ReqEn+`.
|
|
- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe.
|
|
|
|
Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`.
|
|
|
|
## 2. Map the board before recommending anything
|
|
|
|
Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.
|
|
|
|
## 3. Prove what the runtime actually needs (do it OFF the production host)
|
|
|
|
Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:
|
|
|
|
```python
|
|
torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
|
|
world_size=2, rank=r, timeout=timedelta(seconds=25))
|
|
# probe: broadcast, all_reduce, barrier, send/recv, all_gather
|
|
```
|
|
|
|
- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
|
|
- `world_size=1` always succeeds; one-card success proves nothing.
|
|
- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/<stdin>'`).
|
|
- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.
|
|
|
|
## 4. Don't be fooled by "collective-free" claims
|
|
|
|
Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.
|
|
|
|
## 5. The option set, ranked
|
|
|
|
| Option | Works when | Cost | Trade |
|
|
|---|---|---|---|
|
|
| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient |
|
|
| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser |
|
|
| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode |
|
|
| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today |
|
|
| Patch the custom-AR table | never | — | other collectives still die |
|
|
| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host |
|
|
|
|
## Pitfalls
|
|
|
|
- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`).
|
|
- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show <ip>` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP.
|
|
- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`).
|