agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank committed
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: rccl-pcie-atomics-collective-failure
|
||||
description: "Diagnose AMD multi-GPU RCCL/collective launch failures (hipErrorIllegalState, hostcall) by checking PCIe atomic routing on every upstream bridge, decide whether any software path exists, and rank the fix (CPU-attached M.2 riser vs x8/x8 board vs layer split); includes the bare 2-rank probe matrix and the stop-and-mask safety rule."
|
||||
---
|
||||
|
||||
# RCCL / PCIe atomics collective failure
|
||||
|
||||
Use when a multi-GPU AMD host fails to run tensor/pipeline parallelism, or when
|
||||
asked "can these GPUs work together?".
|
||||
|
||||
## 1. Identify the cause from the runtime, not the framework
|
||||
|
||||
The failure surfaces as `hipErrorIllegalState` from `hipLaunchKernel` for
|
||||
`ncclDevKernel_*`. Nothing in vLLM/RCCL/PyTorch names atomics. Get the real
|
||||
message with:
|
||||
|
||||
```bash
|
||||
AMD_LOG_LEVEL=3 <the failing process>
|
||||
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
|
||||
```
|
||||
|
||||
## 2. Check the whole chain, not just the card
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap" # want ReqEn+
|
||||
lspci -tv # every bridge upstream
|
||||
sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
|
||||
```
|
||||
|
||||
A `Routing-` bridge is a **read-only hardware capability** — no software can
|
||||
enable it. `setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) does nothing
|
||||
because the value is consumed at driver probe time. Typical AM5 layout: one x16
|
||||
slot on CPU root ports (atomics fine), the rest behind the chipset (usually not).
|
||||
|
||||
The gate in source, for the record:
|
||||
|
||||
```cpp
|
||||
void Device::checkAtomicSupport() { // rocclr/.../rocdevice.cpp:181
|
||||
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
||||
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
||||
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else
|
||||
}
|
||||
```
|
||||
|
||||
Consumed at `rocvirtual.cpp` `case HiddenHostcallBuffer:` — the kernel *declares*
|
||||
a hostcall buffer and ROCr refuses to fill it without atomics. **No env var,
|
||||
module param, or sysfs toggle exists** (grep `getenv|HSA_|ROCR_|AMD_` around the
|
||||
gate). Do not promise a software override without checking this.
|
||||
|
||||
## 3. Run the probe matrix — one probe answers several design questions
|
||||
|
||||
Bare 2-rank `torch.distributed` (no vLLM, no model), one process per GPU,
|
||||
`init_process_group("nccl", timeout=timedelta(seconds=25))`:
|
||||
|
||||
| Op | CPU-lane card | chipset card | What it decides |
|
||||
|---|---|---|---|
|
||||
| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
|
||||
| `send`/`recv` (P2P) | hang | FAIL | **PP is dead too** — vLLM's PP handoff is this primitive |
|
||||
|
||||
Test `send`/`recv` **in isolation** (a prior failed op leaves the group broken
|
||||
and the result meaningless — it just hangs until the timeout). Set an explicit
|
||||
PG timeout so a blocked pair raises instead of hanging.
|
||||
|
||||
**SAFETY — non-negotiable.** Never run this on a host that is serving:
|
||||
`sudo systemctl stop <svc> && sudo systemctl mask <svc>` first, or use a host
|
||||
that serves nobody. A probe run against a live GPU service preceded a hard host
|
||||
down (no ARP, no ICMP from two subnets, power cycle required). See the
|
||||
benchmarking policy: it covers ANY GPU experiment, not just benchmarks.
|
||||
|
||||
Telling "off" from "wedged" afterwards: `ip neigh show <ip>` from a peer —
|
||||
`FAILED`/`INCOMPLETE` = off or NIC dead; a merely wedged kernel still answers ARP.
|
||||
|
||||
## 4. Rank the fixes
|
||||
|
||||
1. **Move the card to a CPU-attached M.2 slot** (M.2→PCIe x4 riser). Cheap and
|
||||
reversible. Get the board's *own spec page*: it lists which M.2 slots come
|
||||
from the CPU vs the chipset, and the chipset x16 slot's real width (often
|
||||
x1 mode). M.2 slots can be empty even when the x16 slots are full.
|
||||
2. **Board whose CPU x16 bifurcates x8/x8** — both cards on the CPU root port.
|
||||
3. **llama.cpp / ik_llama.cpp `--split-mode layer`** — no collectives at all, so
|
||||
it sidesteps the wall entirely; costs the GGUF conversion plus paged
|
||||
attention, continuous batching, and spec decode.
|
||||
4. Two TP=1 replicas + a routing proxy — what you do when you cannot fix the
|
||||
topology; keeps throughput but the cards never share one model.
|
||||
|
||||
**Verify before benchmarking anything:** `AtomicOpsCtl: ReqEn+` on the moved
|
||||
card. X4 is usually enough for TP (≈10 MB per token-step for a 27B at batch 4
|
||||
≈1.4 ms at Gen4 x4) — compute-bound, not interconnect-bound.
|
||||
|
||||
## 5. Closed software routes (don't re-litigate)
|
||||
|
||||
- vLLM's IPC custom allreduce is gated by a **CUDA-sm-version-keyed table**
|
||||
(`{9.0, 10.0, 10.3, 10.7}`); ROCm reports `12.0` on gfx1201 → `False`. The ops
|
||||
are compiled, the gate is not. Patching the table is still not enough —
|
||||
`broadcast`/`barrier`/TP-logits `all_gather` have no non-RCCL path.
|
||||
- A fork that wraps `.all_reduce` (e.g. radiance's `RADIANCE_USE_R4D_AR` one-shot
|
||||
P2P allreduce, gloo for IPC-handle exchange) does not intercept
|
||||
`PyNcclCommunicator.__init__`'s warmup allreduce or `pynccl_comm.all_gather`.
|
||||
- NCCL symmetric memory: needs an RCCL exposing `ncclDevComm*`; many builds ship
|
||||
the NCCL version string without the device API.
|
||||
- P/D disaggregation: needs a KV connector *and* coordination collectives.
|
||||
- gloo: CPU/TCP, unreachable for per-layer TP.
|
||||
|
||||
**Open, unverified**: recompiling RCCL so its kernels stop declaring a hostcall
|
||||
buffer (`-mprintf-kind=buffered` is the usual claim). Counter-evidence: RCCL's
|
||||
device source has no live `printf`/`assert`/`__assertfail`, so the trigger is
|
||||
unexplained. Test: rebuild, then dump a kernel's declared args
|
||||
(`roc-obj-extract` + `llvm-readelf --notes`) and look for
|
||||
`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
|
||||
Reference in new issue
Block a user