research: how to make the two R9700s work together (and one correction to my own claim)

New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.

**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.

**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.

**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.

**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.

AGENTS.md

@@ -200,6 +200,17 @@ Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
+This covers **any GPU experiment, not just benchmarks** — including a bare
+2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
+against the live service "because it only needs 200 MB of VRAM"; the host went
+hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no
+recovery — a power cycle was required). A hung AQL dispatch wedging the GPU
+driver is the plausible mechanism, and this host has precedent for going
+unreachable until a power cycle. If an experiment needs GPUs, stop and mask
+first, or use a host that serves nobody. Telling an off machine from a wedged
+one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
+merely wedged kernel usually still answers ARP.
+
 1. Stop and mask (self-healing would restart it):
    - Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
@@ -276,9 +287,10 @@ a device without PCIe atomics; the second R9700 sits behind the chipset
 bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a
 read-only capability, so no software can enable atomics on that path. Not the
 kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL
-build (a from-source rebuild fails identically). A slot on a CPU root port —
-there is none free on this AM5 platform for a second x16 card — or a platform
-with more CPU lanes restores TP=2. `frank-medium.container` stays in the repo
+build (a from-source rebuild fails identically). Putting both cards on CPU root
+ports restores TP=2: on this board that means an M.2→PCIe x4 riser into the
+free, CPU-attached M.2_2 slot, or a board whose CPU x16 bifurcates to x8/x8.
+`frank-medium.container` stays in the repo
 as the TP=2 target; the deploy handler does not install it.
 
 Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
@@ -304,6 +316,16 @@ linked runbooks.
   and the rest behind the chipset (usually not) — that asymmetry is what
   blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
   short-circuits at `world_size=1`.
+- **The fix is physical — and cheap on this board.** Two paths put both cards
+  on CPU root ports, which restores the atomics routing and therefore TP=2: an
+  M.2→PCIe x4 riser into a *CPU-attached* M.2 slot (on dankserver's TUF
+  GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page;
+  `PCIEX16_2`, where card #2 sits today, is chipset and x1-mode), or a board
+  whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B
+  TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4.
+  RCCL's P2P `send`/`recv` fails on such a card too, so pipeline parallelism is
+  not an escape either. Full evidence matrix, the open rebuild hypothesis, and
+  the test plan: [[Dual-R9700-Cooperation]].
 - **When a GPU collective dies, read the runtime, not the framework.** The
   message that identified the cause came from `AMD_LOG_LEVEL=3`
   (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch