research: how to make the two R9700s work together (and one correction to my own claim)

New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.

**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.

**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.

**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.

**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.

serverAppConfigs-docs/index.md

@@ -120,6 +120,12 @@ The LLM inference fleet on the two hosts. Full index:
   AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus
   the PCIe-atomics evidence that rules out TP=2 on this host (since
   2026-09-15/16)
+- [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel
+  together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
+  (collectives *and* P2P `send`/`recv` both fail on card #2, which rules out
+  pipeline parallelism too), what is closed and what is still open, and the two
+  hardware paths — a ~$30 M.2→PCIe x4 riser into the free, CPU-attached M.2_2
+  slot, or a board whose CPU x16 bifurcates to x8/x8
 - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
   charlie12345/ROCmFPX serving kingjones777/
   Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it