docs: dankserver board swap clears the PCIe-atomics wall

Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.

- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
  topology table, rebuild-hypothesis door marked moot, probe plan is now
  standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
  07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
  eno2 (192.168.0.247) second 5GbE port holding the low-metric default
  route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
  from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.

Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
This commit is contained in:
Frank committed 2026-09-20 14:39:54 -07:00
1 parent a43ab90105
commit d16e8e554e
7 files changed
+246 -110

No files matched your search

@@ -32,15 +32,22 @@ replicas, the proxy and their unit files are retired.
## Why one process and layer split
The endpoint's whole history is one constraint: **the two cards cannot take
The endpoint's whole history is one constraint: **the two cards could not take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; the second
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
every collective that touches it dies with `hipErrorIllegalState`. That rules
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
whose stage handoff is a `send`/`recv` of the hidden state on the device
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
The full matrix is in [[Dual-R9700-Cooperation]].
and ROCr refuses to supply it to a device without PCIe atomics; on the previous
board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`,
read-only), so every collective that touched it died with
`hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out
**vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of
the hidden state on the device backend
(`vllm/distributed/parallel_state.py`), the very primitive that failed. The
full matrix is in [[Dual-R9700-Cooperation]].
**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both
cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read
`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is
*unblocked, not proven* — and this service still runs the shape below, which
needs no collective either way.
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain
@@ -63,7 +70,7 @@ shape — prefill nearly doubles and one model serves from both cards.
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
| Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |