dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: dankserver board swap clears the PCIe-atomics wall

Browse Source 
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.

- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
  topology table, rebuild-hypothesis door marked moot, probe plan is now
  standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
  07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
  eno2 (192.168.0.247) second 5GbE port holding the low-metric default
  route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
  from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.

Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
This commit is contained in:
Frank
2026-09-20 14:39:54 -07:00
parent a43ab90105
commit d16e8e554e
7 changed files with 246 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
+105 -40
View File
@@ -1,7 +1,7 @@
---
type: research
title: Dual R9700 Cooperation
description: Why frank-medium's two Radeon AI PRO R9700s run as independent replicas instead of tensor-parallel, which options are ruled out with evidence, and the two hardware paths that restore true TP=2.
description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
tags:
- frank-medium
- rocm
@@ -14,32 +14,43 @@ timestamp: 2026-09-16
# Dual R9700 Cooperation
Dankserver's `frank-medium` runs as **two independent single-GPU replicas**
(`:7771` on card #1, `:7772` on card #2) behind an adaptive-routing proxy on
`:7770`. This page answers why, and what would let the two cards work on one
model instead. See [[frank-medium-radiance-mxfp4]] for the deployment itself.
Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a
layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true
tensor parallelism was impossible on the previous board (TUF GAMING B650-E
WIFI), and what the 2026-09-20 board swap changed. See
[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under
the old constraint.
## The short answer
**Card #2 sits behind the B650 chipset, whose bridges cannot route PCIe atomic
operations.** RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr
refuses to supply it to a device without atomics — so *no* RCCL collective or
P2P operation can be dispatched on card #2. Two paths restore cooperation:
**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
hostcall-buffer parameter and ROCr refuses to supply it to a device without
atomics, so *no* RCCL collective or P2P operation could be dispatched on that
card — see the [measured matrix](#measured-matrix) and the
[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
1. **Move card #2 to the free, CPU-attached M.2_2 slot** (~$30 M.2→PCIe x4
riser). Cheapest, and bandwidth is not a constraint (below).
2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class,
~$250-400). Best performance. This board has no x8/x8 mode.
On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
recommended, at B850 rather than X870E money. Both cards now hang off CPU root
ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
the exact precondition this page's own [test plan](#test-plan) names.
The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS
lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as
chipset, while an empty slot does not enumerate — so nothing on a running
system can confirm it until a card is in it. And CPU-attached does not imply
`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth
trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what
the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one
model" — the layer split already delivers that, so this is a performance
project, not a capability one.
What that buys, and what it does not:
- **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp
`--split-mode row`. All three died on the same hostcall gate.
- **Not yet measured.** Nothing has dispatched a collective on this host since
the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask
production first.
- **Not needed for capability.** The layer split already puts both cards on one
model. This is a performance question: an RCCL-backed tensor split, and the
single pooled KV that comes with it.
- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
and M.2_2 is free.
## The gate, in source
@@ -100,6 +111,45 @@ Three things fall out of that table, none of them in the source alone:
collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
x4. The layer split's boundary copy does not care.
## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
LAN/SATA/WiFi:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
`Routing-` bridge. Same one-command diagnosis, opposite answer.
- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
- `rocm-smi --showtopo` still reports `PCIE` between the cards (Navi 48 has no
XGMI), they are still in separate IOMMU groups (15 and 20), and both root
ports have ACS `ReqRedir+`, so P2P hairpins at the root complex. Plain P2P
traffic measured 45.27 GB/s bidirectional on the *old* board — DMA was never
the gate, atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt`. Both were ruled out as causes
([above](#ruled-out-empirically-do-not-re-derive)) and both are unchanged.
**Still unproven.** The gate that matters is ROCclr's `checkAtomicSupport()`
reading the `kLinkAtomicSupport` link attribute; the ReqEn bit is the same cause
seen from the kernel side, but no RCCL kernel has been launched since the swap.
Until the 2-rank probe runs — or a `--split-mode row` boot that does *not* log
`internal AllReduce init failed ... falling back to meta-backend butterfly` —
"collectives work here" is a strong expectation, not a measurement.
## Measured matrix
Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
@@ -121,7 +171,11 @@ RCCL mentions atomics; the runtime log is the only place it appears.
**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
as an option, because vLLM's PP stage handoff is exactly that primitive.
## Software escapes: closed, and one still open
## Software escapes: closed, and the last one is now moot
Every row below was written under the old board. The last row — the only door
that was ever open — stopped mattering on 2026-09-20, because the hardware gate
it existed to route around no longer applies to this host.
| Idea | Finding |
|---|---|
@@ -138,9 +192,11 @@ as an option, because vLLM's PP stage handoff is exactly that primitive.
A 27B TP=2 allreduce at batch 4 moves roughly
$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s). TP is compute-bound here, not
interconnect-bound. That is the whole reason the M.2 riser is the interesting
option rather than the board swap.
or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
That is why the cheap option (the M.2 riser) and the expensive one (the board
swap) were both considered adequate on bandwidth alone — and why the board that
landed, at x8/x8, clears the arithmetic with room to spare.
## Ruled out empirically (do not re-derive)
@@ -157,6 +213,9 @@ option rather than the board swap.
## Test plan (when the hardware changes)
**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so
this is now a standing plan rather than a hypothesis.
The verdict to look for is one line of `lspci`, before touching any model:
```bash
@@ -193,25 +252,31 @@ on the second card, so the atomics gate never comes up.
That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one
process, one model, both cards, 262144 context. Measured against tensor split
on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 /
26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a
build with a working RCCL — which is exactly what the M.2 riser or a board swap
in [the short answer](#the-short-answer) would buy.
26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a
prediction for an RCCL-backed build** — it is what tensor split costs when the
allreduce degrades to a meta-backend butterfly, and it is the number the
board-swap path exists to retire.
So the ranking changed on 2026-09-16: software already gets both cards onto one
model, and the hardware fix is no longer a means to that end — it is a means to
*tensor* parallelism, which buys ~40 % decode and would let this runtime drop
layer-split's serial stages entirely.
The ranking therefore changed twice. On 2026-09-16: software already gets both
cards onto one model, so the hardware fix stopped being a capability question
and became a performance one — *tensor* parallelism, ~40 % decode, plus one
pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
hardware half of that question was answered by the board swap; the measurement
half has not run yet, and `GGML_HIP_RCCL` is still `OFF` in
`scripts/build-llama-rdna4.sh` for exactly the reason this page documents.
## What would change the answer
- An RCCL build whose kernels stop declaring a hostcall buffer — the one
software route that is *not* closed. See the last row of the escape table for
the hypothesis, its counter-evidence, and the two-command test.
- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
since 2026-09-20.** That door only mattered while the hardware gate could not
be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
and its counter-evidence stay in the escape table for the record.
- The symmetric-kernel path inside RCCL, if a release ever ships the device API
(`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped
release does not.
- Firmware or an AGESA update that makes the chipset bridges advertise atomic
routing — a capability bit, so this would show up in `lspci` on either side.
(`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
not.
- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
routing~~ — arrived as new silicon instead.
- A GPU-side workaround: none found in ROCclr, and the gate is per-device,
queried once at probe.