docs: dankserver board swap clears the PCIe-atomics wall

Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.

- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
  topology table, rebuild-hypothesis door marked moot, probe plan is now
  standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
  07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
  eno2 (192.168.0.247) second 5GbE port holding the low-metric default
  route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
  from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.

Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
+|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap — atomic routing verified on both chains), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
 | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
 
@@ -337,10 +337,13 @@ model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
 **one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
 split** — balanced (operator call 2026-09-19, the most even split 65 layers
 allow; ROCm1, the last-layers card that also carries the drafter and output
-layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned
-work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4
-x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s);
-the balanced point trades that lean for symmetry and was accepted. 327680 ctx
+layers, holds the extra one).
+The predecessor's 35:30 skew (2026-09-17) leaned work onto the wider card
+because ROCm1 (then `0b:00.0`) rode the B650 chipset at Gen4 x1 (~1.97 GB/s)
+against ROCm0's CPU-root-port Gen4 x16 (~31.5 GB/s); the balanced point traded
+that lean for symmetry and was accepted. The 2026-09-20 board swap (ASUS ProArt
+B850-Creator WiFi NEO) removed the other axis: both cards are now CPU-root-port
+Gen5 x8 (~31.5 GB/s each), at `03:00.0` and `07:00.0`. 327680 ctx
 over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
 no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
 KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
@@ -348,13 +351,18 @@ tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
 window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
 layers of the split.
 
-**Layer split is why both cards now serve ONE model.** Tensor parallelism is
-impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
-its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact
-primitive that cannot be dispatched on card #2. llama.cpp's layer split needs
+**Layer split is why both cards serve ONE model today.** It was chosen because
+tensor parallelism was then impossible (see the PCIe lesson below) and so was
+vLLM's pipeline parallelism — its stage handoff is an RCCL `send`/`recv` of the
+hidden state, the exact primitive that could not be dispatched on the old
+board's card #2. llama.cpp's layer split needs
 no collective at all: layers are distributed in order and the boundary
 activation is a per-backend copy, once per forward pass. Measured on this host
-(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor.
+(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor — but that
+tensor number is the cost of a butterfly-fallback allreduce with `GGML_HIP_RCCL`
+OFF, not what an RCCL-backed tensor split would do. Since the 2026-09-20 board
+swap cleared the atomics gate, that comparison is stale and the probe is the
+open work; see [[Dual-R9700-Cooperation]].
 
 This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
 routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
@@ -396,25 +404,27 @@ linked runbooks.
   and the rest behind the chipset (usually not) — that asymmetry is what
   blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
   short-circuits at `world_size=1`.
-- **The fix is physical — and cheap on this board.** Two paths put both cards
-  on CPU root ports, which restores the atomics routing and therefore TP=2: an
-  M.2→PCIe x4 riser into a *CPU-attached* M.2 slot (on dankserver's TUF
-  GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page;
-  `PCIEX16_2`, where card #2 sits today, is chipset and x1-mode), or a board
-  whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B
-  TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4.
-  RCCL's P2P `send`/`recv` fails on such a card too, so pipeline parallelism is
-  not an escape either. Full evidence matrix, the open rebuild hypothesis, and
-  the test plan: [[Dual-R9700-Cooperation]].
+- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
+  Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
+  into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
+  x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
+  `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
+  every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
+  precondition the old test plan named. Bandwidth was never the constraint (a
+  27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
+  and under 0.4 ms at Gen5 x8). **What is NOT yet re-measured:** no collective
+  has been dispatched since the swap, so TP=2 is unblocked, not proven. Full
+  evidence matrix and the standing probe plan: [[Dual-R9700-Cooperation]].
 - **When a GPU collective dies, read the runtime, not the framework.** The
   message that identified the cause came from `AMD_LOG_LEVEL=3`
   (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
   failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
   prints the failing HIP call with its return code.
-- **The one software path that lets both cards serve ONE model: llama.cpp's
-  layer split.** vLLM cannot do it here in any of its parallelisms — TP needs
-  an allreduce, PP needs the same p2p primitive, and both die on the card that
-  cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no
+- **The software path that puts both cards on ONE model without a collective:
+  llama.cpp's layer split.** On the old board vLLM could not do it in any of its
+  parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
+  died on the card that could not dispatch an RCCL kernel. `--split-mode
+  layer` in llama.cpp needs no
   collective: layers are distributed in order and only the boundary activation
   crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
   18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -1,14 +1,14 @@
 ---
 type: configuration
 title: Dankserver System Specs
-description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium.
+description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
 tags:
   - dankserver
   - hardware
   - r9700
   - gfx1201
   - rocm
-timestamp: 2026-09-15
+timestamp: 2026-09-20
 ---
 
 # Dankserver System Specs
@@ -22,6 +22,18 @@ two AMD Radeon AI PRO R9700 cards.
 
 ## Hardware Specifications
 
+### Motherboard
+
+| Property | Value |
+|----------|-------|
+| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
+| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
+| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3`; ~31.5 GB/s per card |
+| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board |
+| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
+| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less), trained **x2** at Gen5 (63 Gb/s ≈ 7.9 GB/s) |
+| Onboard | 2x Realtek RTL8126 5GbE, RTL8922AE WiFi 7, PROM21 SATA controller |
+
 ### CPU
 
 | Property | Value |
@@ -29,7 +41,7 @@ two AMD Radeon AI PRO R9700 cards.
 | Architecture | x86_64 |
 | Vendor | AuthenticAMD |
 | Model | AMD Ryzen 7 7700X |
-| Cores | 8 cores / 8 threads (1 socket, SMT off) |
+| Cores | 8 cores / 16 threads (1 socket) — SMT is **on**; the BIOS defaults reset with the board swap. `llama.cpp` still runs `--threads 8 --threads-batch 8`, so this is headroom, not a change in the serving shape |
 | NUMA Nodes | 1 (node 0) |
 
 ### GPUs
@@ -38,10 +50,11 @@ two AMD Radeon AI PRO R9700 cards.
 |----------|-------|
 | Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) |
 | Architecture | gfx1201 (RDNA4) |
-| VRAM | 32,624 MiB each |
-| Link | PCIe, no XGMI; different IOMMU groups |
-| PCI | 03:00.0 and 0b:00.0 |
-| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP |
+| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
+| Link | PCIe 5.0 **x8** each (CPU GPP `00:01.1` / `00:01.3`), no XGMI; separate IOMMU groups (15 and 20), ACS `ReqRedir+` on both root ports, so P2P hairpins at the root complex |
+| Atomic ops | `Routing+` on all four bridges per chain; `ReqEn+` on both endpoints (`CAP_EXP+28.w = 0x1440`) |
+| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
+| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2, hidden from HIP |
 
 The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA
 apt lists remain on disk but no NVIDIA PCI device exists.
@@ -61,27 +74,29 @@ apt lists remain on disk but no NVIDIA PCI device exists.
 | / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
 | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
 
-Models live under `/home/dank/models`; the ROCmFPX build tree under
-`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14
-toolchain under `~/.local/share/rocm-7.14.0a20260624`.
+Models live under `/home/dank/models`; the serving build tree under
+`/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is
+still on disk), both sharing the self-contained ROCm 7.14 nightly toolchain
+under `~/.local/share/rocm-7.14.0a20260624` — which also carries `librccl.so.1`
+and `hipcc`, so an RCCL-backed rebuild needs no new host packages.
 
 ### Network
 
 | Interface | Type | Speed | IP | Notes |
 |-----------|------|-------|----|-------|
-| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. |
-| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. |
-| wlp8s0 | WiFi | - | - | Unused (carrier down). |
+| eno1 | onboard RTL8126 5GbE | 2500 Mb/s (negotiated) | 192.168.0.235, **static** | LLM endpoint (port 7770). The address `models.yml`, Prometheus and Pi-hole all point at — preserved across the swap. |
+| eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, and it took the lower-metric default route (101, against eno1's 102). Two live addresses and two default routes for one host — either declare it or bring the link down. |
+| wlp14s0 | WiFi 7 (RTL8922AE) | - | - | Renamed with the board (was `wlp8s0`); unused. |
 
 ### OS and Software
 
 | Property | Value |
 |----------|-------|
 | OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
-| Kernel | 7.0.0-30-generic |
+| Kernel | 7.0.0-31-generic (`iommu=pt`) |
 | Python | 3.14.4 |
 | ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
-| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) |
+| Serving stack | mainline llama.cpp `fb27a525` at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714`, RPATH-bundled with the ROCm 7.14 nightly runtime |
 
 The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
 hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -89,19 +104,20 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
 
 ### Active Services
 
-The active inference service is `frank-medium.service`: charlie12345/ROCmFPX
-llama-server on port 7770, layer-split across both R9700s with the MTP draft
-head on ROCm1. See [[frank-medium-rocmfpx]]. The former
-`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its
-unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`.
-The earlier PP worker (`frank-pp-worker.service`) was retired with the
-2026-08-17 redo. See [[deployment]].
+The active inference service is `frank-medium.service`: mainline llama-server on
+port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
+ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
+`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
+its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
+and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
+See [[deployment]].
 
 ## Hardware Relevance
 
-- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft
-  and a 131,072-token f16 KV cache fit comfortably; layer split gives each
-  card about half the layers and its own KV share.
+- 2x 32 GB VRAM: the 27.0 GiB Q8_0 checkpoint plus the 1.06 GiB DFlash2 drafter
+  and the f16 KV pool for 327,680 tokens fit with the split at 32:33; live
+  headroom is ~6.3 GiB on ROCm0 and ~5.2 GiB on ROCm1, which is why a batch or
+  context change needs a re-fit and not a guess.
 - 60 GB system RAM is mostly headroom; nothing in the serving path is
   CPU-resident.
 - The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off
@@ -109,6 +125,7 @@ The earlier PP worker (`frank-pp-worker.service`) was retired with the
 
 ## See Also
 
-- [[frank-medium-rocmfpx]] — the active serving runbook
+- [[frank-medium-llamacpp-dflash2]] — the active serving runbook
+- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
 - [[deployment]] — the deploy flow
 - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory

serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md

@@ -1,7 +1,7 @@
 ---
 type: research
 title: Dual R9700 Cooperation
-description: Why frank-medium's two Radeon AI PRO R9700s run as independent replicas instead of tensor-parallel, which options are ruled out with evidence, and the two hardware paths that restore true TP=2.
+description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
 tags:
   - frank-medium
   - rocm
@@ -14,32 +14,43 @@ timestamp: 2026-09-16
 
 # Dual R9700 Cooperation
 
-Dankserver's `frank-medium` runs as **two independent single-GPU replicas**
-(`:7771` on card #1, `:7772` on card #2) behind an adaptive-routing proxy on
-`:7770`. This page answers why, and what would let the two cards work on one
-model instead. See [[frank-medium-radiance-mxfp4]] for the deployment itself.
+Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a
+layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true
+tensor parallelism was impossible on the previous board (TUF GAMING B650-E
+WIFI), and what the 2026-09-20 board swap changed. See
+[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under
+the old constraint.
 
 ## The short answer
 
-**Card #2 sits behind the B650 chipset, whose bridges cannot route PCIe atomic
-operations.** RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr
-refuses to supply it to a device without atomics — so *no* RCCL collective or
-P2P operation can be dispatched on card #2. Two paths restore cooperation:
+**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
+bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
+hostcall-buffer parameter and ROCr refuses to supply it to a device without
+atomics, so *no* RCCL collective or P2P operation could be dispatched on that
+card — see the [measured matrix](#measured-matrix) and the
+[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
 
-1. **Move card #2 to the free, CPU-attached M.2_2 slot** (~$30 M.2→PCIe x4
-   riser). Cheapest, and bandwidth is not a constraint (below).
-2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class,
-   ~$250-400). Best performance. This board has no x8/x8 mode.
+On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
+bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
+recommended, at B850 rather than X870E money. Both cards now hang off CPU root
+ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
+`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
+the exact precondition this page's own [test plan](#test-plan) names.
 
-The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS
-lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as
-chipset, while an empty slot does not enumerate — so nothing on a running
-system can confirm it until a card is in it. And CPU-attached does not imply
-`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth
-trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what
-the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one
-model" — the layer split already delivers that, so this is a performance
-project, not a capability one.
+What that buys, and what it does not:
+
+- **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp
+  `--split-mode row`. All three died on the same hostcall gate.
+- **Not yet measured.** Nothing has dispatched a collective on this host since
+  the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask
+  production first.
+- **Not needed for capability.** The layer split already puts both cards on one
+  model. This is a performance question: an RCCL-backed tensor split, and the
+  single pooled KV that comes with it.
+- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
+  card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
+  ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
+  and M.2_2 is free.
 
 ## The gate, in source
 
@@ -100,6 +111,45 @@ Three things fall out of that table, none of them in the source alone:
   collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
   x4. The layer split's boundary copy does not care.
 
+## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
+
+`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
+`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
+`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
+LAN/SATA/WiFi:
+
+| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
+|---|---|---|---|---|
+| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
+| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
+| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
+| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
+| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
+| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
+| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
+| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
+
+- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
+  kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
+  `Routing-` bridge. Same one-command diagnosis, opposite answer.
+- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
+  by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
+  card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
+- `rocm-smi --showtopo` still reports `PCIE` between the cards (Navi 48 has no
+  XGMI), they are still in separate IOMMU groups (15 and 20), and both root
+  ports have ACS `ReqRedir+`, so P2P hairpins at the root complex. Plain P2P
+  traffic measured 45.27 GB/s bidirectional on the *old* board — DMA was never
+  the gate, atomics were.
+- Kernel 7.0.0-31-generic with `iommu=pt`. Both were ruled out as causes
+  ([above](#ruled-out-empirically-do-not-re-derive)) and both are unchanged.
+
+**Still unproven.** The gate that matters is ROCclr's `checkAtomicSupport()`
+reading the `kLinkAtomicSupport` link attribute; the ReqEn bit is the same cause
+seen from the kernel side, but no RCCL kernel has been launched since the swap.
+Until the 2-rank probe runs — or a `--split-mode row` boot that does *not* log
+`internal AllReduce init failed ... falling back to meta-backend butterfly` —
+"collectives work here" is a strong expectation, not a measurement.
+
 ## Measured matrix
 
 Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
@@ -121,7 +171,11 @@ RCCL mentions atomics; the runtime log is the only place it appears.
 **`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
 as an option, because vLLM's PP stage handoff is exactly that primitive.
 
-## Software escapes: closed, and one still open
+## Software escapes: closed, and the last one is now moot
+
+Every row below was written under the old board. The last row — the only door
+that was ever open — stopped mattering on 2026-09-20, because the hardware gate
+it existed to route around no longer applies to this host.
 
 | Idea | Finding |
 |---|---|
@@ -138,9 +192,11 @@ as an option, because vLLM's PP stage handoff is exactly that primitive.
 A 27B TP=2 allreduce at batch 4 moves roughly
 $2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
 per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
-or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s). TP is compute-bound here, not
-interconnect-bound. That is the whole reason the M.2 riser is the interesting
-option rather than the board swap.
+or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
+~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
+That is why the cheap option (the M.2 riser) and the expensive one (the board
+swap) were both considered adequate on bandwidth alone — and why the board that
+landed, at x8/x8, clears the arithmetic with room to spare.
 
 ## Ruled out empirically (do not re-derive)
 
@@ -157,6 +213,9 @@ option rather than the board swap.
 
 ## Test plan (when the hardware changes)
 
+**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so
+this is now a standing plan rather than a hypothesis.
+
 The verdict to look for is one line of `lspci`, before touching any model:
 
 ```bash
@@ -193,25 +252,31 @@ on the second card, so the atomics gate never comes up.
 That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one
 process, one model, both cards, 262144 context. Measured against tensor split
 on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 /
-26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a
-build with a working RCCL — which is exactly what the M.2 riser or a board swap
-in [the short answer](#the-short-answer) would buy.
+26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a
+prediction for an RCCL-backed build** — it is what tensor split costs when the
+allreduce degrades to a meta-backend butterfly, and it is the number the
+board-swap path exists to retire.
 
-So the ranking changed on 2026-09-16: software already gets both cards onto one
-model, and the hardware fix is no longer a means to that end — it is a means to
-*tensor* parallelism, which buys ~40 % decode and would let this runtime drop
-layer-split's serial stages entirely.
+The ranking therefore changed twice. On 2026-09-16: software already gets both
+cards onto one model, so the hardware fix stopped being a capability question
+and became a performance one — *tensor* parallelism, ~40 % decode, plus one
+pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
+hardware half of that question was answered by the board swap; the measurement
+half has not run yet, and `GGML_HIP_RCCL` is still `OFF` in
+`scripts/build-llama-rdna4.sh` for exactly the reason this page documents.
 
 ## What would change the answer
 
-- An RCCL build whose kernels stop declaring a hostcall buffer — the one
-  software route that is *not* closed. See the last row of the escape table for
-  the hypothesis, its counter-evidence, and the two-command test.
+- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
+  since 2026-09-20.** That door only mattered while the hardware gate could not
+  be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
+  and its counter-evidence stay in the escape table for the record.
 - The symmetric-kernel path inside RCCL, if a release ever ships the device API
-  (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped
-  release does not.
-- Firmware or an AGESA update that makes the chipset bridges advertise atomic
-  routing — a capability bit, so this would show up in `lspci` on either side.
+  (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
+  nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
+  not.
+- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
+  routing~~ — arrived as new silicon instead.
 - A GPU-side workaround: none found in ROCclr, and the gate is per-device,
   queried once at probe.
 

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -32,15 +32,22 @@ replicas, the proxy and their unit files are retired.
 
 ## Why one process and layer split
 
-The endpoint's whole history is one constraint: **the two cards cannot take
+The endpoint's whole history is one constraint: **the two cards could not take
 part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
-and ROCr refuses to supply it to a device without PCIe atomics; the second
-R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
-every collective that touches it dies with `hipErrorIllegalState`. That rules
-out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
-whose stage handoff is a `send`/`recv` of the hidden state on the device
-backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
-The full matrix is in [[Dual-R9700-Cooperation]].
+and ROCr refuses to supply it to a device without PCIe atomics; on the previous
+board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`,
+read-only), so every collective that touched it died with
+`hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out
+**vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of
+the hidden state on the device backend
+(`vllm/distributed/parallel_state.py`), the very primitive that failed. The
+full matrix is in [[Dual-R9700-Cooperation]].
+
+**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both
+cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read
+`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is
+*unblocked, not proven* — and this service still runs the shape below, which
+needs no collective either way.
 
 llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
 distributed across the cards in order and the boundary activation is a plain
@@ -63,7 +70,7 @@ shape — prefill nearly doubles and one model serves from both cards.
 |---|---|
 | GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
 | VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
-| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
+| Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] |
 | iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
 | CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
 

serverAppConfigs-docs/LLM/index.md

@@ -60,15 +60,17 @@ frank-small, port 7771]
   2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
   orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
   drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
-  behind :7770 (tensor parallelism is impossible on this host — the second card
-  cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token
+  behind :7770 (tensor parallelism was impossible on the old board's second
+  card; the 2026-09-20 swap cleared that gate but the runtime still ships
+  RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token
   slots
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
   deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
   ggz14/radiance-vllm-mxfp4 patches) serving
   just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
   behind a round-robin proxy; kept as the record of the deployment it
-  succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host
+  succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous
+  board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]]
 - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
   2026-09-15): charlie12345/ROCmFPX serving
   kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the

serverAppConfigs-docs/index.md

@@ -120,20 +120,21 @@ The LLM inference fleet on the two hosts. Full index:
   serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
   drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
   AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
-  32:33 over two 163,840-token slots — no tensor parallelism, because the
-  second card cannot dispatch an RCCL collective
+  32:33 over two 163,840-token slots — no tensor parallelism: the runtime ships
+  RCCL-less, and the collective has not been re-probed since the 2026-09-20
+  board swap cleared the atomics gate
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
   MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
   Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
   round-robin proxy; kept as the record of the deployment
   [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence
-  that rules out TP=2 on this host (since 2026-09-15/16)
-- [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel
+  that ruled out TP=2 on the previous board (gate cleared 2026-09-20)
+- [[Dual-R9700-Cooperation]] — why the two R9700s *could not* run tensor-parallel
   together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
-  (collectives *and* P2P `send`/`recv` both fail on card #2, which rules out
-  pipeline parallelism too), what is closed and what is still open, and the two
-  hardware paths — a ~$30 M.2→PCIe x4 riser into the free, CPU-attached M.2_2
-  slot, or a board whose CPU x16 bifurcates to x8/x8
+  (collectives *and* P2P `send`/`recv` both failed on card #2, which ruled out
+  pipeline parallelism too), what was closed, and the re-measured 2026-09-20
+  topology after the ProArt B850-Creator x8/x8 swap — both chains `Routing+`,
+  both endpoints `ReqEn+`, so TP=2 is unblocked but still unprobed
 - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
   charlie12345/ROCmFPX serving kingjones777/
   Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it

serverAppConfigs-docs/log.md

@@ -11,6 +11,40 @@ timestamp: 2026-09-14
 
 # Bundle Change Log
 
+## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone
+- **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO**
+  (was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board
+  bifurcates the CPU GPP ports **x8/x8 at Gen5**, so both cards sit on CPU root
+  ports: card 0 `00:01.1 → 01:00.0 → 02:00.0 → 03:00.0`, card 1
+  `00:01.3 → 05:00.0 → 06:00.0 → 07:00.0`. Every bridge reads
+  `AtomicOpsCap: Routing+` and **both endpoints read `AtomicOpsCtl: ReqEn+`**
+  (`CAP_EXP+28.w = 0x1440`) — the exact precondition
+  [[Dual-R9700-Cooperation]]'s test plan named before any model would be
+  touched. Bandwidth per card ~31.5 GB/s (kernel: `252.056 Gb/s … limited by
+  32.0 GT/s PCIe x8`): card 0 unchanged from Gen4 x16, card 1 up ~16x from
+  Gen4 x1.
+- **Verdict, stated honestly.** vLLM TP=2 / PP=2 and llama.cpp `--split-mode
+  row` are **unblocked, not proven** — no collective has been dispatched since
+  the swap, `GGML_HIP_RCCL` is still `OFF` in `scripts/build-llama-rdna4.sh`,
+  and the 272 t/s "tensor" number on record measured a butterfly fallback, not
+  an allreduce. `librccl.so.1` + `hipcc` already exist in the bundled TheRock
+  toolchain, so an RCCL-backed rebuild needs no new host packages. The probe
+  still owes a stop-and-mask window.
+- **Side effects of the swap, recorded.** GPU BDFs moved (`0b:00.0` → `07:00.0`),
+  iGPU `10:00.0` → `11:00.0`, WiFi interface `wlp8s0` → `wlp14s0`, kernel now
+  7.0.0-31-generic, SMT back **on** (16 logical CPUs; llama.cpp still `--threads
+  8`), and the board's second 5GbE port is cabled as `eno2` = 192.168.0.247 with
+  the lower-metric default route while `eno1` keeps the fleet's static
+  192.168.0.235. Two addresses, two default routes — needs a decision.
+- **Docs cut over** (no service change; frank-medium stayed up and healthy
+  throughout): [[Dual-R9700-Cooperation]] short answer rewritten plus a
+  re-measured 2026-09-20 topology table; [[Dankserver System Specs]] gained a
+  Motherboard section and corrected GPU/iGPU/network/kernel rows;
+  [[frank-medium-llamacpp-dflash2]] hardware row and rationale now split past
+  from present; `AGENTS.md` topology table, the frank-medium section, and the
+  PCIe lesson bullets no longer assert a wall that no longer exists. The
+  dated work-log sections that describe the old board stay as history.
+
 ## 2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert
 - **frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca.** The private
   CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja