From d16e8e554e34003a57904d79bcd11f079971dc02 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 20 Sep 2026 14:39:54 -0700 Subject: [PATCH] docs: dankserver board swap clears the PCIe-atomics wall MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation), every bridge reads AtomicOpsCap Routing+, and both endpoints read AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked, not proven: no collective has been dispatched since the swap, GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record measured a butterfly fallback. - Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20 topology table, rebuild-hypothesis door marked moot, probe plan is now standing work. - Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 -> 07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new eno2 (192.168.0.247) second 5GbE port holding the low-metric default route while eno1 keeps the fleet's static .235. - frank-medium-llamacpp-dflash2: rationale and hardware row split past from present; dated work-log sections stay as history. - AGENTS.md + both indexes: no longer assert a wall that no longer exists. Docs only; frank-medium stayed active and healthy throughout (verified /health). ./tests/run.sh 331/331. --- AGENTS.md | 58 ++++--- .../LLM/Dankserver System Specs.md | 69 +++++---- .../LLM/Dual-R9700-Cooperation.md | 145 +++++++++++++----- .../LLM/frank-medium-llamacpp-dflash2.md | 25 +-- serverAppConfigs-docs/LLM/index.md | 8 +- serverAppConfigs-docs/index.md | 17 +- serverAppConfigs-docs/log.md | 34 ++++ 7 files changed, 246 insertions(+), 110 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d623343..7981930 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770| +|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap — atomic routing verified on both chains), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770| | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | @@ -337,10 +337,13 @@ model's own MTP head (`blk..nextn.*`, `nextn_predict_layers=1`), so **one process across both R9700s**, `--split-mode layer` at a **32:33 tensor split** — balanced (operator call 2026-09-19, the most even split 65 layers allow; ROCm1, the last-layers card that also carries the drafter and output -layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned -work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4 -x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s); -the balanced point trades that lean for symmetry and was accepted. 327680 ctx +layers, holds the extra one). +The predecessor's 35:30 skew (2026-09-17) leaned work onto the wider card +because ROCm1 (then `0b:00.0`) rode the B650 chipset at Gen4 x1 (~1.97 GB/s) +against ROCm0's CPU-root-port Gen4 x16 (~31.5 GB/s); the balanced point traded +that lean for symmetry and was accepted. The 2026-09-20 board swap (ASUS ProArt +B850-Creator WiFi NEO) removed the other axis: both cards are now CPU-root-port +Gen5 x8 (~31.5 GB/s each), at `03:00.0` and `07:00.0`. 327680 ctx over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16 KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was @@ -348,13 +351,18 @@ tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four layers of the split. -**Layer split is why both cards now serve ONE model.** Tensor parallelism is -impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — -its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact -primitive that cannot be dispatched on card #2. llama.cpp's layer split needs +**Layer split is why both cards serve ONE model today.** It was chosen because +tensor parallelism was then impossible (see the PCIe lesson below) and so was +vLLM's pipeline parallelism — its stage handoff is an RCCL `send`/`recv` of the +hidden state, the exact primitive that could not be dispatched on the old +board's card #2. llama.cpp's layer split needs no collective at all: layers are distributed in order and the boundary activation is a per-backend copy, once per forward pass. Measured on this host -(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor. +(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor — but that +tensor number is the cost of a butterfly-fallback allreduce with `GGML_HIP_RCCL` +OFF, not what an RCCL-backed tensor split would do. Since the 2026-09-20 board +swap cleared the atomics gate, that comparison is stale and the probe is the +open work; see [[Dual-R9700-Cooperation]]. This is the box's **only** LLM runtime: the vllm-radiance replica pair and its routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went @@ -396,25 +404,27 @@ linked runbooks. and the rest behind the chipset (usually not) — that asymmetry is what blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL short-circuits at `world_size=1`. -- **The fix is physical — and cheap on this board.** Two paths put both cards - on CPU root ports, which restores the atomics routing and therefore TP=2: an - M.2→PCIe x4 riser into a *CPU-attached* M.2 slot (on dankserver's TUF - GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page; - `PCIEX16_2`, where card #2 sits today, is chipset and x1-mode), or a board - whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B - TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4. - RCCL's P2P `send`/`recv` fails on such a card too, so pipeline parallelism is - not an escape either. Full evidence matrix, the open rebuild hypothesis, and - the test plan: [[Dual-R9700-Cooperation]]. +- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.** + Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser + into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to + x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**, + `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`, + every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact + precondition the old test plan named. Bandwidth was never the constraint (a + 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4 + and under 0.4 ms at Gen5 x8). **What is NOT yet re-measured:** no collective + has been dispatched since the swap, so TP=2 is unblocked, not proven. Full + evidence matrix and the standing probe plan: [[Dual-R9700-Cooperation]]. - **When a GPU collective dies, read the runtime, not the framework.** The message that identified the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and prints the failing HIP call with its return code. -- **The one software path that lets both cards serve ONE model: llama.cpp's - layer split.** vLLM cannot do it here in any of its parallelisms — TP needs - an allreduce, PP needs the same p2p primitive, and both die on the card that - cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no +- **The software path that puts both cards on ONE model without a collective: + llama.cpp's layer split.** On the old board vLLM could not do it in any of its + parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both + died on the card that could not dispatch an RCCL kernel. `--split-mode + layer` in llama.cpp needs no collective: layers are distributed in order and only the boundary activation crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 43052df..4f11986 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -1,14 +1,14 @@ --- type: configuration title: Dankserver System Specs -description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium. +description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium. tags: - dankserver - hardware - r9700 - gfx1201 - rocm -timestamp: 2026-09-15 +timestamp: 2026-09-20 --- # Dankserver System Specs @@ -22,6 +22,18 @@ two AMD Radeon AI PRO R9700 cards. ## Hardware Specifications +### Motherboard + +| Property | Value | +|----------|-------| +| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 | +| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) | +| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3`; ~31.5 GB/s per card | +| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board | +| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 | +| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less), trained **x2** at Gen5 (63 Gb/s ≈ 7.9 GB/s) | +| Onboard | 2x Realtek RTL8126 5GbE, RTL8922AE WiFi 7, PROM21 SATA controller | + ### CPU | Property | Value | @@ -29,7 +41,7 @@ two AMD Radeon AI PRO R9700 cards. | Architecture | x86_64 | | Vendor | AuthenticAMD | | Model | AMD Ryzen 7 7700X | -| Cores | 8 cores / 8 threads (1 socket, SMT off) | +| Cores | 8 cores / 16 threads (1 socket) — SMT is **on**; the BIOS defaults reset with the board swap. `llama.cpp` still runs `--threads 8 --threads-batch 8`, so this is headroom, not a change in the serving shape | | NUMA Nodes | 1 (node 0) | ### GPUs @@ -38,10 +50,11 @@ two AMD Radeon AI PRO R9700 cards. |----------|-------| | Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) | | Architecture | gfx1201 (RDNA4) | -| VRAM | 32,624 MiB each | -| Link | PCIe, no XGMI; different IOMMU groups | -| PCI | 03:00.0 and 0b:00.0 | -| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP | +| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) | +| Link | PCIe 5.0 **x8** each (CPU GPP `00:01.1` / `00:01.3`), no XGMI; separate IOMMU groups (15 and 20), ACS `ReqRedir+` on both root ports, so P2P hairpins at the root complex | +| Atomic ops | `Routing+` on all four bridges per chain; `ReqEn+` on both endpoints (`CAP_EXP+28.w = 0x1440`) | +| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) | +| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2, hidden from HIP | The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA apt lists remain on disk but no NVIDIA PCI device exists. @@ -61,27 +74,29 @@ apt lists remain on disk but no NVIDIA PCI device exists. | / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% | | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | -Models live under `/home/dank/models`; the ROCmFPX build tree under -`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14 -toolchain under `~/.local/share/rocm-7.14.0a20260624`. +Models live under `/home/dank/models`; the serving build tree under +`/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is +still on disk), both sharing the self-contained ROCm 7.14 nightly toolchain +under `~/.local/share/rocm-7.14.0a20260624` — which also carries `librccl.so.1` +and `hipcc`, so an RCCL-backed rebuild needs no new host packages. ### Network | Interface | Type | Speed | IP | Notes | |-----------|------|-------|----|-------| -| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. | -| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. | -| wlp8s0 | WiFi | - | - | Unused (carrier down). | +| eno1 | onboard RTL8126 5GbE | 2500 Mb/s (negotiated) | 192.168.0.235, **static** | LLM endpoint (port 7770). The address `models.yml`, Prometheus and Pi-hole all point at — preserved across the swap. | +| eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, and it took the lower-metric default route (101, against eno1's 102). Two live addresses and two default routes for one host — either declare it or bring the link down. | +| wlp14s0 | WiFi 7 (RTL8922AE) | - | - | Renamed with the board (was `wlp8s0`); unused. | ### OS and Software | Property | Value | |----------|-------| | OS | Ubuntu 26.04 LTS (Resolute Raccoon) | -| Kernel | 7.0.0-30-generic | +| Kernel | 7.0.0-31-generic (`iommu=pt`) | | Python | 3.14.4 | | ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) | -| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) | +| Serving stack | mainline llama.cpp `fb27a525` at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714`, RPATH-bundled with the ROCm 7.14 nightly runtime | The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to hijack `libhsa-runtime64` were purged on 2026-09-15 — see @@ -89,19 +104,20 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see ### Active Services -The active inference service is `frank-medium.service`: charlie12345/ROCmFPX -llama-server on port 7770, layer-split across both R9700s with the MTP draft -head on ROCm1. See [[frank-medium-rocmfpx]]. The former -`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its -unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. -The earlier PP worker (`frank-pp-worker.service`) was retired with the -2026-08-17 redo. See [[deployment]]. +The active inference service is `frank-medium.service`: mainline llama-server on +port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to +ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former +`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap; +its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`, +and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`. +See [[deployment]]. ## Hardware Relevance -- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft - and a 131,072-token f16 KV cache fit comfortably; layer split gives each - card about half the layers and its own KV share. +- 2x 32 GB VRAM: the 27.0 GiB Q8_0 checkpoint plus the 1.06 GiB DFlash2 drafter + and the f16 KV pool for 327,680 tokens fit with the split at 32:33; live + headroom is ~6.3 GiB on ROCm0 and ~5.2 GiB on ROCm1, which is why a batch or + context change needs a re-fit and not a guess. - 60 GB system RAM is mostly headroom; nothing in the serving path is CPU-resident. - The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off @@ -109,6 +125,7 @@ The earlier PP worker (`frank-pp-worker.service`) was retired with the ## See Also -- [[frank-medium-rocmfpx]] — the active serving runbook +- [[frank-medium-llamacpp-dflash2]] — the active serving runbook +- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed - [[deployment]] — the deploy flow - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md index bc5c06e..5ad3f19 100644 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md @@ -1,7 +1,7 @@ --- type: research title: Dual R9700 Cooperation -description: Why frank-medium's two Radeon AI PRO R9700s run as independent replicas instead of tensor-parallel, which options are ruled out with evidence, and the two hardware paths that restore true TP=2. +description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed. tags: - frank-medium - rocm @@ -14,32 +14,43 @@ timestamp: 2026-09-16 # Dual R9700 Cooperation -Dankserver's `frank-medium` runs as **two independent single-GPU replicas** -(`:7771` on card #1, `:7772` on card #2) behind an adaptive-routing proxy on -`:7770`. This page answers why, and what would let the two cards work on one -model instead. See [[frank-medium-radiance-mxfp4]] for the deployment itself. +Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a +layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true +tensor parallelism was impossible on the previous board (TUF GAMING B650-E +WIFI), and what the 2026-09-20 board swap changed. See +[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under +the old constraint. ## The short answer -**Card #2 sits behind the B650 chipset, whose bridges cannot route PCIe atomic -operations.** RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr -refuses to supply it to a device without atomics — so *no* RCCL collective or -P2P operation can be dispatched on card #2. Two paths restore cooperation: +**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose +bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden +hostcall-buffer parameter and ROCr refuses to supply it to a device without +atomics, so *no* RCCL collective or P2P operation could be dispatched on that +card — see the [measured matrix](#measured-matrix) and the +[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below. -1. **Move card #2 to the free, CPU-attached M.2_2 slot** (~$30 M.2→PCIe x4 - riser). Cheapest, and bandwidth is not a constraint (below). -2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class, - ~$250-400). Best performance. This board has no x8/x8 mode. +On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which +bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page +recommended, at B850 rather than X870E money. Both cards now hang off CPU root +ports (`00:01.1`, `00:01.3`), every bridge on both chains reads +`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** — +the exact precondition this page's own [test plan](#test-plan) names. -The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS -lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as -chipset, while an empty slot does not enumerate — so nothing on a running -system can confirm it until a card is in it. And CPU-attached does not imply -`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth -trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what -the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one -model" — the layer split already delivers that, so this is a performance -project, not a capability one. +What that buys, and what it does not: + +- **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp + `--split-mode row`. All three died on the same hostcall gate. +- **Not yet measured.** Nothing has dispatched a collective on this host since + the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask + production first. +- **Not needed for capability.** The layer split already puts both cards on one + model. This is a performance question: an RCCL-backed tensor split, and the + single pooled KV that comes with it. +- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each): + card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1, + ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now + and M.2_2 is free. ## The gate, in source @@ -100,6 +111,45 @@ Three things fall out of that table, none of them in the source alone: collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at x4. The layer split's boundary copy does not care. +## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO) + +`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` → +`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` → +`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only +LAN/SATA/WiFi: + +| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | +|---|---|---|---|---| +| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 | +| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 | +| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 | +| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | +| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 | +| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 | +| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 | +| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | + +- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the + kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no + `Routing-` bridge. Same one-command diagnosis, opposite answer. +- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited + by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per + card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8". +- `rocm-smi --showtopo` still reports `PCIE` between the cards (Navi 48 has no + XGMI), they are still in separate IOMMU groups (15 and 20), and both root + ports have ACS `ReqRedir+`, so P2P hairpins at the root complex. Plain P2P + traffic measured 45.27 GB/s bidirectional on the *old* board — DMA was never + the gate, atomics were. +- Kernel 7.0.0-31-generic with `iommu=pt`. Both were ruled out as causes + ([above](#ruled-out-empirically-do-not-re-derive)) and both are unchanged. + +**Still unproven.** The gate that matters is ROCclr's `checkAtomicSupport()` +reading the `kLinkAtomicSupport` link attribute; the ReqEn bit is the same cause +seen from the kernel side, but no RCCL kernel has been launched since the swap. +Until the 2-rank probe runs — or a `--split-mode row` boot that does *not* log +`internal AllReduce init failed ... falling back to meta-backend butterfly` — +"collectives work here" is a strong expectation, not a measurement. + ## Measured matrix Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no @@ -121,7 +171,11 @@ RCCL mentions atomics; the runtime log is the only place it appears. **`send`/`recv` failing is the decisive result**: it kills pipeline parallelism as an option, because vLLM's PP stage handoff is exactly that primitive. -## Software escapes: closed, and one still open +## Software escapes: closed, and the last one is now moot + +Every row below was written under the old board. The last row — the only door +that was ever open — stopped mattering on 2026-09-20, because the hardware gate +it existed to route around no longer applies to this host. | Idea | Finding | |---|---| @@ -138,9 +192,11 @@ as an option, because vLLM's PP stage handoff is exactly that primitive. A 27B TP=2 allreduce at batch 4 moves roughly $2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$ per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**, -or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s). TP is compute-bound here, not -interconnect-bound. That is the whole reason the M.2 riser is the interesting -option rather than the board swap. +or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8 +~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound. +That is why the cheap option (the M.2 riser) and the expensive one (the board +swap) were both considered adequate on bandwidth alone — and why the board that +landed, at x8/x8, clears the arithmetic with room to spare. ## Ruled out empirically (do not re-derive) @@ -157,6 +213,9 @@ option rather than the board swap. ## Test plan (when the hardware changes) +**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so +this is now a standing plan rather than a hypothesis. + The verdict to look for is one line of `lspci`, before touching any model: ```bash @@ -193,25 +252,31 @@ on the second card, so the atomics gate never comes up. That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards, 262144 context. Measured against tensor split on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / -26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a -build with a working RCCL — which is exactly what the M.2 riser or a board swap -in [the short answer](#the-short-answer) would buy. +26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a +prediction for an RCCL-backed build** — it is what tensor split costs when the +allreduce degrades to a meta-backend butterfly, and it is the number the +board-swap path exists to retire. -So the ranking changed on 2026-09-16: software already gets both cards onto one -model, and the hardware fix is no longer a means to that end — it is a means to -*tensor* parallelism, which buys ~40 % decode and would let this runtime drop -layer-split's serial stages entirely. +The ranking therefore changed twice. On 2026-09-16: software already gets both +cards onto one model, so the hardware fix stopped being a capability question +and became a performance one — *tensor* parallelism, ~40 % decode, plus one +pooled KV instead of two independent 122k-token pools. On 2026-09-20 the +hardware half of that question was answered by the board swap; the measurement +half has not run yet, and `GGML_HIP_RCCL` is still `OFF` in +`scripts/build-llama-rdna4.sh` for exactly the reason this page documents. ## What would change the answer -- An RCCL build whose kernels stop declaring a hostcall buffer — the one - software route that is *not* closed. See the last row of the escape table for - the hypothesis, its counter-evidence, and the two-command test. +- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot + since 2026-09-20.** That door only mattered while the hardware gate could not + be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis + and its counter-evidence stay in the escape table for the record. - The symmetric-kernel path inside RCCL, if a release ever ships the device API - (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped - release does not. -- Firmware or an AGESA update that makes the chipset bridges advertise atomic - routing — a capability bit, so this would show up in `lspci` on either side. + (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock + nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does + not. +- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic + routing~~ — arrived as new silicon instead. - A GPU-side workaround: none found in ROCclr, and the gate is per-device, queried once at probe. diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 448190c..ea4df1d 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -32,15 +32,22 @@ replicas, the proxy and their unit files are retired. ## Why one process and layer split -The endpoint's whole history is one constraint: **the two cards cannot take +The endpoint's whole history is one constraint: **the two cards could not take part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer, -and ROCr refuses to supply it to a device without PCIe atomics; the second -R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so -every collective that touches it dies with `hipErrorIllegalState`. That rules -out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**, -whose stage handoff is a `send`/`recv` of the hidden state on the device -backend (`vllm/distributed/parallel_state.py`), the very primitive that fails. -The full matrix is in [[Dual-R9700-Cooperation]]. +and ROCr refuses to supply it to a device without PCIe atomics; on the previous +board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`, +read-only), so every collective that touched it died with +`hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out +**vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of +the hidden state on the device backend +(`vllm/distributed/parallel_state.py`), the very primitive that failed. The +full matrix is in [[Dual-R9700-Cooperation]]. + +**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both +cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read +`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is +*unblocked, not proven* — and this service still runs the shape below, which +needs no collective either way. llama.cpp's `--split-mode layer` needs **no collective at all**: layers are distributed across the cards in order and the boundary activation is a plain @@ -63,7 +70,7 @@ shape — prefill nearly doubles and one model serves from both cards. |---|---| | GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) | | VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it | -| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** | +| Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] | | iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` | | CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap | diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 0f0e7ef..b1d08b8 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -60,15 +60,17 @@ frank-small, port 7771] 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process - behind :7770 (tensor parallelism is impossible on this host — the second card - cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token + behind :7770 (tensor parallelism was impossible on the old board's second + card; the 2026-09-20 swap cleared that gate but the runtime still ships + RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token slots - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + ggz14/radiance-vllm-mxfp4 patches) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas behind a round-robin proxy; kept as the record of the deployment it - succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host + succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous + board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]] - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era, 2026-09-15): charlie12345/ROCmFPX serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 8e99a3c..56a7d3c 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -120,20 +120,21 @@ The LLM inference fleet on the two hosts. Full index: serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced - 32:33 over two 163,840-token slots — no tensor parallelism, because the - second card cannot dispatch an RCCL collective + 32:33 over two 163,840-token slots — no tensor parallelism: the runtime ships + RCCL-less, and the collective has not been re-probed since the 2026-09-20 + board swap cleared the atomics gate - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a round-robin proxy; kept as the record of the deployment [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence - that rules out TP=2 on this host (since 2026-09-15/16) -- [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel + that ruled out TP=2 on the previous board (gate cleared 2026-09-20) +- [[Dual-R9700-Cooperation]] — why the two R9700s *could not* run tensor-parallel together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix - (collectives *and* P2P `send`/`recv` both fail on card #2, which rules out - pipeline parallelism too), what is closed and what is still open, and the two - hardware paths — a ~$30 M.2→PCIe x4 riser into the free, CPU-attached M.2_2 - slot, or a board whose CPU x16 bifurcates to x8/x8 + (collectives *and* P2P `send`/`recv` both failed on card #2, which ruled out + pipeline parallelism too), what was closed, and the re-measured 2026-09-20 + topology after the ProArt B850-Creator x8/x8 swap — both chains `Routing+`, + both endpoints `ReqEn+`, so TP=2 is unblocked but still unprobed - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era): charlie12345/ROCmFPX serving kingjones777/ Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 167d135..f4d4e05 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,40 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone +- **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO** + (was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board + bifurcates the CPU GPP ports **x8/x8 at Gen5**, so both cards sit on CPU root + ports: card 0 `00:01.1 → 01:00.0 → 02:00.0 → 03:00.0`, card 1 + `00:01.3 → 05:00.0 → 06:00.0 → 07:00.0`. Every bridge reads + `AtomicOpsCap: Routing+` and **both endpoints read `AtomicOpsCtl: ReqEn+`** + (`CAP_EXP+28.w = 0x1440`) — the exact precondition + [[Dual-R9700-Cooperation]]'s test plan named before any model would be + touched. Bandwidth per card ~31.5 GB/s (kernel: `252.056 Gb/s … limited by + 32.0 GT/s PCIe x8`): card 0 unchanged from Gen4 x16, card 1 up ~16x from + Gen4 x1. +- **Verdict, stated honestly.** vLLM TP=2 / PP=2 and llama.cpp `--split-mode + row` are **unblocked, not proven** — no collective has been dispatched since + the swap, `GGML_HIP_RCCL` is still `OFF` in `scripts/build-llama-rdna4.sh`, + and the 272 t/s "tensor" number on record measured a butterfly fallback, not + an allreduce. `librccl.so.1` + `hipcc` already exist in the bundled TheRock + toolchain, so an RCCL-backed rebuild needs no new host packages. The probe + still owes a stop-and-mask window. +- **Side effects of the swap, recorded.** GPU BDFs moved (`0b:00.0` → `07:00.0`), + iGPU `10:00.0` → `11:00.0`, WiFi interface `wlp8s0` → `wlp14s0`, kernel now + 7.0.0-31-generic, SMT back **on** (16 logical CPUs; llama.cpp still `--threads + 8`), and the board's second 5GbE port is cabled as `eno2` = 192.168.0.247 with + the lower-metric default route while `eno1` keeps the fleet's static + 192.168.0.235. Two addresses, two default routes — needs a decision. +- **Docs cut over** (no service change; frank-medium stayed up and healthy + throughout): [[Dual-R9700-Cooperation]] short answer rewritten plus a + re-measured 2026-09-20 topology table; [[Dankserver System Specs]] gained a + Motherboard section and corrected GPU/iGPU/network/kernel rows; + [[frank-medium-llamacpp-dflash2]] hardware row and rationale now split past + from present; `AGENTS.md` topology table, the frank-medium section, and the + PCIe lesson bullets no longer assert a wall that no longer exists. The + dated work-log sections that describe the old board stay as history. + ## 2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert - **frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca.** The private CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja