dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: dankserver board swap clears the PCIe-atomics wall

Browse Source 
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.

- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
  topology table, rebuild-hypothesis door marked moot, probe plan is now
  standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
  07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
  eno2 (192.168.0.247) second 5GbE port holding the low-metric default
  route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
  from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.

Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
This commit is contained in:
Frank committed 2026-09-20 14:39:54 -07:00
1 parent a43ab90105
commit d16e8e554e
7 files changed
+246 -110
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+34 -24
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; ROCm0 Gen4 x16, ROCm1 Gen4 x1 chipset path), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770| |Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap — atomic routing verified on both chains), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
@@ -337,10 +337,13 @@ model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
**one process across both R9700s**, `--split-mode layer` at a **32:33 tensor **one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
split** — balanced (operator call 2026-09-19, the most even split 65 layers split** — balanced (operator call 2026-09-19, the most even split 65 layers
allow; ROCm1, the last-layers card that also carries the drafter and output allow; ROCm1, the last-layers card that also carries the drafter and output
layers, holds the extra one). The predecessor's 35:30 skew (2026-09-17) leaned layers, holds the extra one).
work onto the x16 card because ROCm1 (0b:00.0) rides the B650 chipset at Gen4 The predecessor's 35:30 skew (2026-09-17) leaned work onto the wider card
x1 (~1.97 GB/s) while ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s); because ROCm1 (then `0b:00.0`) rode the B650 chipset at Gen4 x1 (~1.97 GB/s)
the balanced point trades that lean for symmetry and was accepted. 327680 ctx against ROCm0's CPU-root-port Gen4 x16 (~31.5 GB/s); the balanced point traded
that lean for symmetry and was accepted. The 2026-09-20 board swap (ASUS ProArt
B850-Creator WiFi NEO) removed the other axis: both cards are now CPU-root-port
Gen5 x8 (~31.5 GB/s each), at `03:00.0` and `07:00.0`. 327680 ctx
over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16 no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
@@ -348,13 +351,18 @@ tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split. layers of the split.
**Layer split is why both cards now serve ONE model.** Tensor parallelism is **Layer split is why both cards serve ONE model today.** It was chosen because
impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — tensor parallelism was then impossible (see the PCIe lesson below) and so was
its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact vLLM's pipeline parallelism — its stage handoff is an RCCL `send`/`recv` of the
primitive that cannot be dispatched on card #2. llama.cpp's layer split needs hidden state, the exact primitive that could not be dispatched on the old
board's card #2. llama.cpp's layer split needs
no collective at all: layers are distributed in order and the boundary no collective at all: layers are distributed in order and the boundary
activation is a per-backend copy, once per forward pass. Measured on this host activation is a per-backend copy, once per forward pass. Measured on this host
(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor. (the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor — but that
tensor number is the cost of a butterfly-fallback allreduce with `GGML_HIP_RCCL`
OFF, not what an RCCL-backed tensor split would do. Since the 2026-09-20 board
swap cleared the atomics gate, that comparison is stale and the probe is the
open work; see [[Dual-R9700-Cooperation]].
This is the box's **only** LLM runtime: the vllm-radiance replica pair and its This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
@@ -396,25 +404,27 @@ linked runbooks.
and the rest behind the chipset (usually not) — that asymmetry is what and the rest behind the chipset (usually not) — that asymmetry is what
blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
short-circuits at `world_size=1`. short-circuits at `world_size=1`.
- **The fix is physical — and cheap on this board.** Two paths put both cards - **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
on CPU root ports, which restores the atomics routing and therefore TP=2: an Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
M.2→PCIe x4 riser into a *CPU-attached* M.2 slot (on dankserver's TUF into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page; x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
`PCIEX16_2`, where card #2 sits today, is chipset and x1-mode), or a board `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4. precondition the old test plan named. Bandwidth was never the constraint (a
RCCL's P2P `send`/`recv` fails on such a card too, so pipeline parallelism is 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
not an escape either. Full evidence matrix, the open rebuild hypothesis, and and under 0.4 ms at Gen5 x8). **What is NOT yet re-measured:** no collective
the test plan: [[Dual-R9700-Cooperation]]. has been dispatched since the swap, so TP=2 is unblocked, not proven. Full
evidence matrix and the standing probe plan: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The - **When a GPU collective dies, read the runtime, not the framework.** The
message that identified the cause came from `AMD_LOG_LEVEL=3` message that identified the cause came from `AMD_LOG_LEVEL=3`
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
prints the failing HIP call with its return code. prints the failing HIP call with its return code.
- **The one software path that lets both cards serve ONE model: llama.cpp's - **The software path that puts both cards on ONE model without a collective:
layer split.** vLLM cannot do it here in any of its parallelisms — TP needs llama.cpp's layer split.** On the old board vLLM could not do it in any of its
an allreduce, PP needs the same p2p primitive, and both die on the card that parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no died on the card that could not dispatch an RCCL kernel. `--split-mode
layer` in llama.cpp needs no
collective: layers are distributed in order and only the boundary activation collective: layers are distributed in order and only the boundary activation
crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+43 -26
View File
@@ -1,14 +1,14 @@
--- ---
type: configuration type: configuration
title: Dankserver System Specs title: Dankserver System Specs
description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium. description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
tags: tags:
- dankserver - dankserver
- hardware - hardware
- r9700 - r9700
- gfx1201 - gfx1201
- rocm - rocm
timestamp: 2026-09-15 timestamp: 2026-09-20
--- ---
# Dankserver System Specs # Dankserver System Specs
@@ -22,6 +22,18 @@ two AMD Radeon AI PRO R9700 cards.
## Hardware Specifications ## Hardware Specifications
### Motherboard
| Property | Value |
|----------|-------|
| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3`; ~31.5 GB/s per card |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board |
| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less), trained **x2** at Gen5 (63 Gb/s ≈ 7.9 GB/s) |
| Onboard | 2x Realtek RTL8126 5GbE, RTL8922AE WiFi 7, PROM21 SATA controller |
### CPU ### CPU
| Property | Value | | Property | Value |
@@ -29,7 +41,7 @@ two AMD Radeon AI PRO R9700 cards.
| Architecture | x86_64 | | Architecture | x86_64 |
| Vendor | AuthenticAMD | | Vendor | AuthenticAMD |
| Model | AMD Ryzen 7 7700X | | Model | AMD Ryzen 7 7700X |
| Cores | 8 cores / 8 threads (1 socket, SMT off) | | Cores | 8 cores / 16 threads (1 socket) — SMT is **on**; the BIOS defaults reset with the board swap. `llama.cpp` still runs `--threads 8 --threads-batch 8`, so this is headroom, not a change in the serving shape |
| NUMA Nodes | 1 (node 0) | | NUMA Nodes | 1 (node 0) |
### GPUs ### GPUs
@@ -38,10 +50,11 @@ two AMD Radeon AI PRO R9700 cards.
|----------|-------| |----------|-------|
| Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) | | Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) |
| Architecture | gfx1201 (RDNA4) | | Architecture | gfx1201 (RDNA4) |
| VRAM | 32,624 MiB each | | VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
| Link | PCIe, no XGMI; different IOMMU groups | | Link | PCIe 5.0 **x8** each (CPU GPP `00:01.1` / `00:01.3`), no XGMI; separate IOMMU groups (15 and 20), ACS `ReqRedir+` on both root ports, so P2P hairpins at the root complex |
| PCI | 03:00.0 and 0b:00.0 | | Atomic ops | `Routing+` on all four bridges per chain; `ReqEn+` on both endpoints (`CAP_EXP+28.w = 0x1440`) |
| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP | | PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2, hidden from HIP |
The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA
apt lists remain on disk but no NVIDIA PCI device exists. apt lists remain on disk but no NVIDIA PCI device exists.
@@ -61,27 +74,29 @@ apt lists remain on disk but no NVIDIA PCI device exists.
| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% | | / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
Models live under `/home/dank/models`; the ROCmFPX build tree under Models live under `/home/dank/models`; the serving build tree under
`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14 `/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is
toolchain under `~/.local/share/rocm-7.14.0a20260624`. still on disk), both sharing the self-contained ROCm 7.14 nightly toolchain
under `~/.local/share/rocm-7.14.0a20260624` — which also carries `librccl.so.1`
and `hipcc`, so an RCCL-backed rebuild needs no new host packages.
### Network ### Network
| Interface | Type | Speed | IP | Notes | | Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------| |-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. | | eno1 | onboard RTL8126 5GbE | 2500 Mb/s (negotiated) | 192.168.0.235, **static** | LLM endpoint (port 7770). The address `models.yml`, Prometheus and Pi-hole all point at — preserved across the swap. |
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. | | eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, and it took the lower-metric default route (101, against eno1's 102). Two live addresses and two default routes for one host — either declare it or bring the link down. |
| wlp8s0 | WiFi | - | - | Unused (carrier down). | | wlp14s0 | WiFi 7 (RTL8922AE) | - | - | Renamed with the board (was `wlp8s0`); unused. |
### OS and Software ### OS and Software
| Property | Value | | Property | Value |
|----------|-------| |----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) | | OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-30-generic | | Kernel | 7.0.0-31-generic (`iommu=pt`) |
| Python | 3.14.4 | | Python | 3.14.4 |
| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) | | ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) | | Serving stack | mainline llama.cpp `fb27a525` at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714`, RPATH-bundled with the ROCm 7.14 nightly runtime |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -89,19 +104,20 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
### Active Services ### Active Services
The active inference service is `frank-medium.service`: charlie12345/ROCmFPX The active inference service is `frank-medium.service`: mainline llama-server on
llama-server on port 7770, layer-split across both R9700s with the MTP draft port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
head on ROCm1. See [[frank-medium-rocmfpx]]. The former ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its `frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
The earlier PP worker (`frank-pp-worker.service`) was retired with the and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
2026-08-17 redo. See [[deployment]]. See [[deployment]].
## Hardware Relevance ## Hardware Relevance
- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft - 2x 32 GB VRAM: the 27.0 GiB Q8_0 checkpoint plus the 1.06 GiB DFlash2 drafter
and a 131,072-token f16 KV cache fit comfortably; layer split gives each and the f16 KV pool for 327,680 tokens fit with the split at 32:33; live
card about half the layers and its own KV share. headroom is ~6.3 GiB on ROCm0 and ~5.2 GiB on ROCm1, which is why a batch or
context change needs a re-fit and not a guess.
- 60 GB system RAM is mostly headroom; nothing in the serving path is - 60 GB system RAM is mostly headroom; nothing in the serving path is
CPU-resident. CPU-resident.
- The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off - The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off
@@ -109,6 +125,7 @@ The earlier PP worker (`frank-pp-worker.service`) was retired with the
## See Also ## See Also
- [[frank-medium-rocmfpx]] — the active serving runbook - [[frank-medium-llamacpp-dflash2]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow - [[deployment]] — the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
+105 -40
View File
@@ -1,7 +1,7 @@
--- ---
type: research type: research
title: Dual R9700 Cooperation title: Dual R9700 Cooperation
description: Why frank-medium's two Radeon AI PRO R9700s run as independent replicas instead of tensor-parallel, which options are ruled out with evidence, and the two hardware paths that restore true TP=2. description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
tags: tags:
- frank-medium - frank-medium
- rocm - rocm
@@ -14,32 +14,43 @@ timestamp: 2026-09-16
# Dual R9700 Cooperation # Dual R9700 Cooperation
Dankserver's `frank-medium` runs as **two independent single-GPU replicas** Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a
(`:7771` on card #1, `:7772` on card #2) behind an adaptive-routing proxy on layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true
`:7770`. This page answers why, and what would let the two cards work on one tensor parallelism was impossible on the previous board (TUF GAMING B650-E
model instead. See [[frank-medium-radiance-mxfp4]] for the deployment itself. WIFI), and what the 2026-09-20 board swap changed. See
[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under
the old constraint.
## The short answer ## The short answer
**Card #2 sits behind the B650 chipset, whose bridges cannot route PCIe atomic **The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
operations.** RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
refuses to supply it to a device without atomics — so *no* RCCL collective or hostcall-buffer parameter and ROCr refuses to supply it to a device without
P2P operation can be dispatched on card #2. Two paths restore cooperation: atomics, so *no* RCCL collective or P2P operation could be dispatched on that
card — see the [measured matrix](#measured-matrix) and the
[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
1. **Move card #2 to the free, CPU-attached M.2_2 slot** (~$30 M.2→PCIe x4 On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
riser). Cheapest, and bandwidth is not a constraint (below). bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class, recommended, at B850 rather than X870E money. Both cards now hang off CPU root
~$250-400). Best performance. This board has no x8/x8 mode. ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
the exact precondition this page's own [test plan](#test-plan) names.
The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS What that buys, and what it does not:
lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as
chipset, while an empty slot does not enumerate — so nothing on a running - **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp
system can confirm it until a card is in it. And CPU-attached does not imply `--split-mode row`. All three died on the same hostcall gate.
`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth - **Not yet measured.** Nothing has dispatched a collective on this host since
trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask
the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one production first.
model" — the layer split already delivers that, so this is a performance - **Not needed for capability.** The layer split already puts both cards on one
project, not a capability one. model. This is a performance question: an RCCL-backed tensor split, and the
single pooled KV that comes with it.
- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
and M.2_2 is free.
## The gate, in source ## The gate, in source
@@ -100,6 +111,45 @@ Three things fall out of that table, none of them in the source alone:
collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
x4. The layer split's boundary copy does not care. x4. The layer split's boundary copy does not care.
## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
LAN/SATA/WiFi:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
`Routing-` bridge. Same one-command diagnosis, opposite answer.
- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
- `rocm-smi --showtopo` still reports `PCIE` between the cards (Navi 48 has no
XGMI), they are still in separate IOMMU groups (15 and 20), and both root
ports have ACS `ReqRedir+`, so P2P hairpins at the root complex. Plain P2P
traffic measured 45.27 GB/s bidirectional on the *old* board — DMA was never
the gate, atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt`. Both were ruled out as causes
([above](#ruled-out-empirically-do-not-re-derive)) and both are unchanged.
**Still unproven.** The gate that matters is ROCclr's `checkAtomicSupport()`
reading the `kLinkAtomicSupport` link attribute; the ReqEn bit is the same cause
seen from the kernel side, but no RCCL kernel has been launched since the swap.
Until the 2-rank probe runs — or a `--split-mode row` boot that does *not* log
`internal AllReduce init failed ... falling back to meta-backend butterfly` —
"collectives work here" is a strong expectation, not a measurement.
## Measured matrix ## Measured matrix
Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
@@ -121,7 +171,11 @@ RCCL mentions atomics; the runtime log is the only place it appears.
**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism **`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
as an option, because vLLM's PP stage handoff is exactly that primitive. as an option, because vLLM's PP stage handoff is exactly that primitive.
## Software escapes: closed, and one still open ## Software escapes: closed, and the last one is now moot
Every row below was written under the old board. The last row — the only door
that was ever open — stopped mattering on 2026-09-20, because the hardware gate
it existed to route around no longer applies to this host.
| Idea | Finding | | Idea | Finding |
|---|---| |---|---|
@@ -138,9 +192,11 @@ as an option, because vLLM's PP stage handoff is exactly that primitive.
A 27B TP=2 allreduce at batch 4 moves roughly A 27B TP=2 allreduce at batch 4 moves roughly
$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$ $2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**, per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s). TP is compute-bound here, not or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
interconnect-bound. That is the whole reason the M.2 riser is the interesting ~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
option rather than the board swap. That is why the cheap option (the M.2 riser) and the expensive one (the board
swap) were both considered adequate on bandwidth alone — and why the board that
landed, at x8/x8, clears the arithmetic with room to spare.
## Ruled out empirically (do not re-derive) ## Ruled out empirically (do not re-derive)
@@ -157,6 +213,9 @@ option rather than the board swap.
## Test plan (when the hardware changes) ## Test plan (when the hardware changes)
**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so
this is now a standing plan rather than a hypothesis.
The verdict to look for is one line of `lspci`, before touching any model: The verdict to look for is one line of `lspci`, before touching any model:
```bash ```bash
@@ -193,25 +252,31 @@ on the second card, so the atomics gate never comes up.
That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one
process, one model, both cards, 262144 context. Measured against tensor split process, one model, both cards, 262144 context. Measured against tensor split
on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 /
26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a 26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a
build with a working RCCL — which is exactly what the M.2 riser or a board swap prediction for an RCCL-backed build** — it is what tensor split costs when the
in [the short answer](#the-short-answer) would buy. allreduce degrades to a meta-backend butterfly, and it is the number the
board-swap path exists to retire.
So the ranking changed on 2026-09-16: software already gets both cards onto one The ranking therefore changed twice. On 2026-09-16: software already gets both
model, and the hardware fix is no longer a means to that end — it is a means to cards onto one model, so the hardware fix stopped being a capability question
*tensor* parallelism, which buys ~40 % decode and would let this runtime drop and became a performance one — *tensor* parallelism, ~40 % decode, plus one
layer-split's serial stages entirely. pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
hardware half of that question was answered by the board swap; the measurement
half has not run yet, and `GGML_HIP_RCCL` is still `OFF` in
`scripts/build-llama-rdna4.sh` for exactly the reason this page documents.
## What would change the answer ## What would change the answer
- An RCCL build whose kernels stop declaring a hostcall buffer — the one - ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
software route that is *not* closed. See the last row of the escape table for since 2026-09-20.** That door only mattered while the hardware gate could not
the hypothesis, its counter-evidence, and the two-command test. be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
and its counter-evidence stay in the escape table for the record.
- The symmetric-kernel path inside RCCL, if a release ever ships the device API - The symmetric-kernel path inside RCCL, if a release ever ships the device API
(`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
release does not. nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
- Firmware or an AGESA update that makes the chipset bridges advertise atomic not.
routing — a capability bit, so this would show up in `lspci` on either side. - ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
routing~~ — arrived as new silicon instead.
- A GPU-side workaround: none found in ROCclr, and the gate is per-device, - A GPU-side workaround: none found in ROCclr, and the gate is per-device,
queried once at probe. queried once at probe.
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+16 -9
View File
@@ -32,15 +32,22 @@ replicas, the proxy and their unit files are retired.
## Why one process and layer split ## Why one process and layer split
The endpoint's whole history is one constraint: **the two cards cannot take The endpoint's whole history is one constraint: **the two cards could not take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer, part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; the second and ROCr refuses to supply it to a device without PCIe atomics; on the previous
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`,
every collective that touches it dies with `hipErrorIllegalState`. That rules read-only), so every collective that touched it died with
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**, `hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out
whose stage handoff is a `send`/`recv` of the hidden state on the device **vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails. the hidden state on the device backend
The full matrix is in [[Dual-R9700-Cooperation]]. (`vllm/distributed/parallel_state.py`), the very primitive that failed. The
full matrix is in [[Dual-R9700-Cooperation]].
**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both
cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read
`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is
*unblocked, not proven* — and this service still runs the shape below, which
needs no collective either way.
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain distributed across the cards in order and the boundary activation is a plain
@@ -63,7 +70,7 @@ shape — prefill nearly doubles and one model serves from both cards.
|---|---| |---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) | | GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it | | VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** | | Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` | | iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap | | CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
serverAppConfigs-docs/LLM/index.md
+5 -3
View File
@@ -60,15 +60,17 @@ frank-small, port 7771]
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving 2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
behind :7770 (tensor parallelism is impossible on this host — the second card behind :7770 (tensor parallelism was impossible on the old board's second
cannot dispatch an RCCL collective), balanced 32:33 over two 163,840-token card; the 2026-09-20 swap cleared that gate but the runtime still ships
RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token
slots slots
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
ggz14/radiance-vllm-mxfp4 patches) serving ggz14/radiance-vllm-mxfp4 patches) serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
behind a round-robin proxy; kept as the record of the deployment it behind a round-robin proxy; kept as the record of the deployment it
succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous
board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]]
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era, - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
2026-09-15): charlie12345/ROCmFPX serving 2026-09-15): charlie12345/ROCmFPX serving
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
serverAppConfigs-docs/index.md
+9 -8
View File
@@ -120,20 +120,21 @@ The LLM inference fleet on the two hosts. Full index:
serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2 serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
32:33 over two 163,840-token slots — no tensor parallelism, because the 32:33 over two 163,840-token slots — no tensor parallelism: the runtime ships
second card cannot dispatch an RCCL collective RCCL-less, and the collective has not been re-probed since the 2026-09-20
board swap cleared the atomics gate
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
round-robin proxy; kept as the record of the deployment round-robin proxy; kept as the record of the deployment
[[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence
that rules out TP=2 on this host (since 2026-09-15/16) that ruled out TP=2 on the previous board (gate cleared 2026-09-20)
- [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel - [[Dual-R9700-Cooperation]] — why the two R9700s *could not* run tensor-parallel
together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
(collectives *and* P2P `send`/`recv` both fail on card #2, which rules out (collectives *and* P2P `send`/`recv` both failed on card #2, which ruled out
pipeline parallelism too), what is closed and what is still open, and the two pipeline parallelism too), what was closed, and the re-measured 2026-09-20
hardware paths — a ~$30 M.2→PCIe x4 riser into the free, CPU-attached M.2_2 topology after the ProArt B850-Creator x8/x8 swap — both chains `Routing+`,
slot, or a board whose CPU x16 bifurcates to x8/x8 both endpoints `ReqEn+`, so TP=2 is unblocked but still unprobed
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era): - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
charlie12345/ROCmFPX serving kingjones777/ charlie12345/ROCmFPX serving kingjones777/
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
serverAppConfigs-docs/log.md
+34
View File
@@ -11,6 +11,40 @@ timestamp: 2026-09-14
# Bundle Change Log # Bundle Change Log
## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone
- **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO**
(was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board
bifurcates the CPU GPP ports **x8/x8 at Gen5**, so both cards sit on CPU root
ports: card 0 `00:01.1 → 01:00.0 → 02:00.0 → 03:00.0`, card 1
`00:01.3 → 05:00.0 → 06:00.0 → 07:00.0`. Every bridge reads
`AtomicOpsCap: Routing+` and **both endpoints read `AtomicOpsCtl: ReqEn+`**
(`CAP_EXP+28.w = 0x1440`) — the exact precondition
[[Dual-R9700-Cooperation]]'s test plan named before any model would be
touched. Bandwidth per card ~31.5 GB/s (kernel: `252.056 Gb/s … limited by
32.0 GT/s PCIe x8`): card 0 unchanged from Gen4 x16, card 1 up ~16x from
Gen4 x1.
- **Verdict, stated honestly.** vLLM TP=2 / PP=2 and llama.cpp `--split-mode
row` are **unblocked, not proven** — no collective has been dispatched since
the swap, `GGML_HIP_RCCL` is still `OFF` in `scripts/build-llama-rdna4.sh`,
and the 272 t/s "tensor" number on record measured a butterfly fallback, not
an allreduce. `librccl.so.1` + `hipcc` already exist in the bundled TheRock
toolchain, so an RCCL-backed rebuild needs no new host packages. The probe
still owes a stop-and-mask window.
- **Side effects of the swap, recorded.** GPU BDFs moved (`0b:00.0` → `07:00.0`),
iGPU `10:00.0` → `11:00.0`, WiFi interface `wlp8s0` → `wlp14s0`, kernel now
7.0.0-31-generic, SMT back **on** (16 logical CPUs; llama.cpp still `--threads
8`), and the board's second 5GbE port is cabled as `eno2` = 192.168.0.247 with
the lower-metric default route while `eno1` keeps the fleet's static
192.168.0.235. Two addresses, two default routes — needs a decision.
- **Docs cut over** (no service change; frank-medium stayed up and healthy
throughout): [[Dual-R9700-Cooperation]] short answer rewritten plus a
re-measured 2026-09-20 topology table; [[Dankserver System Specs]] gained a
Motherboard section and corrected GPU/iGPU/network/kernel rows;
[[frank-medium-llamacpp-dflash2]] hardware row and rationale now split past
from present; `AGENTS.md` topology table, the frank-medium section, and the
PCIe lesson bullets no longer assert a wall that no longer exists. The
dated work-log sections that describe the old board stay as history.
## 2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert ## 2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert
- **frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca.** The private - **frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca.** The private
CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja