dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: dankserver board swap clears the PCIe-atomics wall

Browse Source 
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.

- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
  topology table, rebuild-hypothesis door marked moot, probe plan is now
  standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
  07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
  eno2 (192.168.0.247) second 5GbE port holding the low-metric default
  route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
  from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.

Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
This commit is contained in:
Frank
2026-09-20 14:39:54 -07:00
parent a43ab90105
commit d16e8e554e
7 changed files with 246 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+43 -26
View File
@@ -1,14 +1,14 @@
---
type: configuration
title: Dankserver System Specs
description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium.
description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
tags:
- dankserver
- hardware
- r9700
- gfx1201
- rocm
timestamp: 2026-09-15
timestamp: 2026-09-20
---
# Dankserver System Specs
@@ -22,6 +22,18 @@ two AMD Radeon AI PRO R9700 cards.
## Hardware Specifications
### Motherboard
| Property | Value |
|----------|-------|
| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3`; ~31.5 GB/s per card |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board |
| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less), trained **x2** at Gen5 (63 Gb/s ≈ 7.9 GB/s) |
| Onboard | 2x Realtek RTL8126 5GbE, RTL8922AE WiFi 7, PROM21 SATA controller |
### CPU
| Property | Value |
@@ -29,7 +41,7 @@ two AMD Radeon AI PRO R9700 cards.
| Architecture | x86_64 |
| Vendor | AuthenticAMD |
| Model | AMD Ryzen 7 7700X |
| Cores | 8 cores / 8 threads (1 socket, SMT off) |
| Cores | 8 cores / 16 threads (1 socket) — SMT is **on**; the BIOS defaults reset with the board swap. `llama.cpp` still runs `--threads 8 --threads-batch 8`, so this is headroom, not a change in the serving shape |
| NUMA Nodes | 1 (node 0) |
### GPUs
@@ -38,10 +50,11 @@ two AMD Radeon AI PRO R9700 cards.
|----------|-------|
| Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) |
| Architecture | gfx1201 (RDNA4) |
| VRAM | 32,624 MiB each |
| Link | PCIe, no XGMI; different IOMMU groups |
| PCI | 03:00.0 and 0b:00.0 |
| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP |
| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
| Link | PCIe 5.0 **x8** each (CPU GPP `00:01.1` / `00:01.3`), no XGMI; separate IOMMU groups (15 and 20), ACS `ReqRedir+` on both root ports, so P2P hairpins at the root complex |
| Atomic ops | `Routing+` on all four bridges per chain; `ReqEn+` on both endpoints (`CAP_EXP+28.w = 0x1440`) |
| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2, hidden from HIP |
The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA
apt lists remain on disk but no NVIDIA PCI device exists.
@@ -61,27 +74,29 @@ apt lists remain on disk but no NVIDIA PCI device exists.
| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
Models live under `/home/dank/models`; the ROCmFPX build tree under
`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14
toolchain under `~/.local/share/rocm-7.14.0a20260624`.
Models live under `/home/dank/models`; the serving build tree under
`/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is
still on disk), both sharing the self-contained ROCm 7.14 nightly toolchain
under `~/.local/share/rocm-7.14.0a20260624` — which also carries `librccl.so.1`
and `hipcc`, so an RCCL-backed rebuild needs no new host packages.
### Network
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. |
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. |
| wlp8s0 | WiFi | - | - | Unused (carrier down). |
| eno1 | onboard RTL8126 5GbE | 2500 Mb/s (negotiated) | 192.168.0.235, **static** | LLM endpoint (port 7770). The address `models.yml`, Prometheus and Pi-hole all point at — preserved across the swap. |
| eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, and it took the lower-metric default route (101, against eno1's 102). Two live addresses and two default routes for one host — either declare it or bring the link down. |
| wlp14s0 | WiFi 7 (RTL8922AE) | - | - | Renamed with the board (was `wlp8s0`); unused. |
### OS and Software
| Property | Value |
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-30-generic |
| Kernel | 7.0.0-31-generic (`iommu=pt`) |
| Python | 3.14.4 |
| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) |
| Serving stack | mainline llama.cpp `fb27a525` at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714`, RPATH-bundled with the ROCm 7.14 nightly runtime |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -89,19 +104,20 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
### Active Services
The active inference service is `frank-medium.service`: charlie12345/ROCmFPX
llama-server on port 7770, layer-split across both R9700s with the MTP draft
head on ROCm1. See [[frank-medium-rocmfpx]]. The former
`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its
unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`.
The earlier PP worker (`frank-pp-worker.service`) was retired with the
2026-08-17 redo. See [[deployment]].
The active inference service is `frank-medium.service`: mainline llama-server on
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
See [[deployment]].
## Hardware Relevance
- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft
and a 131,072-token f16 KV cache fit comfortably; layer split gives each
card about half the layers and its own KV share.
- 2x 32 GB VRAM: the 27.0 GiB Q8_0 checkpoint plus the 1.06 GiB DFlash2 drafter
and the f16 KV pool for 327,680 tokens fit with the split at 32:33; live
headroom is ~6.3 GiB on ROCm0 and ~5.2 GiB on ROCm1, which is why a batch or
context change needs a re-fit and not a guess.
- 60 GB system RAM is mostly headroom; nothing in the serving path is
CPU-resident.
- The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off
@@ -109,6 +125,7 @@ The earlier PP worker (`frank-pp-worker.service`) was retired with the
## See Also
- [[frank-medium-rocmfpx]] — the active serving runbook
- [[frank-medium-llamacpp-dflash2]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory