docs: dankserver board swap clears the PCIe-atomics wall
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation), every bridge reads AtomicOpsCap Routing+, and both endpoints read AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked, not proven: no collective has been dispatched since the swap, GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record measured a butterfly fallback. - Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20 topology table, rebuild-hypothesis door marked moot, probe plan is now standing work. - Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 -> 07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new eno2 (192.168.0.247) second 5GbE port holding the low-metric default route while eno1 keeps the fleet's static .235. - frank-medium-llamacpp-dflash2: rationale and hardware row split past from present; dated work-log sections stay as history. - AGENTS.md + both indexes: no longer assert a wall that no longer exists. Docs only; frank-medium stayed active and healthy throughout (verified /health). ./tests/run.sh 331/331.
This commit is contained in:
1 parent
a43ab90105
commit
d16e8e554e
7 files changed
+246
-110
No files matched your search
@@ -32,15 +32,22 @@ replicas, the proxy and their unit files are retired.
|
||||
|
||||
## Why one process and layer split
|
||||
|
||||
The endpoint's whole history is one constraint: **the two cards cannot take
|
||||
The endpoint's whole history is one constraint: **the two cards could not take
|
||||
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
|
||||
and ROCr refuses to supply it to a device without PCIe atomics; the second
|
||||
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
|
||||
every collective that touches it dies with `hipErrorIllegalState`. That rules
|
||||
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
|
||||
whose stage handoff is a `send`/`recv` of the hidden state on the device
|
||||
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
|
||||
The full matrix is in [[Dual-R9700-Cooperation]].
|
||||
and ROCr refuses to supply it to a device without PCIe atomics; on the previous
|
||||
board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`,
|
||||
read-only), so every collective that touched it died with
|
||||
`hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out
|
||||
**vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of
|
||||
the hidden state on the device backend
|
||||
(`vllm/distributed/parallel_state.py`), the very primitive that failed. The
|
||||
full matrix is in [[Dual-R9700-Cooperation]].
|
||||
|
||||
**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both
|
||||
cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read
|
||||
`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is
|
||||
*unblocked, not proven* — and this service still runs the shape below, which
|
||||
needs no collective either way.
|
||||
|
||||
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
|
||||
distributed across the cards in order and the boundary activation is a plain
|
||||
@@ -63,7 +70,7 @@ shape — prefill nearly doubles and one model serves from both cards.
|
||||
|---|---|
|
||||
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
|
||||
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
|
||||
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
|
||||
| Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] |
|
||||
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
|
||||
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
|
||||
|
||||
|
||||
Reference in new issue
Block a user