From c97e359bc245a0b0b7a95274c80964a163826dd0 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 15:39:47 -0700 Subject: [PATCH] docs: the LLM index and the cooperation page catch up with layer split The LLM index still opened with vllm-radiance as frank-medium's runtime and drew the replica pair in its topology diagram; the cooperation page's thesis (collectives are impossible, so the answer is hardware) now has to name the path the fleet actually took: llama.cpp's layer split, which launches no RCCL kernel on either card. That reframes the M.2 riser as a means to TENSOR parallelism rather than a means to using both cards at all. --- .../LLM/Dual-R9700-Cooperation.md | 33 ++++++++++++++--- serverAppConfigs-docs/LLM/index.md | 35 +++++++++++-------- 2 files changed, 49 insertions(+), 19 deletions(-) diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md index 5af528d..7f52571 100644 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md @@ -123,11 +123,11 @@ sudo lspci -vv -s | grep AtomicOpsCtl # want: ReqEn+ ``` If that reads `ReqEn+`, the rest is mechanical: `--tensor-parallel-size 2` with -`HIP_VISIBLE_DEVICES=0,1`, then the same benchmark set used for the replicas -(`scripts/bench-vllm-corpus.py` / `bench-turn-latency.py`) so the two -architectures are compared on identical workloads. If it still reads `ReqEn-`, -the card is still on a non-atomic path — check `lspci -tv` for the bridge chain -rather than assuming the move worked. +`HIP_VISIBLE_DEVICES=0,1`, then the same benchmark set +(`scripts/bench-turn-latency.py`, and the endpoint's own `timings` block) so the +two architectures are compared on identical workloads. If it still reads +`ReqEn-`, the card is still on a non-atomic path — check `lspci -tv` for the +bridge chain rather than assuming the move worked. **Rule for any future multi-GPU collective experiment: stop and mask the production service first** (`sudo systemctl stop frank-medium && sudo systemctl @@ -138,6 +138,29 @@ power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the GPU driver is a plausible cause and this host has precedent for going unreachable until a power cycle. + +## The third path, and the one in production: llama.cpp's layer split + +Everything above is about *collectives*, and vLLM cannot proceed without them +in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage +handoff is a p2p `send`/`recv` of the hidden state, the primitive this host +also fails. **llama.cpp's `--split-mode layer` needs no collective at all**: +layers are distributed across the cards and the boundary activation is a +plain per-backend copy, once per forward pass. No RCCL kernel is ever launched +on the second card, so the atomics gate never comes up. + +That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one +process, one model, both cards, 262144 context. Measured against tensor split +on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / +26.3 tensor (butterfly fallback, no RCCL). Tensor would still win decode on a +build with a working RCCL — which is exactly what the M.2 riser or a board swap +in [the short answer](#the-short-answer) would buy. + +So the ranking changed on 2026-09-16: software already gets both cards onto one +model, and the hardware fix is no longer a means to that end — it is a means to +*tensor* parallelism, which buys ~40 % decode and would let this runtime drop +layer-split's serial stages entirely. + ## What would change the answer - An RCCL build whose kernels stop declaring a hostcall buffer — the one diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index af5ce8f..52e14a9 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,11 +1,13 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on the two hosts (2026-09-16): -vllm-radiance on Dankserver (frank-medium, just1moremodel -Qwen3.8-27B-Uncensored-MXFP4-awq on 2x Radeon AI PRO R9700) and TWO endpoints on -Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next w4b on the -gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, ISTA-DASLab -Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port 7771). +**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split +process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B- +Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO +endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next +w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, +ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port +7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -19,10 +21,10 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[vllm-radiance MXFP4 + B2[llama.cpp, layer split frank-medium, port 7770] - C[Qwen3.8-27B-Uncensored-MXFP4-awq -2 single-GPU replicas, 114688 ctx / 4 seqs, DFlash2] + C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL +ONE process on both cards, 262144 ctx / 1 slot, DFlash2] A --> B2 --> C end @@ -54,13 +56,18 @@ frank-small, port 7771] ## Active Deployment -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the - vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + +- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since + 2026-09-16: mainline llama.cpp serving + ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the + z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE + layer-split process behind :7770 (tensor parallelism is impossible on this + host — the second card cannot dispatch an RCCL collective) +- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was + deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + ggz14/radiance-vllm-mxfp4 patches) serving - just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon - AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, with - the PCIe-atomics evidence that rules out TP=2 on this host (since - 2026-09-15/16) + just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas + behind a round-robin proxy; kept as the record of the deployment it + succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era, 2026-09-15): charlie12345/ROCmFPX serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the