From 65fbef63dcf9c4b6fc58880b8be8625b379bb9fb Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 17:40:37 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20what=20the=20x4=20pair=20can=20and=20ca?= =?UTF-8?q?nnot=20do=20=E2=80=94=20measured,=20not=20argued?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)" -- and both per-layer cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU is the only fallback side that works, which is exactly why -ot exps=CPU runs. What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s -- but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7 spare, so it is not deployable beside the co-tenants. Throughput parity, memory bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than preference. New page LLM/x4-pair-experiment.md carries the tensor inventory, the results table, the two scheduler limits, reproduction commands, and the traps found on the way. Both index pages also stop calling halogen retired -- the ik swap's stale claims were still sitting in them. --- serverAppConfigs-docs/LLM/index.md | 6 + .../LLM/x4-pair-experiment.md | 166 ++++++++++++++++++ serverAppConfigs-docs/index.md | 22 ++- serverAppConfigs-docs/log.md | 38 ++++ 4 files changed, 224 insertions(+), 8 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/x4-pair-experiment.md diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 52e14a9..8eead50 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -92,6 +92,12 @@ frank-small, port 7771] ISTA-DASLab/Qwen3.8-27B-GSQ-RCO IQ3_S on the RTX 3090, with the model's own Multi-Token Prediction head for speculative decoding and the BF16 projector for the fleet's image path +- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and + cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's + Gen4 x4). The tensor inventory that decides any split, the two scheduler + limits that kill a cross-backend one, and the finding that the iGPU alone + matches halogen's deployed decode rate while leaving the 3090 idle — at a + memory cost the co-tenants cannot pay - [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md new file mode 100644 index 0000000..7570ae1 --- /dev/null +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -0,0 +1,166 @@ +--- +type: reference +title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU +description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle. +tags: + - llm + - experiment + - dankclaw + - llama.cpp + - vulkan + - strix-halo + - rtx3090 + - flash-next + - pipeline-parallelism +timestamp: 2026-09-16 +--- + +# The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU + +The question this answers: **is there a 3090 + Strix Halo combination worth +building**, given that the only link between them is the 3090's PCIe Gen4 x4 +(measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable; +idle-downshifts to Gen1 x4). + +Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend +build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM +services stopped per the benchmarking policy. + +## The tensor inventory decides everything + +From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, +1224 tensors, 176.94 B params, **A3B active**): + +| group | size | +|---|---| +| `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** | +| `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** | +| everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** | + +- 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`, + every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. +- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per + position, a 200k-token conversation reads **5.5 GiB of KV per token**. + +That is why the interesting split is *not* "put the experts somewhere else" — +experts are re-read once per token but they are only ~1.7 GB of traffic — it is +"get KV onto the 936 GB/s pool", because KV is the O(context) term. + +## Results + +`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a +depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with +`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to +put KV in the picture. + +| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg32 @64k depth | +|---|---|---|---|---|---| +| **A** | iGPU trunk (Vulkan1) + experts on CPU | 128.7 | 122.1 | 12.10 | — | +| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k = 231.8) | +| **B** | trunk + KV on CUDA0, experts on Vulkan1 | — | — | **abort** | — | +| **B2** | attention layers → CUDA0, rest → Vulkan1 (per-layer `-ot`) | — | — | load failure | — | +| **B3** | 25 % of layers → CUDA0, rest → Vulkan1 (`-sm layer -ts 0.25,0.75`) | — | — | load failure | — | +| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — | + +Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real +contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the +3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill +ceiling 192 tok/s. + +## Finding 1 — the cross-backend split is not expressible + +Per-tensor placement between two **device** backends aborts in the scheduler: + +``` +ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) +in a buffer (Vulkan1) that cannot run the operation (NONE) +``` + +A weight tensor living in one device's buffer cannot be consumed by an op the +scheduler placed on another device. The CPU is the exception — `-ot exps=CPU` +works — because the CPU backend is the universal fallback and host buffers can +be copied. Per-**layer** cross-backend splits (`-sm layer -ts`, and layer-scoped +`-ot`) are no better: both failed at model load with no further diagnostic. + +So on this build, **CUDA + Vulkan cannot serve one model**: the 3090 can be the +trunk *or* the expert host, but not the complement of an iGPU-resident peer. +That is the same wall the R9700 pair hit from the other side (collectives), and +it is why every working multi-device llama.cpp deployment in this fleet is +either single-vendor (frank-medium: two AMD cards) or device+CPU. + +## Finding 2 — the iGPU alone is competitive, and that was not expected + +Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in +mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep +context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25 +tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 +sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %** +busy, so the work is genuinely there and not silently CPU-side. + +The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s +from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the +LPDDR's shared 256 GB/s. + +**Why this is still not deployable here:** the run committed **116 of the host's +121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving +5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, +because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory +bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape +exists. + +## Finding 3 — the fast pool is too small to help + +B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — *worse* +than A\*, which does not use the 3090 at all. And the expressible version of the +KV-on-VRAM design cannot reach its own premise: moving an attention layer to the +3090 costs **1.62 GiB** (its MoE experts travel with it, since only whole layers +move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16, +4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context — +precisely where the KV term is small enough not to matter. + +## Conclusion + +On this host the pair is not additive, and the measured best arrangement remains +**two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen, +[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB). A +single process spanning both is blocked by the scheduler, and every CPU-side +fallback is slower than what the iGPU does alone. + +The one durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 +is a viable engine for this class of model** (25.8 tok/s at 64k depth), which the +2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was +the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better +engine than halogen?" question should start here, with the memory budget as the +first constraint rather than the last. + +## Reproducing + +```bash +# dual-backend build (glslc + libvulkan are already installed on dankclaw) +cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ + -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON +cmake --build build-dual --target llama-bench llama-cli llama-server -j 16 + +# device map on this host +CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card) +Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT + +# config A* (the winner), depth-matched +GGML_CUDA_NO_PINNED=1 llama-bench -m -dev Vulkan1 -ngl 999 \ + -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1 +``` + +Operational notes for the next run: + +- Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used + `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a + pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this + host's co-tenants. +- `systemctl mask` is **refused** for a relinked root unit: frank-small's + `/etc/systemd/system/frank-small.service` is a symlink into the repo, so + `mask` exits with "File ... already exists". Stop it and rely on the absence + of a restart trigger, or move the symlink aside. +- Restarting frank-large needs the quadlet source back in + `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates + the unit from that file only. After the restore the ledger read `400384 + positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 15.2 GiB left. diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 4f14556..23befa3 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -144,14 +144,15 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since - 2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next- - Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd - unit, port 7770) — measured performance envelope and traps -- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and - rollback target: peonist-ai/halogen-flash-server 0.5.8 serving - peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet - container on Dankclaw, port 7770 — flags, memory model, client contract +- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since + 2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server + 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality + overlay) on the gfx1151 iGPU — two 200k conversations over one + 400,384-position pool, deployed flags, memory model, client contract +- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook + and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving + mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 + + Strix Halo hybrid — its measured envelope is why the fleet went back - [[dankclaw-iommu-translation-off]] — why Dankclaw boots with `amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost (the NPU endpoint), and how to revert @@ -193,6 +194,11 @@ The LLM inference fleet on the two hosts. Full index: - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU - [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes +- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot + do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that + decides a split, the scheduler limits that kill a cross-backend one, and the + iGPU-only result that matches halogen's decode rate while leaving the 3090 + idle — for 116 of the host's 121 GiB ### Memory and Hardware diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 5701271..5f92580 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -621,3 +621,41 @@ timestamp: 2026-09-14 - Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service frank-small` — per-service health gates, then the e2e gate PASSED (31 passed, 1 flaky-retried). Suite: 323 seams ok, 0 failures. + +## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do +- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends + (`build-dual`; glslc and libvulkan were already installed), to test the design + the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the + iGPU, PLE in LPDDR, one hop per boundary. +- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and + **`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU + being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures + the wrong device. +- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored): + experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else + **3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only + they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read + per token at 200k context. +- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan, + PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 → + 231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU + experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design + **aborted in the scheduler** — `pre-allocated tensor + (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the + operation (NONE)` — while both per-layer cross-backend forms failed to load. + Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only + fallback side that works. +- The comparison that matters: at fleet-realistic depth the iGPU-only run + matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116 + of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page + cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not + deployable beside the fleet's ~28 co-tenants. Throughput parity, memory + bankruptcy. +- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it + `GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root + unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg` + instances run with `n_prompt = 0`, so `-d` is required to put KV into the + measurement at all. Both services were stopped and restored (both healthy; + halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget + with 15.2 GiB left). +- New page: [[x4-pair-experiment]].