docs: what the x4 pair can and cannot do — measured, not argued

The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.

What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.

New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.
This commit is contained in:
Frank
2026-09-16 17:40:37 -07:00
parent e2dc2b67ce
commit 65fbef63dc
4 changed files with 224 additions and 8 deletions
+38
View File
@@ -621,3 +621,41 @@ timestamp: 2026-09-14
- Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service
frank-small` — per-service health gates, then the e2e gate PASSED (31 passed,
1 flaky-retried). Suite: 323 seams ok, 0 failures.
## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do
- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends
(`build-dual`; glslc and libvulkan were already installed), to test the design
the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the
iGPU, PLE in LPDDR, one hop per boundary.
- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and
**`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU
being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures
the wrong device.
- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored):
experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else
**3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only
they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read
per token at 200k context.
- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan,
PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 →
231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU
experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design
**aborted in the scheduler** — `pre-allocated tensor
(blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the
operation (NONE)` — while both per-layer cross-backend forms failed to load.
Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only
fallback side that works.
- The comparison that matters: at fleet-realistic depth the iGPU-only run
matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116
of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page
cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not
deployable beside the fleet's ~28 co-tenants. Throughput parity, memory
bankruptcy.
- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it
`GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root
unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg`
instances run with `n_prompt = 0`, so `-d` is required to put KV into the
measurement at all. Both services were stopped and restored (both healthy;
halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget
with 15.2 GiB left).
- New page: [[x4-pair-experiment]].