docs: what the x4 pair can and cannot do — measured, not argued
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)" -- and both per-layer cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU is the only fallback side that works, which is exactly why -ot exps=CPU runs. What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s -- but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7 spare, so it is not deployable beside the co-tenants. Throughput parity, memory bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than preference. New page LLM/x4-pair-experiment.md carries the tensor inventory, the results table, the two scheduler limits, reproduction commands, and the traps found on the way. Both index pages also stop calling halogen retired -- the ik swap's stale claims were still sitting in them.
This commit is contained in:
@@ -621,3 +621,41 @@ timestamp: 2026-09-14
|
||||
- Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service
|
||||
frank-small` — per-service health gates, then the e2e gate PASSED (31 passed,
|
||||
1 flaky-retried). Suite: 323 seams ok, 0 failures.
|
||||
|
||||
## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do
|
||||
- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends
|
||||
(`build-dual`; glslc and libvulkan were already installed), to test the design
|
||||
the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the
|
||||
iGPU, PLE in LPDDR, one hop per boundary.
|
||||
- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and
|
||||
**`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU
|
||||
being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures
|
||||
the wrong device.
|
||||
- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored):
|
||||
experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else
|
||||
**3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only
|
||||
they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read
|
||||
per token at 200k context.
|
||||
- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan,
|
||||
PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 →
|
||||
231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU
|
||||
experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design
|
||||
**aborted in the scheduler** — `pre-allocated tensor
|
||||
(blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the
|
||||
operation (NONE)` — while both per-layer cross-backend forms failed to load.
|
||||
Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only
|
||||
fallback side that works.
|
||||
- The comparison that matters: at fleet-realistic depth the iGPU-only run
|
||||
matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116
|
||||
of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page
|
||||
cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not
|
||||
deployable beside the fleet's ~28 co-tenants. Throughput parity, memory
|
||||
bankruptcy.
|
||||
- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it
|
||||
`GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root
|
||||
unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg`
|
||||
instances run with `n_prompt = 0`, so `-d` is required to put KV into the
|
||||
measurement at all. Both services were stopped and restored (both healthy;
|
||||
halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget
|
||||
with 15.2 GiB left).
|
||||
- New page: [[x4-pair-experiment]].
|
||||
|
||||
Reference in New Issue
Block a user