x4 pair round two: pure Vulkan changes nothing, and the first verdict was wrong

Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.

Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:

  pure Vulkan  -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
  cross-vendor -dev CUDA0/Vulkan1   -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
  iGPU alone                                       25.77 tok/s @64k, pp512 231.8

+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.

Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
This commit is contained in:
Frank committed 2026-09-16 19:32:53 -07:00
1 parent d14e0a94eb
commit 803398051a
5 files changed
+287 -55

No files matched your search

+5 -4
View File
@@ -94,10 +94,11 @@ frank-small, port 7771]
for the fleet's image path
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
Gen4 x4). The tensor inventory that decides any split, the two scheduler
limits that kill a cross-backend one, and the finding that the iGPU alone
matches halogen's deployed decode rate while leaving the 3090 idle — at a
memory cost the co-tenants cannot pay
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
that kills per-tensor placement across devices, the two-device layer splits
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
that invalidated the first round, and the iGPU-only result that matches
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090