Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.
Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:
pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
iGPU alone 25.77 tok/s @64k, pp512 231.8
+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.
Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
title:The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
description:What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle.
description:What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.
tags:
- llm
- experiment
- dankclaw
- llama.cpp
- vulkan
- cuda
- strix-halo
- rtx3090
- flash-next
@@ -41,6 +42,9 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
position, a 200k-token conversation reads **5.5 GiB of KV per token**.
- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
every layer (75 GiB / 48). Attention layers are spread across the whole depth
(every 4th), so no contiguous layer range can hold them all.
That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
@@ -48,47 +52,88 @@ experts are re-read once per token but they are only ~1.7 GB of traffic — it i
## Results
`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a
depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with
`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to
put KV in the picture.
`llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the
depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run
with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required
It does not: the per-tensor design still aborts, byte-identically.
- **The real gate is `-ot`, not the driver mix.** `has_tensor_overrides()` is a
required term of `pipeline_parallel` in `llama-context.cpp:433` — the ONLY place
in the tree that reads it — and that flag is what sets `n_copies` in
`ggml_backend_sched_new` (`ggml-backend.cpp:25`). With `n_copies == 1` a
pre-allocated weight on a non-primary device has no backend, so the graph
aborts at `ggml-backend.cpp:941` with the leaf's own op name (`(NONE)`).
`-ot exps=CPU` works only because the CPU backend is last in the list.
- **Then the correction.** `llama-bench --help` says `-dev <dev0/dev1/...>`: `/`
splits a device LIST, `,` splits SWEEP values. So `-dev CUDA0,Vulkan1` was two
*single-device* runs — the earlier "CUDA + Vulkan cannot serve one model", the
"load failures", and the OOMs were artifacts (the OOM was 111 GiB on the 24 GiB
card). The bench table's `dev` column had been printing the truth all along.
- **With the correct syntax two-device splits work, in both flavours** (64k depth,
`-d 65536`; placement proven by `nvidia-smi` sampled during the run):
- pure Vulkan `-dev Vulkan1/Vulkan0 -ts 0.85/0.15` — 12.8 GiB on the 3090,
**27.55 tok/s** (pp512@64k 224.3)
- cross-vendor `-dev CUDA0/Vulkan1 -ts 0.15/0.85` — 15.0 GiB on the 3090,
27.23 tok/s with **pp512@64k 268.5**
- against the iGPU alone: 25.77 tok/s / 231.8 — so **+7 % decode, +16 % prefill**
- **And the split still cannot reach the design's premise**: whole layers move
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
so KV cannot be consolidated on the 3090.
- **Incident, same evening:** the rootless podman pause process died (memory
pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman`
command then failed (`invalid internal status … could not find any running
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
`podman system migrate` — podman's own named remedy — **panics** in 5.7.0
(`teardownNetworkBackend`). Repair was a **reboot**: all 28 containers returned
on their own (linger + `WantedBy=default.target` present), e2e gate green after.
- **A second, pre-existing defect surfaced by that reboot:** `hermes.franksdank.cloud`
returned 502. hermes-webui is `Network=host`, so 8787 is a bare host-netns bind
and netavark DNATs only *published* ports — from inside the Caddy container
`192.168.0.143:8787` and `10.89.1.1:8787` are refused while
`host.containers.internal:8787` (169.254.1.2) answers 200. Fixed in the
Caddyfile, the stale claim in its comment replaced with the measurement, and the
seam that guarded it **strengthened** (it accepted any numeric upstream, so the
LAN IP had been passing it; reverting now fails it — teeth verified).
- Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB`
against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page:
[[x4-pair-experiment]] (rewritten).
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Frank