--- type: reference title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless. tags: - llm - experiment - dankclaw - llama.cpp - vulkan - cuda - strix-halo - rtx3090 - flash-next - pipeline-parallelism timestamp: 2026-09-16 --- # The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU The question this answers: **is there a 3090 + Strix Halo combination worth building**, given that the only link between them is the 3090's PCIe Gen4 x4 (measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable; idle-downshifts to Gen1 x4). Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM services stopped per the benchmarking policy. ## The tensor inventory decides everything From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, 1224 tensors, 176.94 B params, **A3B active**): | group | size | |---|---| | `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** | | `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** | | everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** | - 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`, every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads **5.5 GiB of KV per token**. - A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all. That is why the interesting split is *not* "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term. ## Results `llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to put KV in the picture. | config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth | |---|---|---|---|---|---| | **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — | | **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) | | **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) | | **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) | | **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — | | **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | | **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | | **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — | Real placement in the split runs, sampled from `nvidia-smi` during the run: **12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the split is genuine, not a mislabelled single-device run (see Finding 5 for why that check is mandatory here). Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s. ## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot` The design (trunk + KV on the 3090, experts on the iGPU) needs `--override-tensor` to place two *different* tensor groups on two devices. That aborts in the scheduler: ``` ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) ``` The mechanism, read out of the source rather than inferred: ```c // llama-context.cpp:429 — the scheduler only gets multi-device copy slots if: bool pipeline_parallel = model.n_devices() > 1 && ... && !model.has_tensor_overrides(); // ← line 433; a required term // llama-context.cpp:605 — that flag is passed straight to the scheduler: sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...)); // ggml-backend.cpp:25 — where it decides how many copies each tensor may have: sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1; ``` With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a second home, so a weight whose buffer belongs to a non-primary device has no backend and the graph build aborts. `has_tensor_overrides` is consulted in **exactly one place in the whole tree** (llama-context.cpp:433), so no other flag can rescue it. - **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU backend is last in the scheduler's list and host buffers are always readable (configs A and B′). A *device* target is what aborts. - **Pure Vulkan does not change this.** Config G1 put both Vulkan devices genuinely in the device list (verified by the correct `-dev` syntax, see Finding 5) and aborted with the byte-identical message. The blocker is the override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch. ## Finding 2 — two-device layer splits DO work, in both flavours With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily runs a graph across two device backends. Two arrangements were measured at 64k depth: - **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last 7 layers (which include attention layers 43 and 47, the only KV-bearing layers in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**. - **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090 runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s** and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 % prefill**. So the pair *is* expressible — just not in the shape the design wanted. ## Finding 3 — the iGPU alone is competitive, and that was not expected Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25 tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %** busy, so the work is genuinely there and not silently CPU-side. The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the LPDDR's shared 256 GB/s. **Why this is still not deployable here:** the run committed **116 of the host's 121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape exists. The layer split does relieve some of that pressure (the 3090 absorbs ~12.8–15 GiB of layer+KV) but it does not change the class of the result. ## Finding 4 — the split cannot reach the design's premise The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it: - Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB. - Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable is exceeded. Feasible only to ~64k context — precisely where the KV term is small enough not to matter (the +7 % measured above is that residual). - The 32.8 GiB PLE table must live on the *main* device (the first in the `-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps a fixed 32.8 GiB no matter how the layers are split. `-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and Finding 1 applies: it turns multi-device scheduling off. ## Finding 5 — the harness-parsing trap that invalidated the first round The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion was an artifact. `llama-bench --help` reads: ``` -dev, --device multi values separated by ',' -ts, --tensor-split multi values separated by ',' ``` `parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the outer loop splits the argument on `,` into separate *sweep values*. So `-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone, then Vulkan1 alone) — never a two-device run. Consequences that looked like findings: - "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a device that is not in the list at all (Finding 1's abort, for the trivial reason). - "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0 takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd. The tell that the runs were single-device: the bench table's own `dev` column printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within 0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device sweep looks like. **Rules that follow.** In llama-bench, `/` separates devices *within one configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both follow this. And never trust a placement-dependent number without an independent check of where the memory went — `nvidia-smi` sampled during the run is what distinguishes a real split from a mislabelled sweep. ## Other landmines walked into (worth knowing before the next run) - **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** — 3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this build.) - **A dead rootless podman pause process takes the whole host's container management down.** The first `invalid internal status` in the user journal is **18:58:30**, from containers whose healthchecks shell out to the podman CLI — i.e. the pause process died *while the last bench was still running*, not after the runs finished. Every `podman` command for the user then failed with `invalid internal status … could not find any running process`. The trigger is **not** established: `journalctl -k` holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs squeezed it out") is a guess and is recorded as one. `podman system migrate` — the remedy podman itself names — **panics** in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears `/run/user/` and the whole host comes back — 28 quadlets, `WantedBy` present, linger on). - **A `Network=host` service is not reachable from a bridge container at the host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind is refused. Use `host.containers.internal` (169.254.1.2). This was found the same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and guarded by a strengthened seam (`tests/hermes_webui.bats`). - **The memory budget includes the co-tenants' patience.** The iGPU runs held 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log proves damage came of it, but running a model that big on a host that also serves ~28 containers is asking the question — do it in a window, not alongside. ## Conclusion The fleet's answer is unchanged — **two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen, [[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two claims from the first round of this experiment are now corrected: 1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23 tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint (`frank-small` holds 22.4 of the card's 24.6 GiB). An operator who wants the 7 % must give up the fleet's only image-capable endpoint. 2. **Per-tensor placement across two devices is impossible while `-ot` is used**, for a scheduler-mechanical reason, regardless of driver or vendor mix. The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better engine than halogen?" question should start here, with the memory budget as the first constraint rather than the last. [INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the worse mainline's position, and depth is filled by prefill that mainline would have to spend ~200k/232 ≈ 14 minutes on. ## Reproducing ```bash # dual-backend build (glslc + libvulkan are already installed on dankclaw) cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON cmake --build build-dual --target llama-bench llama-cli llama-server -j 16 # device map on this host CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card) Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT export GGML_CUDA_NO_PINNED=1 # A* — iGPU only (the single-device baseline), depth-matched llama-bench -m -dev Vulkan1 -ngl 999 \ -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1 # G2 — pure-Vulkan two-device layer split (SLASH = within one configuration) llama-bench -m -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \ -ngl 999 -n 32 -d 65536 -r 1 # H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan llama-bench -m -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \ -ngl 999 -n 32 -d 65536 -r 1 # proof of placement — sample the 3090 while the run is loaded nvidia-smi --query-gpu=memory.used --format=csv,noheader # expect 12–15 GiB ``` Operational notes for the next run: - Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants. - `systemctl mask` is **refused** for a relinked root unit: frank-small's `/etc/systemd/system/frank-small.service` is a symlink into the repo, so `mask` exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside. - Restarting frank-large needs the quadlet source back in `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates the unit from that file only. After the restore (and the host reboot, see the landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB before the reboot: the same model, a more fragmented host afterwards — the engine reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints answered `/health` and served `/v1/models` under their names.