--- type: reference title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless. tags: - llm - experiment - dankclaw - llama.cpp - vulkan - cuda - strix-halo - rtx3090 - flash-next - pipeline-parallelism timestamp: 2026-09-16 --- # The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU The question this answers: **is there a 3090 + Strix Halo combination worth building**, given that the only link between them is the 3090's PCIe Gen4 x4 (measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable; idle-downshifts to Gen1 x4). Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM services stopped per the benchmarking policy. ## The tensor inventory decides everything From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, 1224 tensors, 176.94 B params, **A3B active**): | group | size | |---|---| | `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** | | `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** | | everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** | - 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`, every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads **5.5 GiB of KV per token**. - A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all. The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 = 110.97 GiB) concatenated in numeric order. The same repo carries the smaller rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02, i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K 74.92 GiB — which is where a "does a smaller quant buy anything?" test would start (it buys memory, ~5 % decode, and costs quality; see the write-up below). That is why the interesting split is *not* "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term. ## Results `llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to put KV in the picture. | config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth | |---|---|---|---|---|---| | **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — | | **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) | | **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) | | **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) | | **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — | | **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | | **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — | | **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — | Real placement in the split runs, sampled from `nvidia-smi` during the run: **12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the split is genuine, not a mislabelled single-device run (see Finding 5 for why that check is mandatory here). Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s. ## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot` The design (trunk + KV on the 3090, experts on the iGPU) needs `--override-tensor` to place two *different* tensor groups on two devices. That aborts in the scheduler: ``` ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) ``` The mechanism, read out of the source rather than inferred: ```c // llama-context.cpp:429 — the scheduler only gets multi-device copy slots if: bool pipeline_parallel = model.n_devices() > 1 && ... && !model.has_tensor_overrides(); // ← line 433; a required term // llama-context.cpp:605 — that flag is passed straight to the scheduler: sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...)); // ggml-backend.cpp:25 — where it decides how many copies each tensor may have: sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1; ``` With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a second home, so a weight whose buffer belongs to a non-primary device has no backend and the graph build aborts. `has_tensor_overrides` is consulted in **exactly one place in the whole tree** (llama-context.cpp:433), so no other flag can rescue it. - **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU backend is last in the scheduler's list and host buffers are always readable (configs A and B′). A *device* target is what aborts. - **Pure Vulkan does not change this.** Config G1 put both Vulkan devices genuinely in the device list (verified by the correct `-dev` syntax, see Finding 5) and aborted with the byte-identical message. The blocker is the override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch. ## Finding 2 — two-device layer splits DO work, in both flavours With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily runs a graph across two device backends. Two arrangements were measured at 64k depth: - **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last 7 layers (which include attention layers 43 and 47, the only KV-bearing layers in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**. - **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090 runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s** and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 % prefill**. So the pair *is* expressible — just not in the shape the design wanted. ## Finding 3 — the iGPU alone is competitive, and that was not expected Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25 tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %** busy, so the work is genuinely there and not silently CPU-side. The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the LPDDR's shared 256 GB/s. **Why this is still not deployable here:** the run committed **116 of the host's 121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape exists. The layer split does relieve some of that pressure (the 3090 absorbs ~12.8–15 GiB of layer+KV) but it does not change the class of the result. ## Finding 4 — the split cannot reach the design's premise The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it: - Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB. - Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable is exceeded. Feasible only to ~64k context — precisely where the KV term is small enough not to matter (the +7 % measured above is that residual). - The 32.8 GiB PLE table must live on the *main* device (the first in the `-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps a fixed 32.8 GiB no matter how the layers are split. `-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and Finding 1 applies: it turns multi-device scheduling off. ## Finding 5 — the harness-parsing trap that invalidated the first round The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion was an artifact. `llama-bench --help` reads: ``` -dev, --device multi values separated by ',' -ts, --tensor-split multi values separated by ',' ``` `parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the outer loop splits the argument on `,` into separate *sweep values*. So `-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone, then Vulkan1 alone) — never a two-device run. Consequences that looked like findings: - "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a device that is not in the list at all (Finding 1's abort, for the trivial reason). - "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0 takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd. The tell that the runs were single-device: the bench table's own `dev` column printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within 0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device sweep looks like. **Rules that follow.** In llama-bench, `/` separates devices *within one configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both follow this. And never trust a placement-dependent number without an independent check of where the memory went — `nvidia-smi` sampled during the run is what distinguishes a real split from a mislabelled sweep. ## Other landmines walked into (worth knowing before the next run) - **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** — 3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this build.) - **A dead rootless podman pause process takes the whole host's container management down.** The first `invalid internal status` in the user journal is **18:58:30**, from containers whose healthchecks shell out to the podman CLI — i.e. the pause process died *while the last bench was still running*, not after the runs finished. Every `podman` command for the user then failed with `invalid internal status … could not find any running process`. The trigger is **not** established: `journalctl -k` holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs squeezed it out") is a guess and is recorded as one. `podman system migrate` — the remedy podman itself names — **panics** in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears `/run/user/` and the whole host comes back — 28 quadlets, `WantedBy` present, linger on). - **A `Network=host` service is not reachable from a bridge container at the host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind is refused. Use `host.containers.internal` (169.254.1.2). This was found the same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and guarded by a strengthened seam (`tests/hermes_webui.bats`). - **The memory budget includes the co-tenants' patience.** The iGPU runs held 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log proves damage came of it, but running a model that big on a host that also serves ~28 containers is asking the question — do it in a window, not alongside. ## Conclusion The fleet's answer is unchanged — **two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen, [[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two claims from the first round of this experiment are now corrected: 1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23 tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint (`frank-zealot` holds 22.4 of the card's 24.6 GiB). An operator who wants the 7 % must give up the fleet's only image-capable endpoint. 2. **Per-tensor placement across two devices is impossible while `-ot` is used**, for a scheduler-mechanical reason, regardless of driver or vendor mix. The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better engine than halogen?" question should start here, with the memory budget as the first constraint rather than the last. [INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the worse mainline's position, and depth is filled by prefill that mainline would have to spend ~200k/232 ≈ 14 minutes on. ## Reproducing ```bash # dual-backend build (glslc + libvulkan are already installed on dankclaw) cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON cmake --build build-dual --target llama-bench llama-cli llama-server -j 16 # device map on this host CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card) Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT export GGML_CUDA_NO_PINNED=1 # A* — iGPU only (the single-device baseline), depth-matched llama-bench -m -dev Vulkan1 -ngl 999 \ -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1 # G2 — pure-Vulkan two-device layer split (SLASH = within one configuration) llama-bench -m -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \ -ngl 999 -n 32 -d 65536 -r 1 # H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan llama-bench -m -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \ -ngl 999 -n 32 -d 65536 -r 1 # proof of placement — sample the 3090 while the run is loaded nvidia-smi --query-gpu=memory.used --format=csv,noheader # expect 12–15 GiB ``` Operational notes for the next run: - Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants. - `systemctl mask` is **refused** for a relinked root unit: frank-zealot's `/etc/systemd/system/frank-zealot.service` is a symlink into the repo, so `mask` exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside. - Restarting frank-dragoon needs the quadlet source back in `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates the unit from that file only. After the restore (and the host reboot, see the landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB before the reboot: the same model, a more fragmented host afterwards — the engine reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints answered `/health` and served `/v1/models` under their names. ## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM` fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against 110.97 for the Q4_K_M payload the earlier numbers came from. | test | Q4_K_M | Q4_K_S | Δ | |---|---|---|---| | A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % | | A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % | | CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % | | Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % | Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and `nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split), peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB smaller file bought roughly that much headroom. **Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB. Nothing structural moves — `-ot` still gates per-tensor placement and a layer split still cannot consolidate KV on the 3090. If the aim were to *deploy* mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256 `b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. ## The link, measured (2026-09-16) `pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout. | size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D | |---|---|---|---|---|---| | 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 | | 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 | | 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 | | 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 | | 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 | | 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 | Three readings: - **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is x16-capable and the Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under load — which these numbers prove on their own, since Gen1 x4 tops out near 1 GB/s. - **Async buys nothing and pinning buys ~1 %**: one stream of copies already saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and pageable transfers run within noise of pinned ones. On this host the copies are link-limited rather than staging-limited, so the usual "pageable costs 2×" rule does not apply. - **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead. ### What that means for an expert-streaming engine Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB expert mass) — 580 × 3 MiB slabs: | assumption | per-token link time | ceiling | |---|---|---| | every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** | | 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s | | 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s | | *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it | Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with the experts *computed* on the CPU (link carries activations only), 25 for halogen, 26.5 for mainline on the iGPU — an engine that *streams* experts across this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9 tok/s). For the specific engine that raised the question: FreeToken v0.1.2 keeps expert weights in host RAM with an LRU GPU cache — exactly this streaming shape — and it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M **GGUF** is not loadable at all before bandwidth enters the discussion.