serverAppConfigs/serverAppConfigs-docs/LLM/x4-pair-experiment.md

---
type: reference
title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle.
tags:
  - llm
  - experiment
  - dankclaw
  - llama.cpp
  - vulkan
  - strix-halo
  - rtx3090
  - flash-next
  - pipeline-parallelism
timestamp: 2026-09-16
---

# The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU

The question this answers: **is there a 3090 + Strix Halo combination worth
building**, given that the only link between them is the 3090's PCIe Gen4 x4
(measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable;
idle-downshifts to Gen1 x4).

Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend
build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM
services stopped per the benchmarking policy.

## The tensor inventory decides everything

From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
1224 tensors, 176.94 B params, **A3B active**):

| group | size |
|---|---|
| `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** |
| `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** |
| everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** |

- 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`,
  every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
  position, a 200k-token conversation reads **5.5 GiB of KV per token**.

That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
"get KV onto the 936 GB/s pool", because KV is the O(context) term.

## Results

`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a
depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with
`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to
put KV in the picture.

| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg32 @64k depth |
|---|---|---|---|---|---|
| **A** | iGPU trunk (Vulkan1) + experts on CPU | 128.7 | 122.1 | 12.10 | — |
| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k = 231.8) |
| **B** | trunk + KV on CUDA0, experts on Vulkan1 | — | — | **abort** | — |
| **B2** | attention layers → CUDA0, rest → Vulkan1 (per-layer `-ot`) | — | — | load failure | — |
| **B3** | 25 % of layers → CUDA0, rest → Vulkan1 (`-sm layer -ts 0.25,0.75`) | — | — | load failure | — |
| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — |

Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real
contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the
3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill
ceiling 192 tok/s.

## Finding 1 — the cross-backend split is not expressible

Per-tensor placement between two **device** backends aborts in the scheduler:

```
ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)
```

A weight tensor living in one device's buffer cannot be consumed by an op the
scheduler placed on another device. The CPU is the exception — `-ot exps=CPU`
works — because the CPU backend is the universal fallback and host buffers can
be copied. Per-**layer** cross-backend splits (`-sm layer -ts`, and layer-scoped
`-ot`) are no better: both failed at model load with no further diagnostic.

So on this build, **CUDA + Vulkan cannot serve one model**: the 3090 can be the
trunk *or* the expert host, but not the complement of an iGPU-resident peer.
That is the same wall the R9700 pair hit from the other side (collectives), and
it is why every working multi-device llama.cpp deployment in this fleet is
either single-vendor (frank-medium: two AMD cards) or device+CPU.

## Finding 2 — the iGPU alone is competitive, and that was not expected

Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in
mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep
context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25
tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090
sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %**
busy, so the work is genuinely there and not silently CPU-side.

The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s
from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the
LPDDR's shared 256 GB/s.

**Why this is still not deployable here:** the run committed **116 of the host's
121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving
5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare,
because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory
bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape
exists.

## Finding 3 — the fast pool is too small to help

B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — *worse*
than A\*, which does not use the 3090 at all. And the expressible version of the
KV-on-VRAM design cannot reach its own premise: moving an attention layer to the
3090 costs **1.62 GiB** (its MoE experts travel with it, since only whole layers
move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16,
4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context —
precisely where the KV term is small enough not to matter.

## Conclusion

On this host the pair is not additive, and the measured best arrangement remains
**two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen,
[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB). A
single process spanning both is blocked by the scheduler, and every CPU-side
fallback is slower than what the iGPU does alone.

The one durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151
is a viable engine for this class of model** (25.8 tok/s at 64k depth), which the
2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was
the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better
engine than halogen?" question should start here, with the memory budget as the
first constraint rather than the last.

## Reproducing

```bash
# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16

# device map on this host
CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT

# config A* (the winner), depth-matched
GGML_CUDA_NO_PINNED=1 llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1
```

Operational notes for the next run:

- Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used
  `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
  pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
  host's co-tenants.
- `systemctl mask` is **refused** for a relinked root unit: frank-small's
  `/etc/systemd/system/frank-small.service` is a symlink into the repo, so
  `mask` exits with "File ... already exists". Stop it and rely on the absence
  of a restart trigger, or move the symlink aside.
- Restarting frank-large needs the quadlet source back in
  `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
  the unit from that file only. After the restore the ledger read `400384
  positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 15.2 GiB left.