serverAppConfigs/serverAppConfigs-docs/LLM/x4-pair-experiment.md

---
type: reference
title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.
tags:
  - llm
  - experiment
  - dankclaw
  - llama.cpp
  - vulkan
  - cuda
  - strix-halo
  - rtx3090
  - flash-next
  - pipeline-parallelism
timestamp: 2026-09-16
---

# The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU

The question this answers: **is there a 3090 + Strix Halo combination worth
building**, given that the only link between them is the 3090's PCIe Gen4 x4
(measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable;
idle-downshifts to Gen1 x4).

Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend
build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM
services stopped per the benchmarking policy.

## The tensor inventory decides everything

From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
1224 tensors, 176.94 B params, **A3B active**):

| group | size |
|---|---|
| `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** |
| `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** |
| everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** |

- 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`,
  every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
  position, a 200k-token conversation reads **5.5 GiB of KV per token**.

- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
  every layer (75 GiB / 48). Attention layers are spread across the whole depth
  (every 4th), so no contiguous layer range can hold them all.

The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which
ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 =
110.97 GiB) concatenated in numeric order. The same repo carries the smaller
rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02,
i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K
74.92 GiB — which is where a "does a smaller quant buy anything?" test would
start (it buys memory, ~5 % decode, and costs quality; see the write-up below).

That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
"get KV onto the 936 GB/s pool", because KV is the O(context) term.

## Results

`llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the
depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run
with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required
to put KV in the picture.

| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth |
|---|---|---|---|---|---|
| **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — |
| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) |
| **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) |
| **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) |
| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — |
| **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — |

Real placement in the split runs, sampled from `nvidia-smi` during the run:
**12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the
split is genuine, not a mislabelled single-device run (see Finding 5 for why
that check is mandatory here).

Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real
contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the
3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill
ceiling 192 tok/s.

## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot`

The design (trunk + KV on the 3090, experts on the iGPU) needs
`--override-tensor` to place two *different* tensor groups on two devices. That
aborts in the scheduler:

```
ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)
```

The mechanism, read out of the source rather than inferred:

```c
// llama-context.cpp:429  — the scheduler only gets multi-device copy slots if:
bool pipeline_parallel =
    model.n_devices() > 1 && ... &&
    !model.has_tensor_overrides();          // ← line 433; a required term
// llama-context.cpp:605 — that flag is passed straight to the scheduler:
sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...));
// ggml-backend.cpp:25  — where it decides how many copies each tensor may have:
sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1;
```

With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a
second home, so a weight whose buffer belongs to a non-primary device has no
backend and the graph build aborts. `has_tensor_overrides` is consulted in
**exactly one place in the whole tree** (llama-context.cpp:433), so no other
flag can rescue it.

- **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU
  backend is last in the scheduler's list and host buffers are always readable
  (configs A and B′). A *device* target is what aborts.
- **Pure Vulkan does not change this.** Config G1 put both Vulkan devices
  genuinely in the device list (verified by the correct `-dev` syntax, see
  Finding 5) and aborted with the byte-identical message. The blocker is the
  override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch.

## Finding 2 — two-device layer splits DO work, in both flavours

With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily
runs a graph across two device backends. Two arrangements were measured at 64k
depth:

- **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU
  takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last
  7 layers (which include attention layers 43 and 47, the only KV-bearing layers
  in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**.
- **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090
  runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s**
  and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 %
  prefill**.

So the pair *is* expressible — just not in the shape the design wanted.

## Finding 3 — the iGPU alone is competitive, and that was not expected

Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in
mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep
context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25
tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090
sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %**
busy, so the work is genuinely there and not silently CPU-side.

The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s
from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the
LPDDR's shared 256 GB/s.

**Why this is still not deployable here:** the run committed **116 of the host's
121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving
5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare,
because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory
bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape
exists. The layer split does relieve some of that pressure (the 3090 absorbs
~12.8–15 GiB of layer+KV) but it does not change the class of the result.

## Finding 4 — the split cannot reach the design's premise

The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it:

- Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts
  travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB.
- Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable
  is exceeded. Feasible only to ~64k context — precisely where the KV term is
  small enough not to matter (the +7 % measured above is that residual).
- The 32.8 GiB PLE table must live on the *main* device (the first in the
  `-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps
  a fixed 32.8 GiB no matter how the layers are split.

`-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and
Finding 1 applies: it turns multi-device scheduling off.

## Finding 5 — the harness-parsing trap that invalidated the first round

The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion
was an artifact. `llama-bench --help` reads:

```
-dev, --device <dev0/dev1/...>        multi values separated by ','
-ts,  --tensor-split <ts0/ts1/..>     multi values separated by ','
```

`parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the
outer loop splits the argument on `,` into separate *sweep values*. So
`-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone,
then Vulkan1 alone) — never a two-device run. Consequences that looked like
findings:

- "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a
  device that is not in the list at all (Finding 1's abort, for the trivial
  reason).
- "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep
  values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0
  takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd.

The tell that the runs were single-device: the bench table's own `dev` column
printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within
0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device
sweep looks like.

**Rules that follow.** In llama-bench, `/` separates devices *within one
configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both
follow this. And never trust a placement-dependent number without an independent
check of where the memory went — `nvidia-smi` sampled during the run is what
distinguishes a real split from a mislabelled sweep.

## Other landmines walked into (worth knowing before the next run)

- **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** —
  3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
  runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
  build.)
- **A dead rootless podman pause process takes the whole host's container
  management down.** The first `invalid internal status` in the user journal is
  **18:58:30**, from containers whose healthchecks shell out to the podman CLI —
  i.e. the pause process died *while the last bench was still running*, not
  after the runs finished. Every `podman` command for the user then failed with
  `invalid internal status … could not find any running process`. The trigger is
  **not** established: `journalctl -k` holds no OOM kill for that window, so the
  tempting reading ("the 116 GiB runs squeezed it out") is a guess and is
  recorded as one.
  `podman system migrate` — the remedy podman itself names — **panics**
  in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
  podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
  `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
  `/run/user/<uid>` and the whole host comes back — 28 quadlets, `WantedBy`
  present, linger on).
- **A `Network=host` service is not reachable from a bridge container at the
  host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind
  is refused. Use `host.containers.internal` (169.254.1.2). This was found the
  same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
  guarded by a strengthened seam (`tests/hermes_webui.bats`).
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
  116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
  proves damage came of it, but running a model that big on a host that also
  serves ~28 containers is asking the question — do it in a window, not
  alongside.

## Conclusion

The fleet's answer is unchanged — **two independent endpoints**
([[halogen-flash-server]] on the iGPU via halogen,
[[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
claims from the first round of this experiment are now corrected:

1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
   tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
   and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint
   (`frank-zealot` holds 22.4 of the card's 24.6 GiB). An operator who wants the
   7 % must give up the fleet's only image-capable endpoint.
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
   for a scheduler-mechanical reason, regardless of driver or vendor mix.

The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a
viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a
two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did
*not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening
the "is there a better engine than halogen?" question should start here, with the
memory budget as the first constraint rather than the last.

[INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k
depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where
halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the
worse mainline's position, and depth is filled by prefill that mainline would
have to spend ~200k/232 ≈ 14 minutes on.

## Reproducing

```bash
# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16

# device map on this host
CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT

export GGML_CUDA_NO_PINNED=1

# A* — iGPU only (the single-device baseline), depth-matched
llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1

# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration)
llama-bench -m <gguf> -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \
  -ngl 999 -n 32 -d 65536 -r 1

# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan
llama-bench -m <gguf> -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \
  -ngl 999 -n 32 -d 65536 -r 1

# proof of placement — sample the 3090 while the run is loaded
nvidia-smi --query-gpu=memory.used --format=csv,noheader   # expect 12–15 GiB
```

Operational notes for the next run:

- Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used
  `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
  pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
  host's co-tenants.
- `systemctl mask` is **refused** for a relinked root unit: frank-zealot's
  `/etc/systemd/system/frank-zealot.service` is a symlink into the repo, so
  `mask` exits with "File ... already exists". Stop it and rely on the absence
  of a restart trigger, or move the symlink aside.
- Restarting frank-dragoon needs the quadlet source back in
  `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
  the unit from that file only. After the restore (and the host reboot, see the
  landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB
  budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB before
  the reboot: the same model, a more fragmented host afterwards — the engine
  reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints
  answered `/health` and served `/v1/models` under their names.


## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers

Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
110.97 for the Q4_K_M payload the earlier numbers came from.

| test | Q4_K_M | Q4_K_S | Δ |
|---|---|---|---|
| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |

Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
smaller file bought roughly that much headroom.

**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
Nothing structural moves — `-ot` still gates per-tensor placement and a layer
split still cannot consolidate KV on the 3090. If the aim were to *deploy*
mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.

## The link, measured (2026-09-16)

`pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both
services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout.

| size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D |
|---|---|---|---|---|---|
| 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 |
| 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 |
| 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 |
| 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 |
| 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 |
| 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 |

Three readings:

- **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is
  x16-capable and the Strix Halo port beneath it is x8-capable
  (`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the
  adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under
  load — which these numbers prove on their own, since Gen1 x4 tops out near
  1 GB/s.
- **Async buys nothing and pinning buys ~1 %**: one stream of copies already
  saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and
  pageable transfers run within noise of pinned ones. On this host the copies
  are link-limited rather than staging-limited, so the usual "pageable costs 2×"
  rule does not apply.
- **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus
  ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead.

### What that means for an expert-streaming engine

Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB
expert mass) — 580 × 3 MiB slabs:

| assumption | per-token link time | ceiling |
|---|---|---|
| every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** |
| 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s |
| 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s |
| *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it |

Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with
the experts *computed* on the CPU (link carries activations only), 25 for
halogen, 26.5 for mainline on the iGPU — an engine that *streams* experts across
this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9
tok/s).

For the specific engine that raised the question: FreeToken v0.1.2 keeps expert
weights in host RAM with an LRU GPU cache — exactly this streaming shape — and
it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M
**GGUF** is not loadable at all before bandwidth enters the discussion.