serverAppConfigs/serverAppConfigs-docs/LLM/x4-pair-experiment.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU	What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle.	llm experiment dankclaw llama.cpp vulkan strix-halo rtx3090 flash-next pipeline-parallelism	2026-09-16

The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU

The question this answers: is there a 3090 + Strix Halo combination worth building, given that the only link between them is the 3090's PCIe Gen4 x4 (measured: LnkSta: Speed 16GT/s, Width x4 under load, Width x16 capable; idle-downshifts to Gen1 x4).

Run 2026-09-16 with mainline llama.cpp 37b53fd, a CUDA + Vulkan dual-backend build (build-dual, -DGGML_CUDA=ON -DGGML_VULKAN=ON), both Dankclaw LLM services stopped per the benchmarking policy.

The tensor inventory decides everything

From the 111 GiB GGUF (mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M, 1224 tensors, 176.94 B params, A3B active):

group	size
ffn_{gate,up,down}_exps × 48 layers, 512 experts each	75.0 GiB
per_layer_token_embd — one PLE lookup table	32.8 GiB
everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output)	3.2 GiB

• 48 layers, of which 12 are full attention (indices [3,7,11,15,19,23,27,31,35,39,43,47], every 4th — full_attention_interval = 4) and 36 are GDN/linear.
• Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads 5.5 GiB of KV per token.

That is why the interesting split is not "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term.

Results

llama-bench, same model, 2 tests per config (-p 512,16384 -n 64), plus a depth-matched pair (-n 32 -d 65536). Note llama-bench's tg instances run with n_prompt = 0, so a plain tg number is ~zero depth: -d is required to put KV in the picture.

config	placement	pp512	pp16384	tg (≈0 depth)	tg32 @64k depth
A	iGPU trunk (Vulkan1) + experts on CPU	128.7	122.1	12.10	—
A*	iGPU trunk and experts (Vulkan1), PLE on CPU	456.7	403.1	32.64	25.77 (pp512@64k = 231.8)
B	trunk + KV on CUDA0, experts on Vulkan1	—	—	abort	—
B2	attention layers → CUDA0, rest → Vulkan1 (per-layer -ot)	—	—	load failure	—
B3	25 % of layers → CUDA0, rest → Vulkan1 (-sm layer -ts 0.25,0.75)	—	—	load failure	—
B′	trunk + KV on CUDA0, experts on CPU	60.1	81.4	20.46	—

Reference points on the same silicon: halogen (deployed) ≈ 25 tok/s at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s.

Finding 1 — the cross-backend split is not expressible

Per-tensor placement between two device backends aborts in the scheduler:

ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)

A weight tensor living in one device's buffer cannot be consumed by an op the scheduler placed on another device. The CPU is the exception — -ot exps=CPU works — because the CPU backend is the universal fallback and host buffers can be copied. Per-layer cross-backend splits (-sm layer -ts, and layer-scoped -ot) are no better: both failed at model load with no further diagnostic.

So on this build, CUDA + Vulkan cannot serve one model: the 3090 can be the trunk or the expert host, but not the complement of an iGPU-resident peer. That is the same wall the R9700 pair hit from the other side (collectives), and it is why every working multi-device llama.cpp deployment in this fleet is either single-vendor (frank-medium: two AMD cards) or device+CPU.

Finding 2 — the iGPU alone is competitive, and that was not expected

Config A* — mainline llama.cpp, Vulkan/RADV, everything on the iGPU, PLE in mmap page cache — delivers 32.6 tok/s shallow and 25.8 tok/s at a 64k-deep context, i.e. at fleet-realistic depth it matches halogen's deployed ~25 tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at 99–100 % busy, so the work is genuinely there and not silently CPU-side.

The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the LPDDR's shared 256 GB/s.

Why this is still not deployable here: the run committed 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape exists.

Finding 3 — the fast pool is too small to help

B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — worse than A*, which does not use the 3090 at all. And the expressible version of the KV-on-VRAM design cannot reach its own premise: moving an attention layer to the 3090 costs 1.62 GiB (its MoE experts travel with it, since only whole layers move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16, 4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context — precisely where the KV term is small enough not to matter.

Conclusion

On this host the pair is not additive, and the measured best arrangement remains two independent endpoints (halogen-flash-server on the iGPU via halogen, frank-small-qwen38-gsq-rco holding a whole 27B in the 3090's 24 GB). A single process spanning both is blocked by the scheduler, and every CPU-side fallback is slower than what the iGPU does alone.

The one durable surprise worth keeping: mainline llama.cpp + Vulkan on gfx1151 is a viable engine for this class of model (25.8 tok/s at 64k depth), which the 2026-09-15 mainline measurement of 1.01 tok/s did not show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better engine than halogen?" question should start here, with the memory budget as the first constraint rather than the last.

Reproducing

# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16

# device map on this host
CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT

# config A* (the winner), depth-matched
GGML_CUDA_NO_PINNED=1 llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1

Operational notes for the next run:

• Mainline's pinning guard is GGML_CUDA_NO_PINNED; ik used GGML_CUDA_NO_PINNED_WEIGHTS. Without it a CPU-overridden tensor can land in a pinned CUDA_Host buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants.
• systemctl mask is refused for a relinked root unit: frank-small's /etc/systemd/system/frank-small.service is a symlink into the repo, so mask exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside.
• Restarting frank-large needs the quadlet source back in ~/.config/containers/systemd/ plus daemon-reload; the generator re-creates the unit from that file only. After the restore the ledger read 400384 positions, ~27.2 GiB against a 33.5 GiB budget — no clamp — with 15.2 GiB left.