serverAppConfigs/serverAppConfigs-docs/LLM/x4-pair-experiment.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU	What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.	llm experiment dankclaw llama.cpp vulkan cuda strix-halo rtx3090 flash-next pipeline-parallelism	2026-09-16

The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU

The question this answers: is there a 3090 + Strix Halo combination worth building, given that the only link between them is the 3090's PCIe Gen4 x4 (measured: LnkSta: Speed 16GT/s, Width x4 under load, Width x16 capable; idle-downshifts to Gen1 x4).

Run 2026-09-16 with mainline llama.cpp 37b53fd, a CUDA + Vulkan dual-backend build (build-dual, -DGGML_CUDA=ON -DGGML_VULKAN=ON), both Dankclaw LLM services stopped per the benchmarking policy.

The tensor inventory decides everything

From the 111 GiB GGUF (mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M, 1224 tensors, 176.94 B params, A3B active):

group	size
ffn_{gate,up,down}_exps × 48 layers, 512 experts each	75.0 GiB
per_layer_token_embd — one PLE lookup table	32.8 GiB
everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output)	3.2 GiB

• 48 layers, of which 12 are full attention (indices [3,7,11,15,19,23,27,31,35,39,43,47], every 4th — full_attention_interval = 4) and 36 are GDN/linear.

• Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads 5.5 GiB of KV per token.

• A layer split moves whole layers, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all.

The GGUF comes from mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF, which ships this quant as three .partNofM fragments (37.00 + 37.00 + 36.97 = 110.97 GiB) concatenated in numeric order. The same repo carries the smaller rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02, i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K 74.92 GiB — which is where a "does a smaller quant buy anything?" test would start (it buys memory, ~5 % decode, and costs quality; see the write-up below).

That is why the interesting split is not "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term.

Results

llama-bench, same model. Shallow runs are -p 512,16384 -n 64; the depth-matched pair is -n 32 -d 65536. Note llama-bench's tg instances run with n_prompt = 0, so a plain tg number is ~zero depth: -d is required to put KV in the picture.

config	placement	pp512	pp16384	tg (≈0 depth)	tg @64k depth
A	iGPU trunk + experts on CPU	128.7	122.1	12.10	—
A*	iGPU trunk and experts (Vulkan1), PLE on CPU	456.7	403.1	32.64	25.77 (pp512@64k 231.8)
G2	pure Vulkan layer split, iGPU 85 % / 3090 15 %	—	—	—	27.55 (pp512@64k 224.3)
H	cross-vendor layer split, CUDA0 15 % / Vulkan1 85 %	—	—	—	27.23 (pp512@64k 268.5)
B′	trunk + KV on CUDA0, experts on CPU	60.1	81.4	20.46	—
G1	the design: trunk + KV on Vulkan0, experts on Vulkan1 (-ot)	—	—	abort	—
B	trunk + KV on CUDA0, experts on Vulkan1 (-ot)	—	—	abort	—
B2/B3	per-layer -ot / -sm layer -ts across CUDA+Vulkan	—	—	load failure	—

Real placement in the split runs, sampled from nvidia-smi during the run: 12,804–12,832 MiB resident on the 3090 in G2 and 14,994 MiB in H — the split is genuine, not a mislabelled single-device run (see Finding 5 for why that check is mandatory here).

Reference points on the same silicon: halogen (deployed) ≈ 25 tok/s at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s.

Finding 1 — per-tensor placement across devices is killed by one flag: -ot

The design (trunk + KV on the 3090, experts on the iGPU) needs --override-tensor to place two different tensor groups on two devices. That aborts in the scheduler:

ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)

The mechanism, read out of the source rather than inferred:

// llama-context.cpp:429  — the scheduler only gets multi-device copy slots if:
bool pipeline_parallel =
    model.n_devices() > 1 && ... &&
    !model.has_tensor_overrides();          // ← line 433; a required term
// llama-context.cpp:605 — that flag is passed straight to the scheduler:
sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...));
// ggml-backend.cpp:25  — where it decides how many copies each tensor may have:
sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1;

With n_copies == 1 the scheduler cannot give a pre-allocated (weight) tensor a second home, so a weight whose buffer belongs to a non-primary device has no backend and the graph build aborts. has_tensor_overrides is consulted in exactly one place in the whole tree (llama-context.cpp:433), so no other flag can rescue it.

• The CPU is the only working target. -ot exps=CPU runs because the CPU backend is last in the scheduler's list and host buffers are always readable (configs A and B′). A device target is what aborts.
• Pure Vulkan does not change this. Config G1 put both Vulkan devices genuinely in the device list (verified by the correct -dev syntax, see Finding 5) and aborted with the byte-identical message. The blocker is the override/scheduler interaction, not a CUDA-vs-Vulkan driver mismatch.

Finding 2 — two-device layer splits DO work, in both flavours

With no tensor overrides, pipeline_parallel stays on and the scheduler happily runs a graph across two device backends. Two arrangements were measured at 64k depth:

• G2 — pure Vulkan (-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15): iGPU takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last 7 layers (which include attention layers 43 and 47, the only KV-bearing layers in that range). 27.55 tok/s vs 25.77 for the iGPU alone: +6.9 %.
• H — cross-vendor (-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85): the 3090 runs its layers on its CUDA backend, the iGPU on Vulkan. 27.23 tok/s and pp512@64k 268.5 against 231.8 for the iGPU alone: same decode, +16 % prefill.

So the pair is expressible — just not in the shape the design wanted.

Finding 3 — the iGPU alone is competitive, and that was not expected

Config A* — mainline llama.cpp, Vulkan/RADV, everything on the iGPU, PLE in mmap page cache — delivers 32.6 tok/s shallow and 25.8 tok/s at a 64k-deep context, i.e. at fleet-realistic depth it matches halogen's deployed ~25 tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at 99–100 % busy, so the work is genuinely there and not silently CPU-side.

The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the LPDDR's shared 256 GB/s.

Why this is still not deployable here: the run committed 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape exists. The layer split does relieve some of that pressure (the 3090 absorbs ~12.8–15 GiB of layer+KV) but it does not change the class of the result.

Finding 4 — the split cannot reach the design's premise

The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it:

• Moving an attention layer to the 3090 costs 1.62 GiB (its MoE experts travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB.
• Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable is exceeded. Feasible only to ~64k context — precisely where the KV term is small enough not to matter (the +7 % measured above is that residual).
• The 32.8 GiB PLE table must live on the main device (the first in the -dev list, or whichever -ts share owns the output layer), so the iGPU keeps a fixed 32.8 GiB no matter how the layers are split.

-ot per_layer_token_embd=CPU would move the PLE — but it is an override, and Finding 1 applies: it turns multi-device scheduling off.

Finding 5 — the harness-parsing trap that invalidated the first round

The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion was an artifact. llama-bench --help reads:

-dev, --device <dev0/dev1/...>        multi values separated by ','
-ts,  --tensor-split <ts0/ts1/..>     multi values separated by ','

parse_devices_arg splits each -dev value on / into a device list, and the outer loop splits the argument on , into separate sweep values. So -dev CUDA0,Vulkan1 means two single-device configurations (CUDA0 alone, then Vulkan1 alone) — never a two-device run. Consequences that looked like findings:

• "the abort" — a single-device run with -ot exps=Vulkan1 placing a weight on a device that is not in the list at all (Finding 1's abort, for the trivial reason).
• "load failure" for -sm layer -ts 0.25,0.75 — -ts 0.25,0.75 is two sweep values, each becoming [0.25, 0, 0, …] / [0.75, 0, 0, …], i.e. device 0 takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd.

The tell that the runs were single-device: the bench table's own dev column printed Vulkan1 for every row. Four separate "iGPU-only" runs agree to within 0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device sweep looks like.

Rules that follow. In llama-bench, / separates devices within one configuration and , separates configurations to sweep; -dev and -ts both follow this. And never trust a placement-dependent number without an independent check of where the memory went — nvidia-smi sampled during the run is what distinguishes a real split from a mislabelled sweep.

Other landmines walked into (worth knowing before the next run)

• llama-cli in conversation mode with EOF on stdin spams > forever — 3.2 GB in a few minutes, no error, no exit. Use llama-server for scripted runs, or -st/--single-turn with llama-cli. (-no-cnv is not a flag in this build.)
• A dead rootless podman pause process takes the whole host's container management down. The first invalid internal status in the user journal is 18:58:30, from containers whose healthchecks shell out to the podman CLI — i.e. the pause process died while the last bench was still running, not after the runs finished. Every podman command for the user then failed with invalid internal status … could not find any running process. The trigger is not established: journalctl -k holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs squeezed it out") is a guess and is recorded as one. podman system migrate — the remedy podman itself names — panics in 5.7.0 (teardownNetworkBackend nil deref). The rootless netns that holds podman's bridge addresses is gone with it, so aardvark-dns cannot bind 10.89.x.1:53 and every container start fails. Repair: reboot (it clears /run/user/<uid> and the whole host comes back — 28 quadlets, WantedBy present, linger on).
• A Network=host service is not reachable from a bridge container at the host's LAN IP. netavark DNATs published ports only; a bare host-netns bind is refused. Use host.containers.internal (169.254.1.2). This was found the same evening when hermes.franksdank.cloud 502'd — fixed in the Caddyfile and guarded by a strengthened seam (tests/hermes_webui.bats).
• The memory budget includes the co-tenants' patience. The iGPU runs held 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log proves damage came of it, but running a model that big on a host that also serves ~28 containers is asking the question — do it in a window, not alongside.

Conclusion

The fleet's answer is unchanged — two independent endpoints (halogen-flash-server on the iGPU via halogen, frank-small-qwen38-gsq-rco holding a whole 27B in the 3090's 24 GB) — but two claims from the first round of this experiment are now corrected:

1. Cross-vendor layer split is possible (measured: CUDA0 + Vulkan1, 27.23 tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint (frank-small holds 22.4 of the card's 24.6 GiB). An operator who wants the 7 % must give up the fleet's only image-capable endpoint.
2. Per-tensor placement across two devices is impossible while -ot is used, for a scheduler-mechanical reason, regardless of driver or vendor mix.

The durable surprise worth keeping: mainline llama.cpp + Vulkan on gfx1151 is a viable engine for this class of model (25.8 tok/s at 64k depth, and 27.6 with a two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did not show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better engine than halogen?" question should start here, with the memory budget as the first constraint rather than the last.

[INFERENCE] Extrapolating A*'s measured KV slope (32.64 → 25.77 from 0 to 64k depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the worse mainline's position, and depth is filled by prefill that mainline would have to spend ~200k/232 ≈ 14 minutes on.

Reproducing

# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16

# device map on this host
CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT

export GGML_CUDA_NO_PINNED=1

# A* — iGPU only (the single-device baseline), depth-matched
llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1

# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration)
llama-bench -m <gguf> -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \
  -ngl 999 -n 32 -d 65536 -r 1

# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan
llama-bench -m <gguf> -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \
  -ngl 999 -n 32 -d 65536 -r 1

# proof of placement — sample the 3090 while the run is loaded
nvidia-smi --query-gpu=memory.used --format=csv,noheader   # expect 12–15 GiB

Operational notes for the next run:

• Mainline's pinning guard is GGML_CUDA_NO_PINNED; ik used GGML_CUDA_NO_PINNED_WEIGHTS. Without it a CPU-overridden tensor can land in a pinned CUDA_Host buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants.
• systemctl mask is refused for a relinked root unit: frank-small's /etc/systemd/system/frank-small.service is a symlink into the repo, so mask exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside.
• Restarting frank-large needs the quadlet source back in ~/.config/containers/systemd/ plus daemon-reload; the generator re-creates the unit from that file only. After the restore (and the host reboot, see the landmines) the ledger read 400384 positions, ~27.2 GiB against a 33.5 GiB budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB before the reboot: the same model, a more fragmented host afterwards — the engine reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints answered /health and served /v1/models under their names.

Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers

Same host, same build-dual, same test shapes (-r 1, f16 KV, flash-attn auto, -d 65536 for depth). The repo ships i1-Q4_K_S as three .partNofM fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against 110.97 for the Q4_K_M payload the earlier numbers came from.

test	Q4_K_M	Q4_K_S	Δ
A* iGPU-only pp512 / pp16384 / tg64	456.7 / 403.1 / 32.64	466.4 / 409.6 / 33.50	+2.1 / +1.6 / +2.6 %
A* iGPU-only pp512 / tg32 @64k	231.8 / 25.77	232.0 / 26.47	~0 / +2.7 %
CUDA0/Vulkan1 split pp512 / tg32 @64k	268.5 / 27.23	268.0 / 28.35	~0 / +4.1 %
Vulkan1/Vulkan0 split pp512 / tg32 @64k	224.3 / 27.55	233.6 / 28.11	+4.2 / +2.0 %

Memory sampled through the run (/proc/meminfo, mem_info_gtt_used and nvidia-smi, every 15 s): peak 3090 VRAM 13,466 MiB (the CUDA-side split), peak iGPU GTT 71.6 GiB (A*), and host RAM bottomed at 10.0 GiB available during A* — against ~5 GiB for the Q4_K_M A* run, so the 6.9 GiB smaller file bought roughly that much headroom.

Read it as a memory lever with a small speed dividend: 2–4 % for 6.9 GiB. Nothing structural moves — -ot still gates per-tensor placement and a layer split still cannot consolidate KV on the 3090. If the aim were to deploy mainline on this iGPU, i1-IQ4_XS (90.78 GiB, single file, sha256 b4787265…d0574f9d) is the rung that changes the memory arithmetic and q8_0 KV is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.