serverAppConfigs/serverAppConfigs-docs/LLM/x4-pair-experiment.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU	What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.	llm experiment dankclaw llama.cpp vulkan cuda strix-halo rtx3090 flash-next pipeline-parallelism	2026-09-16

The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU

The question this answers: is there a 3090 + Strix Halo combination worth building, given that the only link between them is the 3090's PCIe Gen4 x4 (measured: LnkSta: Speed 16GT/s, Width x4 under load, Width x16 capable; idle-downshifts to Gen1 x4).

Run 2026-09-16 with mainline llama.cpp 37b53fd, a CUDA + Vulkan dual-backend build (build-dual, -DGGML_CUDA=ON -DGGML_VULKAN=ON), both Dankclaw LLM services stopped per the benchmarking policy.

The tensor inventory decides everything

From the 111 GiB GGUF (mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M, 1224 tensors, 176.94 B params, A3B active):

group	size
ffn_{gate,up,down}_exps × 48 layers, 512 experts each	75.0 GiB
per_layer_token_embd — one PLE lookup table	32.8 GiB
everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output)	3.2 GiB

• 48 layers, of which 12 are full attention (indices [3,7,11,15,19,23,27,31,35,39,43,47], every 4th — full_attention_interval = 4) and 36 are GDN/linear.

• Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads 5.5 GiB of KV per token.

• A layer split moves whole layers, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all.

The GGUF comes from mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF, which ships this quant as three .partNofM fragments (37.00 + 37.00 + 36.97 = 110.97 GiB) concatenated in numeric order. The same repo carries the smaller rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02, i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K 74.92 GiB — which is where a "does a smaller quant buy anything?" test would start (it buys memory, ~5 % decode, and costs quality; see the write-up below).

That is why the interesting split is not "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term.

Results

llama-bench, same model. Shallow runs are -p 512,16384 -n 64; the depth-matched pair is -n 32 -d 65536. Note llama-bench's tg instances run with n_prompt = 0, so a plain tg number is ~zero depth: -d is required to put KV in the picture.

config	placement	pp512	pp16384	tg (≈0 depth)	tg @64k depth
A	iGPU trunk + experts on CPU	128.7	122.1	12.10	—
A*	iGPU trunk and experts (Vulkan1), PLE on CPU	456.7	403.1	32.64	25.77 (pp512@64k 231.8)
G2	pure Vulkan layer split, iGPU 85 % / 3090 15 %	—	—	—	27.55 (pp512@64k 224.3)
H	cross-vendor layer split, CUDA0 15 % / Vulkan1 85 %	—	—	—	27.23 (pp512@64k 268.5)
B′	trunk + KV on CUDA0, experts on CPU	60.1	81.4	20.46	—
G1	the design: trunk + KV on Vulkan0, experts on Vulkan1 (-ot)	—	—	abort	—
B	trunk + KV on CUDA0, experts on Vulkan1 (-ot)	—	—	abort	—
B2/B3	per-layer -ot / -sm layer -ts across CUDA+Vulkan	—	—	load failure	—

Real placement in the split runs, sampled from nvidia-smi during the run: 12,804–12,832 MiB resident on the 3090 in G2 and 14,994 MiB in H — the split is genuine, not a mislabelled single-device run (see Finding 5 for why that check is mandatory here).

Reference points on the same silicon: halogen (deployed) ≈ 25 tok/s at real contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the 3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill ceiling 192 tok/s.

Finding 1 — per-tensor placement across devices is killed by one flag: -ot

The design (trunk + KV on the 3090, experts on the iGPU) needs --override-tensor to place two different tensor groups on two devices. That aborts in the scheduler:

ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)

The mechanism, read out of the source rather than inferred:

// llama-context.cpp:429  — the scheduler only gets multi-device copy slots if:
bool pipeline_parallel =
    model.n_devices() > 1 && ... &&
    !model.has_tensor_overrides();          // ← line 433; a required term
// llama-context.cpp:605 — that flag is passed straight to the scheduler:
sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...));
// ggml-backend.cpp:25  — where it decides how many copies each tensor may have:
sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1;

With n_copies == 1 the scheduler cannot give a pre-allocated (weight) tensor a second home, so a weight whose buffer belongs to a non-primary device has no backend and the graph build aborts. has_tensor_overrides is consulted in exactly one place in the whole tree (llama-context.cpp:433), so no other flag can rescue it.

• The CPU is the only working target. -ot exps=CPU runs because the CPU backend is last in the scheduler's list and host buffers are always readable (configs A and B′). A device target is what aborts.
• Pure Vulkan does not change this. Config G1 put both Vulkan devices genuinely in the device list (verified by the correct -dev syntax, see Finding 5) and aborted with the byte-identical message. The blocker is the override/scheduler interaction, not a CUDA-vs-Vulkan driver mismatch.

Finding 2 — two-device layer splits DO work, in both flavours

With no tensor overrides, pipeline_parallel stays on and the scheduler happily runs a graph across two device backends. Two arrangements were measured at 64k depth:

• G2 — pure Vulkan (-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15): iGPU takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last 7 layers (which include attention layers 43 and 47, the only KV-bearing layers in that range). 27.55 tok/s vs 25.77 for the iGPU alone: +6.9 %.
• H — cross-vendor (-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85): the 3090 runs its layers on its CUDA backend, the iGPU on Vulkan. 27.23 tok/s and pp512@64k 268.5 against 231.8 for the iGPU alone: same decode, +16 % prefill.

So the pair is expressible — just not in the shape the design wanted.

Finding 3 — the iGPU alone is competitive, and that was not expected

Config A* — mainline llama.cpp, Vulkan/RADV, everything on the iGPU, PLE in mmap page cache — delivers 32.6 tok/s shallow and 25.8 tok/s at a 64k-deep context, i.e. at fleet-realistic depth it matches halogen's deployed ~25 tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090 sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at 99–100 % busy, so the work is genuinely there and not silently CPU-side.

The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the LPDDR's shared 256 GB/s.

Why this is still not deployable here: the run committed 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving 5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare, because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape exists. The layer split does relieve some of that pressure (the 3090 absorbs ~12.8–15 GiB of layer+KV) but it does not change the class of the result.

Finding 4 — the split cannot reach the design's premise

The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it:

• Moving an attention layer to the 3090 costs 1.62 GiB (its MoE experts travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB.
• Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable is exceeded. Feasible only to ~64k context — precisely where the KV term is small enough not to matter (the +7 % measured above is that residual).
• The 32.8 GiB PLE table must live on the main device (the first in the -dev list, or whichever -ts share owns the output layer), so the iGPU keeps a fixed 32.8 GiB no matter how the layers are split.

-ot per_layer_token_embd=CPU would move the PLE — but it is an override, and Finding 1 applies: it turns multi-device scheduling off.

Finding 5 — the harness-parsing trap that invalidated the first round

The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion was an artifact. llama-bench --help reads:

-dev, --device <dev0/dev1/...>        multi values separated by ','
-ts,  --tensor-split <ts0/ts1/..>     multi values separated by ','

parse_devices_arg splits each -dev value on / into a device list, and the outer loop splits the argument on , into separate sweep values. So -dev CUDA0,Vulkan1 means two single-device configurations (CUDA0 alone, then Vulkan1 alone) — never a two-device run. Consequences that looked like findings:

• "the abort" — a single-device run with -ot exps=Vulkan1 placing a weight on a device that is not in the list at all (Finding 1's abort, for the trivial reason).
• "load failure" for -sm layer -ts 0.25,0.75 — -ts 0.25,0.75 is two sweep values, each becoming [0.25, 0, 0, …] / [0.75, 0, 0, …], i.e. device 0 takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd.

The tell that the runs were single-device: the bench table's own dev column printed Vulkan1 for every row. Four separate "iGPU-only" runs agree to within 0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device sweep looks like.

Rules that follow. In llama-bench, / separates devices within one configuration and , separates configurations to sweep; -dev and -ts both follow this. And never trust a placement-dependent number without an independent check of where the memory went — nvidia-smi sampled during the run is what distinguishes a real split from a mislabelled sweep.

Other landmines walked into (worth knowing before the next run)

• llama-cli in conversation mode with EOF on stdin spams > forever — 3.2 GB in a few minutes, no error, no exit. Use llama-server for scripted runs, or -st/--single-turn with llama-cli. (-no-cnv is not a flag in this build.)
• A dead rootless podman pause process takes the whole host's container management down. The first invalid internal status in the user journal is 18:58:30, from containers whose healthchecks shell out to the podman CLI — i.e. the pause process died while the last bench was still running, not after the runs finished. Every podman command for the user then failed with invalid internal status … could not find any running process. The trigger is not established: journalctl -k holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs squeezed it out") is a guess and is recorded as one. podman system migrate — the remedy podman itself names — panics in 5.7.0 (teardownNetworkBackend nil deref). The rootless netns that holds podman's bridge addresses is gone with it, so aardvark-dns cannot bind 10.89.x.1:53 and every container start fails. Repair: reboot (it clears /run/user/<uid> and the whole host comes back — 28 quadlets, WantedBy present, linger on).
• A Network=host service is not reachable from a bridge container at the host's LAN IP. netavark DNATs published ports only; a bare host-netns bind is refused. Use host.containers.internal (169.254.1.2). This was found the same evening when hermes.franksdank.cloud 502'd — fixed in the Caddyfile and guarded by a strengthened seam (tests/hermes_webui.bats).
• The memory budget includes the co-tenants' patience. The iGPU runs held 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log proves damage came of it, but running a model that big on a host that also serves ~28 containers is asking the question — do it in a window, not alongside.

Conclusion

The fleet's answer is unchanged — two independent endpoints (halogen-flash-server on the iGPU via halogen, frank-zealot-qwen38-gsq-rco holding a whole 27B in the 3090's 24 GB) — but two claims from the first round of this experiment are now corrected:

1. Cross-vendor layer split is possible (measured: CUDA0 + Vulkan1, 27.23 tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint (frank-zealot holds 22.4 of the card's 24.6 GiB). An operator who wants the 7 % must give up the fleet's only image-capable endpoint.
2. Per-tensor placement across two devices is impossible while -ot is used, for a scheduler-mechanical reason, regardless of driver or vendor mix.

The durable surprise worth keeping: mainline llama.cpp + Vulkan on gfx1151 is a viable engine for this class of model (25.8 tok/s at 64k depth, and 27.6 with a two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did not show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better engine than halogen?" question should start here, with the memory budget as the first constraint rather than the last.

[INFERENCE] Extrapolating A*'s measured KV slope (32.64 → 25.77 from 0 to 64k depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the worse mainline's position, and depth is filled by prefill that mainline would have to spend ~200k/232 ≈ 14 minutes on.

Reproducing

# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16

# device map on this host
CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT

export GGML_CUDA_NO_PINNED=1

# A* — iGPU only (the single-device baseline), depth-matched
llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1

# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration)
llama-bench -m <gguf> -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \
  -ngl 999 -n 32 -d 65536 -r 1

# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan
llama-bench -m <gguf> -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \
  -ngl 999 -n 32 -d 65536 -r 1

# proof of placement — sample the 3090 while the run is loaded
nvidia-smi --query-gpu=memory.used --format=csv,noheader   # expect 12–15 GiB

Operational notes for the next run:

• Mainline's pinning guard is GGML_CUDA_NO_PINNED; ik used GGML_CUDA_NO_PINNED_WEIGHTS. Without it a CPU-overridden tensor can land in a pinned CUDA_Host buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants.
• systemctl mask is refused for a relinked root unit: frank-zealot's /etc/systemd/system/frank-zealot.service is a symlink into the repo, so mask exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside.
• Restarting frank-dragoon needs the quadlet source back in ~/.config/containers/systemd/ plus daemon-reload; the generator re-creates the unit from that file only. After the restore (and the host reboot, see the landmines) the ledger read 400384 positions, ~27.2 GiB against a 33.5 GiB budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB before the reboot: the same model, a more fragmented host afterwards — the engine reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints answered /health and served /v1/models under their names.

Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers

Same host, same build-dual, same test shapes (-r 1, f16 KV, flash-attn auto, -d 65536 for depth). The repo ships i1-Q4_K_S as three .partNofM fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against 110.97 for the Q4_K_M payload the earlier numbers came from.

test	Q4_K_M	Q4_K_S	Δ
A* iGPU-only pp512 / pp16384 / tg64	456.7 / 403.1 / 32.64	466.4 / 409.6 / 33.50	+2.1 / +1.6 / +2.6 %
A* iGPU-only pp512 / tg32 @64k	231.8 / 25.77	232.0 / 26.47	~0 / +2.7 %
CUDA0/Vulkan1 split pp512 / tg32 @64k	268.5 / 27.23	268.0 / 28.35	~0 / +4.1 %
Vulkan1/Vulkan0 split pp512 / tg32 @64k	224.3 / 27.55	233.6 / 28.11	+4.2 / +2.0 %

Memory sampled through the run (/proc/meminfo, mem_info_gtt_used and nvidia-smi, every 15 s): peak 3090 VRAM 13,466 MiB (the CUDA-side split), peak iGPU GTT 71.6 GiB (A*), and host RAM bottomed at 10.0 GiB available during A* — against ~5 GiB for the Q4_K_M A* run, so the 6.9 GiB smaller file bought roughly that much headroom.

Read it as a memory lever with a small speed dividend: 2–4 % for 6.9 GiB. Nothing structural moves — -ot still gates per-tensor placement and a layer split still cannot consolidate KV on the 3090. If the aim were to deploy mainline on this iGPU, i1-IQ4_XS (90.78 GiB, single file, sha256 b4787265…d0574f9d) is the rung that changes the memory arithmetic and q8_0 KV is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.

The link, measured (2026-09-16)

pcie-link-test.cu — CUDA, pinned host buffers, cudaEvent timing, both services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout.

size	H2D sync	H2D async	D2H sync	1× sync xfer	pageable H2D
3 MiB	6.65	6.71	6.54	474 µs	6.33
16 MiB	6.72	6.72	6.59	2.50 ms	6.61
64 MiB	6.72	6.71	6.60	9.97 ms	6.64
256 MiB	6.73	6.73	6.60	39.9 ms	6.67
1 GiB	6.72	6.72	6.60	160 ms	6.67
2 GiB	6.72	6.72	6.60	320 ms	6.67

Three readings:

• 6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling (D2H 6.60 = 84 %). The card is x16-capable and the Strix Halo port beneath it is x8-capable (LnkCap: 16GT/s, Width x8 on 00:03.1), so the x4 lives in the adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under load — which these numbers prove on their own, since Gen1 x4 tops out near 1 GB/s.
• Async buys nothing and pinning buys ~1 %: one stream of copies already saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and pageable transfers run within noise of pinned ones. On this host the copies are link-limited rather than staging-limited, so the usual "pageable costs 2×" rule does not apply.
• A 3 MiB slab costs 474 µs, essentially 3 MiB / 6.72 GB/s (447 µs) plus ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead.

What that means for an expert-streaming engine

Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB expert mass) — 580 × 3 MiB slabs:

assumption	per-token link time	ceiling
every slab misses (bandwidth-bound)	272 ms	3.7 tok/s
20 % resident in a 24 GiB card's expert cache	217 ms	4.6 tok/s
50 % hit rate (generous locality)	136 ms	7.4 tok/s
the same bytes computed in host RAM at 256 GB/s	7.1 ms	the link is 31× slower than the memory behind it

Against this host's measured alternatives — 20.46 tok/s for llama.cpp with the experts computed on the CPU (link carries activations only), 25 for halogen, 26.5 for mainline on the iGPU — an engine that streams experts across this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9 tok/s).

For the specific engine that raised the question: FreeToken v0.1.2 keeps expert weights in host RAM with an LRU GPU cache — exactly this streaming shape — and it loads only modelopt block-FP8 HF safetensors for qwen3_5_moe, so a Q4_K_M GGUF is not loadable at all before bandwidth enters the discussion.