dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

x4 pair round two: pure Vulkan changes nothing, and the first verdict was wrong

Browse Source 
Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.

Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:

  pure Vulkan  -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
  cross-vendor -dev CUDA0/Vulkan1   -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
  iGPU alone                                       25.77 tok/s @64k, pp512 231.8

+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.

Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
This commit is contained in:
Frank
2026-09-16 19:32:53 -07:00
parent d14e0a94eb
commit 803398051a
5 changed files with 287 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/index.md
+5 -4
View File
@@ -94,10 +94,11 @@ frank-small, port 7771]
for the fleet's image path
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
Gen4 x4). The tensor inventory that decides any split, the two scheduler
limits that kill a cross-backend one, and the finding that the iGPU alone
matches halogen's deployed decode rate while leaving the 3090 idle — at a
memory cost the co-tenants cannot pay
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
that kills per-tensor placement across devices, the two-device layer splits
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
that invalidated the first round, and the iGPU-only result that matches
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
serverAppConfigs-docs/LLM/x4-pair-experiment.md
+186 -48
View File
@@ -1,13 +1,14 @@
---
type: reference
title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle.
description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.
tags:
- llm
- experiment
- dankclaw
- llama.cpp
- vulkan
- cuda
- strix-halo
- rtx3090
- flash-next
@@ -41,6 +42,9 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
position, a 200k-token conversation reads **5.5 GiB of KV per token**.
- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
every layer (75 GiB / 48). Attention layers are spread across the whole depth
(every 4th), so no contiguous layer range can hold them all.
That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
@@ -48,47 +52,88 @@ experts are re-read once per token but they are only ~1.7 GB of traffic — it i
## Results
`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a
depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with
`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to
put KV in the picture.
`llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the
depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run
with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required
to put KV in the picture.
| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg32 @64k depth |
| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth |
|---|---|---|---|---|---|
| **A** | iGPU trunk (Vulkan1) + experts on CPU | 128.7 | 122.1 | 12.10 | — |
| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k = 231.8) |
| **B** | trunk + KV on CUDA0, experts on Vulkan1 | — | — | **abort** | — |
| **B2** | attention layers → CUDA0, rest → Vulkan1 (per-layer `-ot`) | — | — | load failure | — |
| **B3** | 25 % of layers → CUDA0, rest → Vulkan1 (`-sm layer -ts 0.25,0.75`) | — | — | load failure | — |
| **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — |
| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) |
| **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) |
| **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) |
| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — |
| **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — |
Real placement in the split runs, sampled from `nvidia-smi` during the run:
**12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the
split is genuine, not a mislabelled single-device run (see Finding 5 for why
that check is mandatory here).
Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real
contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the
3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill
ceiling 192 tok/s.
## Finding 1 — the cross-backend split is not expressible
## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot`
Per-tensor placement between two **device** backends aborts in the scheduler:
The design (trunk + KV on the 3090, experts on the iGPU) needs
`--override-tensor` to place two *different* tensor groups on two devices. That
aborts in the scheduler:
```
ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)
```
A weight tensor living in one device's buffer cannot be consumed by an op the
scheduler placed on another device. The CPU is the exception — `-ot exps=CPU`
works — because the CPU backend is the universal fallback and host buffers can
be copied. Per-**layer** cross-backend splits (`-sm layer -ts`, and layer-scoped
`-ot`) are no better: both failed at model load with no further diagnostic.
The mechanism, read out of the source rather than inferred:
So on this build, **CUDA + Vulkan cannot serve one model**: the 3090 can be the
trunk *or* the expert host, but not the complement of an iGPU-resident peer.
That is the same wall the R9700 pair hit from the other side (collectives), and
it is why every working multi-device llama.cpp deployment in this fleet is
either single-vendor (frank-medium: two AMD cards) or device+CPU.
```c
// llama-context.cpp:429 — the scheduler only gets multi-device copy slots if:
bool pipeline_parallel =
model.n_devices() > 1 && ... &&
!model.has_tensor_overrides(); // ← line 433; a required term
// llama-context.cpp:605 — that flag is passed straight to the scheduler:
sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...));
// ggml-backend.cpp:25 — where it decides how many copies each tensor may have:
sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1;
```
## Finding 2 — the iGPU alone is competitive, and that was not expected
With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a
second home, so a weight whose buffer belongs to a non-primary device has no
backend and the graph build aborts. `has_tensor_overrides` is consulted in
**exactly one place in the whole tree** (llama-context.cpp:433), so no other
flag can rescue it.
- **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU
backend is last in the scheduler's list and host buffers are always readable
(configs A and B′). A *device* target is what aborts.
- **Pure Vulkan does not change this.** Config G1 put both Vulkan devices
genuinely in the device list (verified by the correct `-dev` syntax, see
Finding 5) and aborted with the byte-identical message. The blocker is the
override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch.
## Finding 2 — two-device layer splits DO work, in both flavours
With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily
runs a graph across two device backends. Two arrangements were measured at 64k
depth:
- **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU
takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last
7 layers (which include attention layers 43 and 47, the only KV-bearing layers
in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**.
- **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090
runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s**
and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 %
prefill**.
So the pair *is* expressible — just not in the shape the design wanted.
## Finding 3 — the iGPU alone is competitive, and that was not expected
Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in
mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep
@@ -106,32 +151,111 @@ LPDDR's shared 256 GB/s.
5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare,
because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory
bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape
exists.
exists. The layer split does relieve some of that pressure (the 3090 absorbs
~12.8–15 GiB of layer+KV) but it does not change the class of the result.
## Finding 3 — the fast pool is too small to help
## Finding 4 — the split cannot reach the design's premise
B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — *worse*
than A\*, which does not use the 3090 at all. And the expressible version of the
KV-on-VRAM design cannot reach its own premise: moving an attention layer to the
3090 costs **1.62 GiB** (its MoE experts travel with it, since only whole layers
move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16,
4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context —
precisely where the KV term is small enough not to matter.
The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it:
- Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts
travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB.
- Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable
is exceeded. Feasible only to ~64k context — precisely where the KV term is
small enough not to matter (the +7 % measured above is that residual).
- The 32.8 GiB PLE table must live on the *main* device (the first in the
`-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps
a fixed 32.8 GiB no matter how the layers are split.
`-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and
Finding 1 applies: it turns multi-device scheduling off.
## Finding 5 — the harness-parsing trap that invalidated the first round
The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion
was an artifact. `llama-bench --help` reads:
```
-dev, --device <dev0/dev1/...> multi values separated by ','
-ts, --tensor-split <ts0/ts1/..> multi values separated by ','
```
`parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the
outer loop splits the argument on `,` into separate *sweep values*. So
`-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone,
then Vulkan1 alone) — never a two-device run. Consequences that looked like
findings:
- "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a
device that is not in the list at all (Finding 1's abort, for the trivial
reason).
- "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep
values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0
takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd.
The tell that the runs were single-device: the bench table's own `dev` column
printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within
0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device
sweep looks like.
**Rules that follow.** In llama-bench, `/` separates devices *within one
configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both
follow this. And never trust a placement-dependent number without an independent
check of where the memory went — `nvidia-smi` sampled during the run is what
distinguishes a real split from a mislabelled sweep.
## Other landmines walked into (worth knowing before the next run)
- **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** —
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
build.)
- **A killed rootless podman pause process takes the whole host's container
management down.** After the experiment the pause process was gone (plausible
cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command
for the user failed with `invalid internal status … could not find any running
process`. `podman system migrate` — the remedy podman itself names — **panics**
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
`/run/user/<uid>` and the whole host comes back — 28 quadlets, `WantedBy`
present, linger on).
- **A `Network=host` service is not reachable from a bridge container at the
host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind
is refused. Use `host.containers.internal` (169.254.1.2). This was found the
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
guarded by a strengthened seam (`tests/hermes_webui.bats`).
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
116 of 121 GiB; the pause-process death above is the visible cost of running a
model that big on a host that also serves.
## Conclusion
On this host the pair is not additive, and the measured best arrangement remains
**two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen,
[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB). A
single process spanning both is blocked by the scheduler, and every CPU-side
fallback is slower than what the iGPU does alone.
The fleet's answer is unchanged — **two independent endpoints**
([[halogen-flash-server]] on the iGPU via halogen,
[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
claims from the first round of this experiment are now corrected:
The one durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151
is a viable engine for this class of model** (25.8 tok/s at 64k depth), which the
2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was
the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better
engine than halogen?" question should start here, with the memory budget as the
first constraint rather than the last.
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint
(`frank-small` holds 22.4 of the card's 24.6 GiB). An operator who wants the
7 % must give up the fleet's only image-capable endpoint.
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
for a scheduler-mechanical reason, regardless of driver or vendor mix.
The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a
viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a
two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did
*not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening
the "is there a better engine than halogen?" question should start here, with the
memory budget as the first constraint rather than the last.
[INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k
depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where
halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the
worse mainline's position, and depth is filled by prefill that mainline would
have to spend ~200k/232 ≈ 14 minutes on.
## Reproducing
@@ -145,9 +269,22 @@ cmake --build build-dual --target llama-bench llama-cli llama-server -j 16
CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT
# config A* (the winner), depth-matched
GGML_CUDA_NO_PINNED=1 llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
export GGML_CUDA_NO_PINNED=1
# A* — iGPU only (the single-device baseline), depth-matched
llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
-ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1
# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration)
llama-bench -m <gguf> -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \
-ngl 999 -n 32 -d 65536 -r 1
# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan
llama-bench -m <gguf> -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \
-ngl 999 -n 32 -d 65536 -r 1
# proof of placement — sample the 3090 while the run is loaded
nvidia-smi --query-gpu=memory.used --format=csv,noheader # expect 12–15 GiB
```
Operational notes for the next run:
@@ -162,5 +299,6 @@ Operational notes for the next run:
of a restart trigger, or move the symlink aside.
- Restarting frank-large needs the quadlet source back in
`~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
the unit from that file only. After the restore the ledger read `400384
positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 15.2 GiB left.
the unit from that file only. After the restore (and a host reboot, see the
landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB
budget — no clamp — with 15.2 GiB left, and both endpoints answered `/health`.