docs: what the x4 pair can and cannot do — measured, not argued

The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.

What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.

New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.

serverAppConfigs-docs/LLM/index.md

@@ -92,6 +92,12 @@ frank-small, port 7771]
   ISTA-DASLab/Qwen3.8-27B-GSQ-RCO IQ3_S on the RTX 3090, with the model's own
   Multi-Token Prediction head for speculative decoding and the BF16 projector
   for the fleet's image path
+- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
+  cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
+  Gen4 x4). The tensor inventory that decides any split, the two scheduler
+  limits that kill a cross-backend one, and the finding that the iGPU alone
+  matches halogen's deployed decode rate while leaving the 3090 idle — at a
+  memory cost the co-tenants cannot pay
 - [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
   ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
   mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090

serverAppConfigs-docs/LLM/x4-pair-experiment.md

@@ -0,0 +1,166 @@
+---
+type: reference
+title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
+description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the two scheduler limits that kill a cross-backend one, and the finding that the iGPU alone (Vulkan) matches halogen's deployed decode rate while leaving the 3090 idle.
+tags:
+  - llm
+  - experiment
+  - dankclaw
+  - llama.cpp
+  - vulkan
+  - strix-halo
+  - rtx3090
+  - flash-next
+  - pipeline-parallelism
+timestamp: 2026-09-16
+---
+
+# The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
+
+The question this answers: **is there a 3090 + Strix Halo combination worth
+building**, given that the only link between them is the 3090's PCIe Gen4 x4
+(measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable;
+idle-downshifts to Gen1 x4).
+
+Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend
+build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM
+services stopped per the benchmarking policy.
+
+## The tensor inventory decides everything
+
+From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
+1224 tensors, 176.94 B params, **A3B active**):
+
+| group | size |
+|---|---|
+| `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** |
+| `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** |
+| everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** |
+
+- 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`,
+  every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
+- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
+  position, a 200k-token conversation reads **5.5 GiB of KV per token**.
+
+That is why the interesting split is *not* "put the experts somewhere else" —
+experts are re-read once per token but they are only ~1.7 GB of traffic — it is
+"get KV onto the 936 GB/s pool", because KV is the O(context) term.
+
+## Results
+
+`llama-bench`, same model, 2 tests per config (`-p 512,16384 -n 64`), plus a
+depth-matched pair (`-n 32 -d 65536`). Note llama-bench's `tg` instances run with
+`n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required to
+put KV in the picture.
+
+| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg32 @64k depth |
+|---|---|---|---|---|---|
+| **A** | iGPU trunk (Vulkan1) + experts on CPU | 128.7 | 122.1 | 12.10 | — |
+| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k = 231.8) |
+| **B** | trunk + KV on CUDA0, experts on Vulkan1 | — | — | **abort** | — |
+| **B2** | attention layers → CUDA0, rest → Vulkan1 (per-layer `-ot`) | — | — | load failure | — |
+| **B3** | 25 % of layers → CUDA0, rest → Vulkan1 (`-sm layer -ts 0.25,0.75`) | — | — | load failure | — |
+| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — |
+
+Reference points on the same silicon: halogen (deployed) ≈ **25 tok/s** at real
+contexts; the retired ik hybrid 3.8–4.6 tok/s; mainline with the trunk on the
+3090 and CPU experts 1.01 tok/s (2026-09-15); the CPU expert path's prefill
+ceiling 192 tok/s.
+
+## Finding 1 — the cross-backend split is not expressible
+
+Per-tensor placement between two **device** backends aborts in the scheduler:
+
+```
+ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
+in a buffer (Vulkan1) that cannot run the operation (NONE)
+```
+
+A weight tensor living in one device's buffer cannot be consumed by an op the
+scheduler placed on another device. The CPU is the exception — `-ot exps=CPU`
+works — because the CPU backend is the universal fallback and host buffers can
+be copied. Per-**layer** cross-backend splits (`-sm layer -ts`, and layer-scoped
+`-ot`) are no better: both failed at model load with no further diagnostic.
+
+So on this build, **CUDA + Vulkan cannot serve one model**: the 3090 can be the
+trunk *or* the expert host, but not the complement of an iGPU-resident peer.
+That is the same wall the R9700 pair hit from the other side (collectives), and
+it is why every working multi-device llama.cpp deployment in this fleet is
+either single-vendor (frank-medium: two AMD cards) or device+CPU.
+
+## Finding 2 — the iGPU alone is competitive, and that was not expected
+
+Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in
+mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep
+context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25
+tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090
+sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %**
+busy, so the work is genuinely there and not silently CPU-side.
+
+The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s
+from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the
+LPDDR's shared 256 GB/s.
+
+**Why this is still not deployable here:** the run committed **116 of the host's
+121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving
+5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare,
+because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory
+bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape
+exists.
+
+## Finding 3 — the fast pool is too small to help
+
+B′ (trunk + KV on the 3090, experts on CPU) manages only 20.5 tok/s — *worse*
+than A\*, which does not use the 3090 at all. And the expressible version of the
+KV-on-VRAM design cannot reach its own premise: moving an attention layer to the
+3090 costs **1.62 GiB** (its MoE experts travel with it, since only whole layers
+move), so 12 attention layers = 19.4 GiB, and adding 200k of KV (9.2 GiB at f16,
+4.6 at q8_0) exceeds the card's 24.1 GiB usable. Feasible only to ~64k context —
+precisely where the KV term is small enough not to matter.
+
+## Conclusion
+
+On this host the pair is not additive, and the measured best arrangement remains
+**two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen,
+[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB). A
+single process spanning both is blocked by the scheduler, and every CPU-side
+fallback is slower than what the iGPU does alone.
+
+The one durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151
+is a viable engine for this class of model** (25.8 tok/s at 64k depth), which the
+2026-09-15 mainline measurement of 1.01 tok/s did *not* show — that number was
+the CPU+3090 hybrid, not the iGPU. Anyone re-opening the "is there a better
+engine than halogen?" question should start here, with the memory budget as the
+first constraint rather than the last.
+
+## Reproducing
+
+```bash
+# dual-backend build (glslc + libvulkan are already installed on dankclaw)
+cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
+  -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
+cmake --build build-dual --target llama-bench llama-cli llama-server -j 16
+
+# device map on this host
+CUDA0   = RTX 3090            Vulkan0 = RTX 3090 (same card)
+Vulkan1 = Radeon Graphics (RADV STRIX_HALO)   <- the iGPU, uma=1, 114 GiB GTT
+
+# config A* (the winner), depth-matched
+GGML_CUDA_NO_PINNED=1 llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
+  -ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1
+```
+
+Operational notes for the next run:
+
+- Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used
+  `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
+  pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
+  host's co-tenants.
+- `systemctl mask` is **refused** for a relinked root unit: frank-small's
+  `/etc/systemd/system/frank-small.service` is a symlink into the repo, so
+  `mask` exits with "File ... already exists". Stop it and rely on the absence
+  of a restart trigger, or move the symlink aside.
+- Restarting frank-large needs the quadlet source back in
+  `~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
+  the unit from that file only. After the restore the ledger read `400384
+  positions, ~27.2 GiB` against a 33.5 GiB budget — no clamp — with 15.2 GiB left.

serverAppConfigs-docs/index.md

@@ -144,14 +144,15 @@ The LLM inference fleet on the two hosts. Full index:
   agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
   Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
   rollback record)
-- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since
-  2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-
-  Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd
-  unit, port 7770) — measured performance envelope and traps
-- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and
-  rollback target: peonist-ai/halogen-flash-server 0.5.8 serving
-  peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
-  container on Dankclaw, port 7770 — flags, memory model, client contract
+- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since
+  2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server
+  0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality
+  overlay) on the gfx1151 iGPU — two 200k conversations over one
+  400,384-position pool, deployed flags, memory model, client contract
+- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
+  and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
+  mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
+  Strix Halo hybrid — its measured envelope is why the fleet went back
 - [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
   `amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
   (the NPU endpoint), and how to revert
@@ -193,6 +194,11 @@ The LLM inference fleet on the two hosts. Full index:
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
 - [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
+- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
+  do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
+  decides a split, the scheduler limits that kill a cross-backend one, and the
+  iGPU-only result that matches halogen's decode rate while leaving the 3090
+  idle — for 116 of the host's 121 GiB
 
 ### Memory and Hardware
 

serverAppConfigs-docs/log.md

@@ -621,3 +621,41 @@ timestamp: 2026-09-14
 - Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service
   frank-small` — per-service health gates, then the e2e gate PASSED (31 passed,
   1 flaky-retried). Suite: 323 seams ok, 0 failures.
+
+## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do
+- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends
+  (`build-dual`; glslc and libvulkan were already installed), to test the design
+  the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the
+  iGPU, PLE in LPDDR, one hop per boundary.
+- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and
+  **`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU
+  being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures
+  the wrong device.
+- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored):
+  experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else
+  **3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only
+  they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read
+  per token at 200k context.
+- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan,
+  PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 →
+  231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU
+  experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design
+  **aborted in the scheduler** — `pre-allocated tensor
+  (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the
+  operation (NONE)` — while both per-layer cross-backend forms failed to load.
+  Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only
+  fallback side that works.
+- The comparison that matters: at fleet-realistic depth the iGPU-only run
+  matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116
+  of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page
+  cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not
+  deployable beside the fleet's ~28 co-tenants. Throughput parity, memory
+  bankruptcy.
+- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it
+  `GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root
+  unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg`
+  instances run with `n_prompt = 0`, so `-d` is required to put KV into the
+  measurement at all. Both services were stopped and restored (both healthy;
+  halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget
+  with 15.2 GiB left).
+- New page: [[x4-pair-experiment]].