docs: what the x4 pair can and cannot do — measured, not argued
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)" -- and both per-layer cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU is the only fallback side that works, which is exactly why -ot exps=CPU runs. What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s -- but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7 spare, so it is not deployable beside the co-tenants. Throughput parity, memory bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than preference. New page LLM/x4-pair-experiment.md carries the tensor inventory, the results table, the two scheduler limits, reproduction commands, and the traps found on the way. Both index pages also stop calling halogen retired -- the ik swap's stale claims were still sitting in them.
This commit is contained in:
@@ -144,14 +144,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since
|
||||
2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-
|
||||
Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd
|
||||
unit, port 7770) — measured performance envelope and traps
|
||||
- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and
|
||||
rollback target: peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
|
||||
container on Dankclaw, port 7770 — flags, memory model, client contract
|
||||
- [[halogen-flash-server]] — the ACTIVE frank-large runbook again since
|
||||
2026-09-16 (rootless Quadlet, port 7770): peonist-ai/halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b `.hgn` + quality
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract
|
||||
- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
|
||||
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
|
||||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
|
||||
Strix Halo hybrid — its measured envelope is why the fleet went back
|
||||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
|
||||
(the NPU endpoint), and how to revert
|
||||
@@ -193,6 +194,11 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
|
||||
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
|
||||
- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
|
||||
do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
|
||||
decides a split, the scheduler limits that kill a cross-backend one, and the
|
||||
iGPU-only result that matches halogen's decode rate while leaving the 3090
|
||||
idle — for 116 of the host's 121 GiB
|
||||
|
||||
### Memory and Hardware
|
||||
|
||||
|
||||
Reference in New Issue
Block a user