---
name: ik-llama-hybrid-moe-performance
description: Diagnose and tune ik_llama.cpp hybrid CPU/GPU MoE decode/prefill on the dankclaw 3090+Strix Halo frank-large deployment; includes the barrier-split root-cause method and the systemd-unit test-interference trap
---

# ik_llama.cpp hybrid MoE performance campaign (dankclaw frank-large)

Use when tuning or debugging decode/prefill speed of the ik_llama.cpp hybrid deployment (RTX 3090 + Strix Halo LPDDR, CPU-resident MoE experts) on dankclaw, or when a config experiment gives surprising numbers.

## Measured envelope (2026-09-16)

- Decode 3.8-4.3 tok/s, flat across thread counts (T1 ≈ T16). Prefill ~192 tok/s at 1.8k-32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts (batch amortizes expert reads; compute-bound, NOT memory-bound).
- Decode root cause: ~100 CPU↔GPU backend splits/token; GPU at 5% SM during decode (launch/sync-bound over the PCIe x4 dock); T16 profile shows ~90% of cycles in `pause` (ggml_barrier spin) because threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does NOT matter — all variants land 3.8-4.6.
- CUDA-graph capture aborts: qwen4exp's hc broadcasts use per-layer `ggml_repeat_4d` whose CUDA path does a capture-illegal op. Until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead. Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4x slower than ik iqk) — do not switch for speed.
- MTP/speculative impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).

## Tuning levers, ranked

1. `-ncmoe 42` puts 6 layers' experts on the 3090: decode 4.54, prefill (small prompt) 32 — but VRAM 22/24.6 GiB at 262k ctx; needs an OOM-margin decision from the operator.
2. `--prefetch-experts` is a no-op when the page cache is resident.
3. `-thp` huge-mmap usually fails (`Cannot allocate memory`) and falls back silently.

## Traps

- The frank-large root systemd unit has Restart=always: `pkill`ing llama-server respawns it in ~4 s. For manual experiments: `sudo systemctl stop frank-large.service` first (root unit — NOT `systemctl --user`, that is stale halogen-era docs); verify `pgrep -xc llama-server` == 1 and 'HTTP server listening' in the current log before trusting any measurement.
- Any `-ot`/`-ncmoe` override disables mmap for the whole model load in ik (`create_tensors` returns false).
- `-ncmoe N` without `GGML_CUDA_NO_PINNED_WEIGHTS=1` page-locks ~108 GiB (co-tenant killer — the 2026-09-15 incident).
- `--defer-experts` defers only PLE; under `-ngl 999` the experts still target CUDA0 → cudaMalloc OOM (~79.7 GiB on a 24 GiB card).
- Never run `-j16` builds concurrently with decode measurements — both results are garbage.