command-center/agent/managed-skills/ik-llama-hybrid-moe-performance/SKILL.md

name, description

name	description
ik-llama-hybrid-moe-performance	Diagnose and tune ik_llama.cpp hybrid CPU/GPU MoE decode/prefill on the dankclaw 3090+Strix Halo frank-large deployment; includes the barrier-split root-cause method and the systemd-unit test-interference trap

ik_llama.cpp hybrid MoE performance campaign (dankclaw frank-large)

Use when tuning or debugging decode/prefill speed of the ik_llama.cpp hybrid deployment (RTX 3090 + Strix Halo LPDDR, CPU-resident MoE experts) on dankclaw, or when a config experiment gives surprising numbers.

Measured envelope (2026-09-16)

• Decode 3.8-4.3 tok/s, flat across thread counts (T1 ≈ T16). Prefill ~192 tok/s at 1.8k-32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts (batch amortizes expert reads; compute-bound, NOT memory-bound).
• Decode root cause: ~100 CPU↔GPU backend splits/token; GPU at 5% SM during decode (launch/sync-bound over the PCIe x4 dock); T16 profile shows ~90% of cycles in pause (ggml_barrier spin) because threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does NOT matter — all variants land 3.8-4.6.
• CUDA-graph capture aborts: qwen4exp's hc broadcasts use per-layer ggml_repeat_4d whose CUDA path does a capture-illegal op. Until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead. Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4x slower than ik iqk) — do not switch for speed.
• MTP/speculative impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).

Tuning levers, ranked

1. -ncmoe 42 puts 6 layers' experts on the 3090: decode 4.54, prefill (small prompt) 32 — but VRAM 22/24.6 GiB at 262k ctx; needs an OOM-margin decision from the operator.
2. --prefetch-experts is a no-op when the page cache is resident.
3. -thp huge-mmap usually fails (Cannot allocate memory) and falls back silently.

Traps

• The frank-large root systemd unit has Restart=always: pkilling llama-server respawns it in ~4 s. For manual experiments: sudo systemctl stop frank-large.service first (root unit — NOT systemctl --user, that is stale halogen-era docs); verify pgrep -xc llama-server == 1 and 'HTTP server listening' in the current log before trusting any measurement.
• Any -ot/-ncmoe override disables mmap for the whole model load in ik (create_tensors returns false).
• -ncmoe N without GGML_CUDA_NO_PINNED_WEIGHTS=1 page-locks ~108 GiB (co-tenant killer — the 2026-09-15 incident).
• --defer-experts defers only PLE; under -ngl 999 the experts still target CUDA0 → cudaMalloc OOM (~79.7 GiB on a 24 GiB card).
• Never run -j16 builds concurrently with decode measurements — both results are garbage.