--- type: reference description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance] timestamp: 2026-09-16T03:30:00-07:00 --- # frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M **RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]], which was the runtime before it as well — the fleet went ik → halogen → back. Its unit, env, serve script and the Qwen-Sharp template are archived in the repo under `scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`, and the payload, build and chat template stay on disk. The measurements below are not a stale plan: they are the complete account of what this model costs on this hardware, and they are why the fleet went back. It served `mradermacher/Qwen3.8-Flash-Next-Uncensored` (111 GiB GGUF + mmproj-f16) at `192.168.0.143:7770` while it ran. ## Shape - Runtime: ik_llama.cpp master `d85a1d7`, built at `/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`. - Unit: **root systemd unit** `/etc/systemd/system/frank-large.service` symlinked to `scripts/frank-large.service` in the repo; enabled at boot; `Restart=always`. Serve script `scripts/serve-frank-large.sh` + env `scripts/frank-large.env`. - Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the ~33 GiB PLE n-gram table stay in LPDDR via CPU buffers. - Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek format, unlimited budget), vision via mmproj, slot-state persistence under `scripts/slot-state`, 262144 ctx, 1 slot. ## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified) | config | decode tok/s | prefill tok/s | verdict | |---|---|---|---| | shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped | | T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads | | `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile | | `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** | | `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) | | `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM | | mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk | - Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling** for CPU-resident experts (expert weight reads amortize across the batch; compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore unreachable with experts on CPU — the physics caps hybrid MoE prefill here. - Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU backend splits per token (`graph splits = 100` in the log); the 3090 sits at **5% SM utilization** during decode; thread count does not matter (T1 ≈ T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound. ## Root cause chain (perf evidence) 1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in `ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause` instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%. 2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on a serial critical path, not parallel overhead. 3. GPU 95% idle during decode ⇒ the critical path is the GPU-section launch/sync overhead per split (x4 dock link, eager kernel launches). 4. ik's CUDA-graph capture would collapse those launches but **aborts** on qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`, 4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose CUDA path does a capture-illegal operation (`CUDA error: operation not permitted when stream is capturing` → abort). 5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all land at 3.8-4.6 tok/s. ## Why the 30 tok/s halogen bar is out of reach (honest record) Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR (`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public runtime pays the split penalty above. Paths to >30 tok/s, in order of realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops — mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops"); re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model. ## Upstream issues tracking the slow TG path ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433 (cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative decoding is unavailable for this GGUF — mradermacher stripped the NextN head (verified: no `blk.48.*`/nextn tensors in the file). ## Traps (learned the hard way 2026-09-15/16) - **Pinned host buffers strand the host**: a default-policy start page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the serve-script guard exist for this. `-ncmoe N` with default NO_PINNED settings produces the same 108 GiB pinned buffer — never run it without the guard. - **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load (`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed CPU buffer" comment in the serve script is wrong on the mmap word; the buffer is anonymous. All placement variants perform alike anyway. - **`--defer-experts` does not keep experts off the GPU** under `-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with `cudaMalloc failed: out of memory` at ~79.7 GiB on device 0. - **The systemd unit self-heals through manual tests**: `pkill` the server and the unit respawns it ~4 s later (Restart=always) — two instances, port races, phantom measurements. For manual experiments: `sudo systemctl stop frank-large.service` first (it is a root unit; plain `systemctl --user` lines in older docs are stale), and restore with `sudo systemctl start frank-large.service`. - Verify "HTTP server listening" in the CURRENT log and `pgrep -xc llama-server` == 1 before trusting any measurement. - FA must stay ON — FA-off crashes the loader on this arch. - Do not run builds concurrently with decode measurements (`-j16` starves the service and corrupts both results). ## Rollback Stop and disable the root unit, trash it, restore the halogen quadlet from `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start; the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.