serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps	llm frank-large ik_llama dankclaw rtx3090 strix-halo performance	2026-09-16T03:30:00-07:00

frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M

RETIRED 2026-09-16, after one day. Replaced by halogen-flash-server, which was the runtime before it as well — the fleet went ik → halogen → back. Its unit, env, serve script and the Qwen-Sharp template are archived in the repo under scripts/.AGENTTRASH/*.ik-llama.cpp and *.ik-llama-qwen-sharp, and the payload, build and chat template stay on disk. The measurements below are not a stale plan: they are the complete account of what this model costs on this hardware, and they are why the fleet went back. It served mradermacher/Qwen3.8-Flash-Next-Uncensored (111 GiB GGUF + mmproj-f16) at 192.168.0.143:7770 while it ran.

Shape

• Runtime: ik_llama.cpp master d85a1d7, built at /home/dank/GitPackages/ik_llama.cpp/build with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF.
• Unit: root systemd unit /etc/systemd/system/frank-large.service symlinked to scripts/frank-large.service in the repo; enabled at boot; Restart=always. Serve script scripts/serve-frank-large.sh + env scripts/frank-large.env.
• Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the ~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
• Client contract: greedy default (--temp 0.0), reasoning on (deepseek format, unlimited budget), vision via mmproj, slot-state persistence under scripts/slot-state, 262144 ctx, 1 slot.

Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)

config	decode tok/s	prefill tok/s	verdict
shipped (T16, -ot exps=CPU, NO_PINNED_WEIGHTS)	3.8-4.0	~193 at 1.8k / 191 at 32k	shipped
T8 / T4 / T2 / T1	4.18 / 4.20 / 4.26 / 3.97	small-prompt only	decode flat across threads
-ncmoe 42 (6 layers' experts on GPU)	4.54	32 (small prompt)	VRAM 22.0/24.6 GiB — fragile
-ncmoe 48 -thp	4.60	18	110 GiB anon host buffer — co-tenant killer, never ship
--prefetch-experts	4.03	—	no effect (page cache already resident)
--defer-experts --defer-ple -thp	crashes	—	experts under -ngl 999 still target CUDA0 → cudaMalloc OOM
mainline llama.cpp 37b53fd (--n-cpu-moe 48)	1.01	34	ruled out — CPU MoE kernels 4x slower than ik's iqk

• Large-prompt prefill ≈ 192 tok/s is the CPU AVX512-VNNI compute ceiling for CPU-resident experts (expert weight reads amortize across the batch; compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
• Decode is gated by a serial per-token critical path: ~100 CPU↔GPU backend splits per token (graph splits = 100 in the log); the 3090 sits at 5% SM utilization during decode; thread count does not matter (T1 ≈ T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.

Root cause chain (perf evidence)

1. perf at T16: ~53% cycles in ggml_graph_compute_thread + ~37% in ggml_compute_forward_mul_mat_id, of which ~98% is the pause instruction (thread-pool barrier spin). Actual vec_dot math: ~4%.
2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on a serial critical path, not parallel overhead.
3. GPU 95% idle during decode ⇒ the critical path is the GPU-section launch/sync overhead per split (x4 dock link, eager kernel launches).
4. ik's CUDA-graph capture would collapse those launches but aborts on qwen4exp: the per-layer hyper-connection broadcasts (ggml_repeat_4d, 4 sites in src/graphs/build_qwen4exp.cpp) resolve to a REPEAT node whose CUDA path does a capture-illegal operation (CUDA error: operation not permitted when stream is capturing → abort).
5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all land at 3.8-4.6 tok/s.

Why the 30 tok/s halogen bar is out of reach (honest record)

Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR (HALOGEN_FLASH_PIN_TRUNK=1) and served from one engine. This deployment must split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public runtime pays the split penalty above. Paths to >30 tok/s, in order of realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops — mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops"); re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.

Upstream issues tracking the slow TG path

ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433 (cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative decoding is unavailable for this GGUF — mradermacher stripped the NextN head (verified: no blk.48.*/nextn tensors in the file).

Traps (learned the hard way 2026-09-15/16)

• Pinned host buffers strand the host: a default-policy start page-locks ~108 GiB (CUDA_Host buffer), evicting co-tenants into swap and starving sshd until a power cycle. GGML_CUDA_NO_PINNED_WEIGHTS=1 + the serve-script guard exist for this. -ncmoe N with default NO_PINNED settings produces the same 108 GiB pinned buffer — never run it without the guard.
• Any -ot/-ncmoe override disables mmap for the whole model load (create_tensors returns false → ml.use_mmap = false): the "mmap-backed CPU buffer" comment in the serve script is wrong on the mmap word; the buffer is anonymous. All placement variants perform alike anyway.
• --defer-experts does not keep experts off the GPU under -ngl 999 — only PLE defers (32.78 GiB). With ngl 999 the load dies with cudaMalloc failed: out of memory at ~79.7 GiB on device 0.
• The systemd unit self-heals through manual tests: pkill the server and the unit respawns it ~4 s later (Restart=always) — two instances, port races, phantom measurements. For manual experiments: sudo systemctl stop frank-large.service first (it is a root unit; plain systemctl --user lines in older docs are stale), and restore with sudo systemctl start frank-large.service.
• Verify "HTTP server listening" in the CURRENT log and pgrep -xc llama-server == 1 before trusting any measurement.
• FA must stay ON — FA-off crashes the loader on this arch.
• Do not run builds concurrently with decode measurements (-j16 starves the service and corrupts both results).

Rollback

Stop and disable the root unit, trash it, restore the halogen quadlet from ~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet to ~/.config/containers/systemd/, systemctl --user daemon-reload, start; the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.

Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)

Why this deployment could not be tuned into a production endpoint: decode 3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at 1.8k–32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts. The decode root cause is structural: ~100 CPU↔GPU backend splits per token with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4 dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in pause (ggml_barrier spin): threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does not matter — all variants landed 3.8–4.6.

• CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use per-layer ggml_repeat_4d, whose CUDA path is capture-illegal; until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead.
• Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4× slower than ik's iqk) — do not switch engines for speed.
• MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).
• Tuning lever ranked first: -ncmoe 42 (6 layers' experts on the 3090) → decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx, an operator OOM-margin decision, and it needs GGML_CUDA_NO_PINNED_WEIGHTS=1.