serverAppConfigs/docs/LLM/ikllama-ornith-dankserver.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	ik_llama.cpp on Dankserver (Ornith Q8_0)	ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, q6_0 KV cache with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 160k single slot, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.	ikllama llamacpp cuda dankserver moe ornith kv-cache hadamard	2026-08-26

ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

Active 2026-08-26. frank-default-small switched from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (commit b166e269) on the RTX 3090. This page documents the build, the mandatory FA build flags, and the MoE / CPU-offload runtime arguments.

The model is dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision encoder). Served as a single 160000-context slot (PARALLEL=1). Speculative decoding is disabled. See dealign-ornith-crack for the model runbook.

Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA optimizations, IQK/AVX-512 kernels, --fit auto-offload, and fast q8_KV cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is exactly the supported hybrid the fork targets. Only CPU and CUDA backends are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is removed (nvidia-cuda-toolkit is in rc state; no /usr/local/cuda*, no /usr/bin/nvcc). The CUDA 13.2 toolkit is installed via pip inside the vllm venv at /home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/. A staged, self-contained toolchain at /home/dank/cuda13-stage adds the unversioned .so aliases and static libs CMake needs.

# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8

⚠️ MANDATORY: always include the FA all-quants flags

Rule. Any ik_llama.cpp build that serves a quantized KV cache (q8_KV / q8_0 / Hadamard / any non-f16 KV) MUST be compiled with BOTH:

• -DGGML_CUDA_FA_ALL_QUANTS=ON — compiles every K/V cache quant combo into the CUDA FlashAttention kernels. Without it, CUDA FA only ships a subset and errors at runtime telling you to recompile when a combo (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on the GPU, so this is the flag that matters here.
• -DGGML_IQK_FA_ALL_QUANTS=ON — compiles all quant types (Q4_0/Q4_1/ IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the CPU/IQK FA kernels. Relevant because the MoE experts run on the CPU; the KV cache that backs CPU-side attention uses the IQK FA path.

These are the old "asymmetrical quants" compile-time gates. On a CUDA host, omitting GGML_CUDA_FA_ALL_QUANTS is the #1 way to get a "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

GGML_NATIVE=ON lets GCC target the host (the Ryzen 7 7700X is Zen4, so -march=native enables the AVX512 / VNNI IQK kernels automatically — verifiable with system_info: AVX512_VNNI = 1 in the boot log).

CUDA stage note

/home/dank/cuda13-stage is the pip cu13 toolkit plus the symlinks CMake's (CUDAToolkit) finder needs (libcudart.so, libcublas.so, libcublasLt.so and static libcudadevrt.a/libcudart_static.a). It also works around the CCCL CUDA version skew by defining -DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in CMAKE_CUDA_FLAGS (the pip package ships nvcc 13.2.78 but reports CUDART_VERSION 13000, which trips a strict header lint). Do not re-run configure without that flag or you get a hard compile error.

Runtime configuration

Files: scripts/serve-frank-small.sh, scripts/frank-default-small.env, scripts/frank-default-small.service in the repo. The service just calls the serve script; the env file holds all tunables.

KV cache: q8_0 → q6_0 + Hadamard

• CACHE_TYPE_K=q6_0, CACHE_TYPE_V=q6_0 — 6-bit k-quant KV cache (passes --cache-type-k q6_0 --cache-type-v q6_0).
• K_CACHE_HADAMARD=1, V_CACHE_HADAMARD=1 — --k-cache-hadamard and --v-cache-hadamard. The Hadamard transform preserves KV quality at 6-bit. Implemented for both CUDA (ggml-cuda/hadamard.cu) and CPU (IQK) paths. Flash-attention is on by default.

KV cache type history on this build (b166e269):

• iq5_k is not a KV-cache type — the cache whitelist is f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV; iq5_* only exist as weight quants. --cache-type-k iq5_k throws Invalid cache type.
• q8_KV crashes with a ggml view size assert (ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)) on this QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
• q6_0 + Hadamard works and is what this deployment uses.

At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving ~2.2 GiB headroom on the 24 GiB card after the GPU-side experts.

Split mode: none (single GPU)

• SPLIT_MODE=none → --split-mode none (the default). Dankserver has a single RTX 3090 yet ik_llama rejects --split-mode graph on a box with a single device:

  llama error loading model: It is not possible to use split mode 'graph' with less than 2 devices

  So graph is not a harmless no-op on one GPU — it is a hard load error (this bit us on the first deploy and loop the service). It also still works with the --override-tensor expert pins.

Lesson. ik_llama --split-mode graph requires ≥2 devices. On a single-GPU host you must use none (or layer, which also needs >1). Do not assume --split-mode graph degrades gracefully on one GPU.

Expert offload (MoE: CPU with a partial GPU slice)

The expert-offload pattern from upstream llama.cpp carries over: the --override-tensor regex pins the sparse expert (ffn_*_exps) tensors to the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. On 2026-08-26 we brought expert blocks 30, 32, 33, 34 back to CUDA (~2.5 GiB), then 28 and 29 (~1.26 GiB) for the fused-MoE run, to use the GPU headroom for lower latency; the rest of that CPU range stays on CPU.

• OVERRIDE_EXPERT — the layer-bounded regex listing only the blocks whose experts stay on CPU: (0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| 24|25|26). Blocks 28,29,30,32,33,34 (and the always-on full-attention + 35-40 set) are left off the list so their experts load to CUDA.
• OVERRIDE_BUFT=CPU.
• N_GPU_LAYERS=999 — ik_llama -ngl / --gpu-layers only accepts an integer (the literal all is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the listed exps to CPU.

Fused MoE (fused_moe, fused_up_gate, fused_mmad) is on by default in ik_llama — there is no positive flag; only -no-fmoe disables it. only_active_experts is also on by default. Confirmed in the boot log for the running service.

To move more experts back to the GPU later, drop more block numbers from OVERRIDE_EXPERT (each block is ~630 MiB; verify nvidia-smi headroom first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the free headroom was ~1.6 GiB, which is tighter than ideal — add more only with a headroom check.

Qwen-Sharp chat template via --jinja

--chat-template-file alone only stores the template text; ik_llama does not render a custom template as real Jinja unless --jinja is also given (inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja fallback>). Without --jinja it warns it only supports "commonly used templates" and silently uses built-in handlers, dropping the Qwen-Sharp template's thinking / image / tool formatting. So the serve script passes both --chat-template-file and --jinja (gated on USE_JINJA, set to 1).

Speculative decoding: disabled (3 parallel slots)

SPEC_TYPE=none (passed as --spec-type none). Speculative decoding is intentionally disabled because ik_llama limits any speculative stage to a a single server slot: with --parallel 3 a configured spec stage hard-exits at load with

ERR [load_model] Speculative decoding is currently limited to a single server slot (-np 1). / n_parallel=3

This service keeps 3 parallel slots (one per user family via PARALLEL=3), so speculative decoding cannot be used here. MTP stays disabled. The model is a low-parameter MoE (3B active of 35B) and does not need the speculation it would cost a slot for.

Lesson. Do not combine speculative decoding with --parallel > 1 in ik_llama. Either drop to -np 1 (one slot) and keep a spec stage, or keep multiple slots and set SPEC_TYPE=none. For a multi-user service the extra slots win.

Flash-attention + sampling

Flash-attention is on by default in ik_llama (flash_attn=true), so no -fa/--flash-attn flag is needed. Sampling, thread, batch, context, and checkpoint tunables (TEMPERATURE, TOP_K, TOP_P, MIN_P, thread, BATCH_SIZE, UBATCH_SIZE, CTX_SIZE, PARALLEL, CTX_CHECKPOINTS, NO_MMAP) are unchanged and pass straight through.

MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

flag	meaning	our use
--split-mode graph	split tensors+graph across devices; strong for MoE; requires 2+ devices	single 3090 → use none
-ooae / --offload-only-active-experts	default ON: copy only active experts RAM→VRAM	keep ON (huge for A3B: few experts active)
-grt q8_0	reduce type for inter-GPU data exchange	not needed (single GPU)
--fit	auto-offload to fit VRAM	conflicts with --override-tensor; do not use together (runtime error)
-n-cpu-moe N / --cpu-moe	keep MoE weights of first N / all layers on CPU	also conflicts with --fit; our --override-tensor is finer-grained
--fit-margin	VRAM safety margin for --fit	only relevant with --fit

The current deployment uses --override-tensor (not --fit or --n-cpu-moe) for the expert offload, so the --fit-conflict rules do not apply.

Rollback

The previous runtime upstream llama.cpp b10121 and its binary (/home/dank/llama.cpp/build/bin/llama-server) are left untouched. To rollback to the old stack, change LLAMA_SERVER= in scripts/frank-default-small.env back to that path and CACHE_TYPE_K/V back to f16 with Hadamard off, and re-enable the ngram spec / --spec-type flags. The model file is not changed either way.

See Also

• dealign-ornith-crack — the Ornith Q6_K model run
• Dankserver System Specs — the host
• deployment — service-scoped git deploy

Citations

• ikawrakow/ik_llama.cpp README — MoE offload guidance and the --split-mode graph + CPU-offload warning
• ik_llama.cpp docs/build.md — build flags incl. GGML_CUDA_FA_ALL_QUANTS and GGML_IQK_FA_ALL_QUANTS
• ik_llama.cpp docs/parameters.md — q8_KV (PR 208), K/V Hadamard options, -cuda graphs=0 workaround, and --split-mode graph MoE note (PR 1080)