serverAppConfigs/docs/LLM/ikllama-ornith-dankserver.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	ik_llama.cpp on Dankserver (Ornith Q6_K)	ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.	ikllama llamacpp cuda dankserver moe ornith kv-cache hadamard	2026-08-26

ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

Active 2026-08-26. frank-default-small switched from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (commit b166e269) on the RTX 3090. This page documents the build, the mandatory FA build flags, and the MoE / CPU-offload runtime arguments.

The model is unchanged: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF. Speculative decoding is disabled (ik_llama limits spec to a single slot; this service keeps 3 parallel slots). The low-parameter MoE does not need it. See dealign-ornith-crack for the model runbook.

Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA optimizations, IQK/AVX-512 kernels, --fit auto-offload, and fast q8_KV cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is exactly the supported hybrid the fork targets. Only CPU and CUDA backends are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is removed (nvidia-cuda-toolkit is in rc state; no /usr/local/cuda*, no /usr/bin/nvcc). The CUDA 13.2 toolkit is installed via pip inside the vllm venv at /home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/. A staged, self-contained toolchain at /home/dank/cuda13-stage adds the unversioned .so aliases and static libs CMake needs.

# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8

⚠️ MANDATORY: always include the FA all-quants flags

Rule. Any ik_llama.cpp build that serves a quantized KV cache (q8_KV / q8_0 / Hadamard / any non-f16 KV) MUST be compiled with BOTH:

• -DGGML_CUDA_FA_ALL_QUANTS=ON — compiles every K/V cache quant combo into the CUDA FlashAttention kernels. Without it, CUDA FA only ships a subset and errors at runtime telling you to recompile when a combo (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on the GPU, so this is the flag that matters here.
• -DGGML_IQK_FA_ALL_QUANTS=ON — compiles all quant types (Q4_0/Q4_1/ IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the CPU/IQK FA kernels. Relevant because the MoE experts run on the CPU; the KV cache that backs CPU-side attention uses the IQK FA path.

These are the old "asymmetrical quants" compile-time gates. On a CUDA host, omitting GGML_CUDA_FA_ALL_QUANTS is the #1 way to get a "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

GGML_NATIVE=ON lets GCC target the host (the Ryzen 7 7700X is Zen4, so -march=native enables the AVX512 / VNNI IQK kernels automatically — verifiable with system_info: AVX512_VNNI = 1 in the boot log).

CUDA stage note

/home/dank/cuda13-stage is the pip cu13 toolkit plus the symlinks CMake's (CUDAToolkit) finder needs (libcudart.so, libcublas.so, libcublasLt.so and static libcudadevrt.a/libcudart_static.a). It also works around the CCCL CUDA version skew by defining -DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in CMAKE_CUDA_FLAGS (the pip package ships nvcc 13.2.78 but reports CUDART_VERSION 13000, which trips a strict header lint). Do not re-run configure without that flag or you get a hard compile error.

Runtime configuration

Files: scripts/serve-frank-small.sh, scripts/frank-default-small.env, scripts/frank-default-small.service in the repo. The service just calls the serve script; the env file holds all tunables.

KV cache: q8_0 + Hadamard

• CACHE_TYPE_K=q8_0, CACHE_TYPE_V=q8_0 — standard 8-bit KV cache type (passes --cache-type-k q8_0 --cache-type-v q8_0).
• K_CACHE_HADAMARD=1, V_CACHE_HADAMARD=1 — --k-cache-hadamard and --v-cache-hadamard. The Hadamard transform preserves KV quality at 8-bit (docs recommend it for quantized KV below Q6_0). Implemented for both the CUDA (ggml-cuda/hadamard.cu) and CPU (IQK) paths.
• Flash-attention is on by default (no -fa flag needed).

⚠️ Do not use q8_KV (fast type) here. On this commit (b166e269) the fast q8_KV cache type crashes at model init on this QWEN3MOE/MTP + multimodal arch with a ggml view size assertion (ggml.c:5427: GGML_ASSERT(view_src == NULL || ... data_size + view_offs) right after the output buffer is created. It loops the service. Plain q8_0 (and f16) work and were smoke-tested with correct output; this config uses q8_0. This was isolated empirically by toggling cache type while keeping the override/hadamard flags.

These keep the 163840-slot / 491520-pool context while roughly halving KV cache RAM vs f16.

Split mode: none (single GPU)

• SPLIT_MODE=none → --split-mode none (the default). Dankserver has a single RTX 3090 yet ik_llama rejects --split-mode graph on a box with a single device:

  llama error loading model: It is not possible to use split mode 'graph' with less than 2 devices

  So graph is not a harmless no-op on one GPU — it is a hard load error (this bit us on the first deploy and loop the service). It also still works with the --override-tensor expert pins.

Lesson. ik_llama --split-mode graph requires ≥2 devices. On a single-GPU host you must use none (or layer, which also needs >1). Do not assume --split-mode graph degrades gracefully on one GPU.

Expert offload (MoE: CPU with a partial GPU slice)

The expert-offload pattern from upstream llama.cpp carries over: the --override-tensor regex pins the sparse expert (ffn_*_exps) tensors to the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. On 2026-08-26 we brought expert blocks 30, 32, 33, 34 back to CUDA (~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of that CPU range stays on CPU.

• OVERRIDE_EXPERT — the layer-bounded regex listing only the blocks whose experts stay on CPU: (0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| 24|25|26|28|29). Blocks 30,32,33,34 (and the always-on full-attention + 35-40 set) are left off the list so their experts load to CUDA.
• OVERRIDE_BUFT=CPU.
• N_GPU_LAYERS=999 — ik_llama -ngl / --gpu-layers only accepts an integer (the literal all is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the listed exps to CPU.

To move more experts back to the GPU later, drop more block numbers from OVERRIDE_EXPERT (each block is ~630 MiB; verify nvidia-smi headroom first and keep ~2-3 GiB margin for KV/compute).

Qwen-Sharp chat template via --jinja

--chat-template-file alone only stores the template text; ik_llama does not render a custom template as real Jinja unless --jinja is also given (inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja fallback>). Without --jinja it warns it only supports "commonly used templates" and silently uses built-in handlers, dropping the Qwen-Sharp template's thinking / image / tool formatting. So the serve script passes both --chat-template-file and --jinja (gated on USE_JINJA, set to 1).

Speculative decoding: disabled (3 parallel slots)

SPEC_TYPE=none (passed as --spec-type none). Speculative decoding is intentionally disabled because ik_llama limits any speculative stage to a a single server slot: with --parallel 3 a configured spec stage hard-exits at load with

ERR [load_model] Speculative decoding is currently limited to a single server slot (-np 1). / n_parallel=3

This service keeps 3 parallel slots (one per user family via PARALLEL=3), so speculative decoding cannot be used here. MTP stays disabled. The model is a low-parameter MoE (3B active of 35B) and does not need the speculation it would cost a slot for.

Lesson. Do not combine speculative decoding with --parallel > 1 in ik_llama. Either drop to -np 1 (one slot) and keep a spec stage, or keep multiple slots and set SPEC_TYPE=none. For a multi-user service the extra slots win.

Flash-attention + sampling

Flash-attention is on by default in ik_llama (flash_attn=true), so no -fa/--flash-attn flag is needed. Sampling, thread, batch, context, and checkpoint tunables (TEMPERATURE, TOP_K, TOP_P, MIN_P, thread, BATCH_SIZE, UBATCH_SIZE, CTX_SIZE, PARALLEL, CTX_CHECKPOINTS, NO_MMAP) are unchanged and pass straight through.

MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

flag	meaning	our use
--split-mode graph	split tensors+graph across devices; strong for MoE; requires 2+ devices	single 3090 → use none
-ooae / --offload-only-active-experts	default ON: copy only active experts RAM→VRAM	keep ON (huge for A3B: few experts active)
-grt q8_0	reduce type for inter-GPU data exchange	not needed (single GPU)
--fit	auto-offload to fit VRAM	conflicts with --override-tensor; do not use together (runtime error)
-n-cpu-moe N / --cpu-moe	keep MoE weights of first N / all layers on CPU	also conflicts with --fit; our --override-tensor is finer-grained
--fit-margin	VRAM safety margin for --fit	only relevant with --fit

The current deployment uses --override-tensor (not --fit or --n-cpu-moe) for the expert offload, so the --fit-conflict rules do not apply.

Rollback

The previous runtime upstream llama.cpp b10121 and its binary (/home/dank/llama.cpp/build/bin/llama-server) are left untouched. To rollback to the old stack, change LLAMA_SERVER= in scripts/frank-default-small.env back to that path and CACHE_TYPE_K/V back to f16 with Hadamard off, and re-enable the ngram spec / --spec-type flags. The model file is not changed either way.

See Also

• dealign-ornith-crack — the Ornith Q6_K model run
• Dankserver System Specs — the host
• deployment — service-scoped git deploy

Citations

• ikawrakow/ik_llama.cpp README — MoE offload guidance and the --split-mode graph + CPU-offload warning
• ik_llama.cpp docs/build.md — build flags incl. GGML_CUDA_FA_ALL_QUANTS and GGML_IQK_FA_ALL_QUANTS
• ik_llama.cpp docs/parameters.md — q8_KV (PR 208), K/V Hadamard options, -cuda graphs=0 workaround, and --split-mode graph MoE note (PR 1080)