serverAppConfigs/docs/LLM/ikllama-ornith-dankserver.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	ik_llama.cpp on Dankserver (Ornith Q6_K)	ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.	ikllama llamacpp cuda dankserver moe ornith kv-cache hadamard	2026-08-26

ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

Active 2026-08-26. frank-default-small switched from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (commit b166e269) on the RTX 3090. This page documents the build, the mandatory FA build flags, and the MoE / CPU-offload runtime arguments.

The model is unchanged: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF. Speculation stays ngram-only (MTP intentionally disabled). See dealign-ornith-crack for the model runbook.

Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA optimizations, IQK/AVX-512 kernels, --fit auto-offload, and fast q8_KV cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is exactly the supported hybrid the fork targets. Only CPU and CUDA backends are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is removed (nvidia-cuda-toolkit is in rc state; no /usr/local/cuda*, no /usr/bin/nvcc). The CUDA 13.2 toolkit is installed via pip inside the vllm venv at /home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/. A staged, self-contained toolchain at /home/dank/cuda13-stage adds the unversioned .so aliases and static libs CMake needs.

# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8

⚠️ MANDATORY: always include the FA all-quants flags

Rule. Any ik_llama.cpp build that serves a quantized KV cache (q8_KV / q8_0 / Hadamard / any non-f16 KV) MUST be compiled with BOTH:

• -DGGML_CUDA_FA_ALL_QUANTS=ON — compiles every K/V cache quant combo into the CUDA FlashAttention kernels. Without it, CUDA FA only ships a subset and errors at runtime telling you to recompile when a combo (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on the GPU, so this is the flag that matters here.
• -DGGML_IQK_FA_ALL_QUANTS=ON — compiles all quant types (Q4_0/Q4_1/ IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the CPU/IQK FA kernels. Relevant because the MoE experts run on the CPU; the KV cache that backs CPU-side attention uses the IQK FA path.

These are the old "asymmetrical quants" compile-time gates. On a CUDA host, omitting GGML_CUDA_FA_ALL_QUANTS is the #1 way to get a "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

GGML_NATIVE=ON lets GCC target the host (the Ryzen 7 7700X is Zen4, so -march=native enables the AVX512 / VNNI IQK kernels automatically — verifiable with system_info: AVX512_VNNI = 1 in the boot log).

CUDA stage note

/home/dank/cuda13-stage is the pip cu13 toolkit plus the symlinks CMake's (CUDAToolkit) finder needs (libcudart.so, libcublas.so, libcublasLt.so and static libcudadevrt.a/libcudart_static.a). It also works around the CCCL CUDA version skew by defining -DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in CMAKE_CUDA_FLAGS (the pip package ships nvcc 13.2.78 but reports CUDART_VERSION 13000, which trips a strict header lint). Do not re-run configure without that flag or you get a hard compile error.

Runtime configuration

Files: scripts/serve-frank-small.sh, scripts/frank-default-small.env, scripts/frank-default-small.service in the repo. The service just calls the serve script; the env file holds all tunables.

KV cache: q8_KV + Hadamard

• CACHE_TYPE_K=q8_KV, CACHE_TYPE_V=q8_KV — fast quant type (ik PR 208). Passes --cache-type-k q8_KV --cache-type-v q8_KV.
• K_CACHE_HADAMARD=1, V_CACHE_HADAMARD=1 — --k-cache-hadamard and --v-cache-hadamard. The Hadamard transform preserves KV quality at 8-bit (docs recommend it for quantized KV below Q6_0). Implemented for both the CUDA (ggml-cuda/hadamard.cu) and CPU (IQK) paths.

These keep the 163840-slot / 491520-pool context while roughly halving KV cache RAM vs f16.

Split mode: none (single GPU)

• SPLIT_MODE=none → --split-mode none (the default). Dankserver has a single RTX 3090 yet ik_llama rejects --split-mode graph on a box with a single device:

  llama error loading model: It is not possible to use split mode 'graph' with less than 2 devices

  So graph is not a harmless no-op on one GPU — it is a hard load error (this bit us on the first deploy and loop the service). It also still works with the --override-tensor expert pins.

Lesson. ik_llama --split-mode graph requires ≥2 devices. On a single-GPU host you must use none (or layer, which also needs >1). Do not assume --split-mode graph degrades gracefully on one GPU.

Expert offload (MoE on CPU)

The expert-offload pattern is unchanged from upstream llama.cpp: the --override-tensor regex pins the sparse expert (ffn_*_exps) tensors to the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.

• OVERRIDE_EXPERT — the layer-bounded regex (same as before the switch).
• OVERRIDE_BUFT=CPU.
• N_GPU_LAYERS=999 — ik_llama -ngl / --gpu-layers only accepts an integer (the literal all is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the exps to CPU.

Speculative decoding: canonical ngram-mod

ik_llama main rejects the legacy --spec-ngram-mod-n-min/max/match flags. The canonical form is a repeated --spec-type ngram-mod:<k=v>:

• SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24

Mapping from the old upstream config:

legacy flag	old value	new canonical key
--spec-ngram-mod-n-min	48	n_min=48
--spec-ngram-mod-n-max	64	n_max=64
--spec-ngram-mod-n-match	24	ngram_size_n=24

MTP remains intentionally disabled.

Flash-attention + sampling

Flash-attention is on by default in ik_llama (flash_attn=true), so no -fa/--flash-attn flag is needed. Sampling, thread, batch, context, and checkpoint tunables (TEMPERATURE, TOP_K, TOP_P, MIN_P, thread, BATCH_SIZE, UBATCH_SIZE, CTX_SIZE, PARALLEL, CTX_CHECKPOINTS, NO_MMAP) are unchanged and pass straight through.

MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

flag	meaning	our use
--split-mode graph	split tensors+graph across devices; strong for MoE; requires 2+ devices	single 3090 → use none
-ooae / --offload-only-active-experts	default ON: copy only active experts RAM→VRAM	keep ON (huge for A3B: few experts active)
-grt q8_0	reduce type for inter-GPU data exchange	not needed (single GPU)
--fit	auto-offload to fit VRAM	conflicts with --override-tensor; do not use together (runtime error)
-n-cpu-moe N / --cpu-moe	keep MoE weights of first N / all layers on CPU	also conflicts with --fit; our --override-tensor is finer-grained
--fit-margin	VRAM safety margin for --fit	only relevant with --fit

The current deployment uses --override-tensor (not --fit or --n-cpu-moe) for the expert offload, so the --fit-conflict rules do not apply.

Rollback

The previous runtime upstream llama.cpp b10121 and its binary (/home/dank/llama.cpp/build/bin/llama-server) are left untouched. To rollback to the old stack, change LLAMA_SERVER= in scripts/frank-default-small.env back to that path and CACHE_TYPE_K/V back to f16 with Hadamard off, and re-append the --spec-ngram-mod-* flags. The model file is not changed either way.

See Also

• dealign-ornith-crack — the Ornith Q6_K model run
• Dankserver System Specs — the host
• deployment — service-scoped git deploy

Citations

• ikawrakow/ik_llama.cpp README — MoE offload guidance and the --split-mode graph + CPU-offload warning
• ik_llama.cpp docs/build.md — build flags incl. GGML_CUDA_FA_ALL_QUANTS and GGML_IQK_FA_ALL_QUANTS
• ik_llama.cpp docs/parameters.md — q8_KV (PR 208), K/V Hadamard options, -cuda graphs=0 workaround, and --split-mode graph MoE note (PR 1080)