--- type: reference title: ik_llama.cpp on Dankserver (Ornith Q6_K) description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. tags: - ikllama - llamacpp - cuda - dankserver - moe - ornith - kv-cache - hadamard timestamp: 2026-08-26 --- # ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK) > **Active 2026-08-26.** `frank-default-small` switched from upstream > llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the > RTX 3090. This page documents the build, the mandatory FA build flags, and > the MoE / CPU-offload runtime arguments. The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF. Speculative decoding is **disabled** (ik_llama limits spec to a single slot; this service keeps 3 parallel slots). The low-parameter MoE does not need it. See [[dealign-ornith-crack]] for the model runbook. ## Why ik_llama.cpp ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is exactly the supported hybrid the fork targets. Only CPU and CUDA backends are fully functional; ROCm / Vulkan / Metal are unsupported upstream. ## Build recipe (CUDA 13.2 via staged pip toolkit) The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit` is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA 13.2 toolkit is installed via **pip** inside the vllm venv at `/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged, self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned `.so` aliases and static libs CMake needs. ```bash # 1. Clone a pinned ik_llama.cpp git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp cd /home/dank/ik_llama.cpp git checkout b166e269 # the version this deployment was built from # 2. Configure (stage already symlinks nvcc/include/lib for cu13) cmake -B build \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_CUDA=ON \ -DGGML_CUDA_FA_ALL_QUANTS=ON \ -DGGML_IQK_FA_ALL_QUANTS=ON \ -DCMAKE_CUDA_ARCHITECTURES=86 \ -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \ -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \ -DGGML_NATIVE=ON # 3. Build cmake --build build --config Release -j 8 ``` ### ⚠️ MANDATORY: always include the FA all-quants flags > **Rule.** Any ik_llama.cpp build that serves a quantized KV cache > (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with > BOTH: > - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo > into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships > a subset and **errors at runtime** telling you to recompile when a combo > (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on > the GPU, so this is the flag that matters here. > - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/ > IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK** > FA kernels. Relevant because the MoE experts run on the CPU; the KV > cache that backs CPU-side attention uses the IQK FA path. > > These are the old "asymmetrical quants" compile-time gates. On a CUDA > host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a > "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time. `GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so `-march=native` enables the AVX512 / VNNI IQK kernels automatically — verifiable with `system_info: AVX512_VNNI = 1` in the boot log). ### CUDA stage note `/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's (CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so` and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the CCCL CUDA version skew by defining `-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips a strict header lint). Do not re-run configure without that flag or you get a hard compile error. ## Runtime configuration Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`, `scripts/frank-default-small.service` in the repo. The service just calls the serve script; the env file holds all tunables. ### KV cache: q8_0 + Hadamard - `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` — standard 8-bit KV cache type (passes `--cache-type-k q8_0 --cache-type-v q8_0`). - `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and `--v-cache-hadamard`. The Hadamard transform preserves KV quality at 8-bit (docs recommend it for quantized KV below Q6_0). Implemented for both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. - Flash-attention is **on by default** (no `-fa` flag needed). > ⚠️ **Do not use `q8_KV` (fast type) here.** On this commit (`b166e269`) > the fast `q8_KV` cache type crashes at model init on this QWEN3MOE/MTP + > multimodal arch with a `ggml view` size assertion > (`ggml.c:5427: GGML_ASSERT(view_src == NULL || ... data_size + view_offs`) > right after the output buffer is created. It loops the service. Plain > `q8_0` (and `f16`) work and were smoke-tested with correct output; this > config uses `q8_0`. This was isolated empirically by toggling cache type > while keeping the override/hadamard flags. These keep the 163840-slot / 491520-pool context while roughly halving KV cache RAM vs f16. ### Split mode: none (single GPU) - `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a single device: > `llama error loading model: It is not possible to use split mode 'graph' > with less than 2 devices` So `graph` is **not** a harmless no-op on one GPU — it is a hard load error (this bit us on the first deploy and loop the service). It also still works with the `--override-tensor` expert pins. > **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a > single-GPU host you must use `none` (or `layer`, which also needs >1). > Do not assume `--split-mode graph` degrades gracefully on one GPU. ### Expert offload (MoE: CPU with a partial GPU slice) The expert-offload pattern from upstream llama.cpp carries over: the `--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA** (~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of that CPU range stays on CPU. - `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| 24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention + 35-40 set) are left off the list so their experts load to CUDA. - `OVERRIDE_BUFT=CPU`. - `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an integer (the literal `all` is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the listed exps to CPU. To move more experts back to the GPU later, drop more block numbers from `OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom first and keep ~2-3 GiB margin for KV/compute). ### Qwen-Sharp chat template via --jinja `--chat-template-file` alone only stores the template text; ik_llama **does not** render a custom template as real Jinja unless `--jinja` is also given (`inputs.use_jinja ? common_chat_templates_apply_jinja : `). Without `--jinja` it warns it only supports "commonly used templates" and silently uses built-in handlers, dropping the Qwen-Sharp template's thinking / image / tool formatting. So the serve script passes both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`). ### Speculative decoding: disabled (3 parallel slots) `SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is intentionally disabled because ik_llama **limits any speculative stage to a a single server slot**: with `--parallel 3` a configured spec stage hard-exits at load with > `ERR [load_model] Speculative decoding is currently limited to a single > server slot (-np 1).` / `n_parallel=3` This service keeps **3 parallel slots** (one per user family via `PARALLEL=3`), so speculative decoding cannot be used here. MTP stays disabled. The model is a low-parameter MoE (3B active of 35B) and does not need the speculation it would cost a slot for. > **Lesson.** Do not combine speculative decoding with `--parallel > 1` in > ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep > multiple slots and set `SPEC_TYPE=none`. For a multi-user service the > extra slots win. ### Flash-attention + sampling Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no `-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread, `BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`, `NO_MMAP`) are unchanged and pass straight through. ## MoE / hybrid-arg reference (not in upstream llama.cpp) Several ik_llama knobs matter for a CPU-MoE offload host: | flag | meaning | our use | |------|---------|---------| | `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none` | `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) | | `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) | | `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) | | `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained | | `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` | The current deployment uses `--override-tensor` (not `--fit` or `--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not apply. ## Rollback The previous runtime `upstream llama.cpp b10121` and its binary (`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To rollback to the old stack, change `LLAMA_SERVER=` in `scripts/frank-default-small.env` back to that path and `CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram spec / `--spec-type` flags. The model file is not changed either way. ## See Also - [[dealign-ornith-crack]] — the Ornith Q6_K model run - [[Dankserver System Specs]] — the host - [[deployment]] — service-scoped git deploy ## Citations - [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) — MoE offload guidance and the `--split-mode graph` + CPU-offload warning - [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) — build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS` - [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) — q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and `--split-mode graph` MoE note (PR 1080)