---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
  - ikllama
  - llamacpp
  - cuda
  - dankserver
  - moe
  - ornith
  - kv-cache
  - hadamard
timestamp: 2026-08-26
---

# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

> **Active 2026-08-26.** `frank-default-small` switched from upstream
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
> RTX 3090. This page documents the build, the mandatory FA build flags, and
> the MoE / CPU-offload runtime arguments.

The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
Speculation stays ngram-only (MTP intentionally disabled). See
[[dealign-ornith-crack]] for the model runbook.

## Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

## Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
13.2 toolkit is installed via **pip** inside the vllm venv at
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
`.so` aliases and static libs CMake needs.

```bash
# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8
```

### ⚠️ MANDATORY: always include the FA all-quants flags

> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
> BOTH:
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
>   into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
>   a subset and **errors at runtime** telling you to recompile when a combo
>   (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
>   the GPU, so this is the flag that matters here.
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
>   IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
>   FA kernels. Relevant because the MoE experts run on the CPU; the KV
>   cache that backs CPU-side attention uses the IQK FA path.
>
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).

### CUDA stage note

`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
CCCL CUDA version skew by defining
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
a strict header lint). Do not re-run configure without that flag or you get
a hard compile error.

## Runtime configuration

Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
`scripts/frank-default-small.service` in the repo. The service just calls
the serve script; the env file holds all tunables.

### KV cache: q8_KV + Hadamard

- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik
  PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`.
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
  `--v-cache-hadamard`. The Hadamard transform preserves KV quality at
  8-bit (docs recommend it for quantized KV below Q6_0). Implemented for
  both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths.

These keep the 163840-slot / 491520-pool context while roughly halving KV
cache RAM vs f16.

### Split mode: none (single GPU)

- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
  single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
  single device:

  > `llama error loading model: It is not possible to use split mode 'graph'
  > with less than 2 devices`

  So `graph` is **not** a harmless no-op on one GPU — it is a hard load
  error (this bit us on the first deploy and loop the service). It also still
  works with the `--override-tensor` expert pins.

> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
> single-GPU host you must use `none` (or `layer`, which also needs >1).
> Do not assume `--split-mode graph` degrades gracefully on one GPU.

### Expert offload (MoE on CPU)

The expert-offload pattern is unchanged from upstream llama.cpp: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.

- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
  integer (the literal `all` is rejected). 999 means "load as many tensor
  layers as possible", while the expert override still pins the exps to CPU.

### Speculative decoding: canonical ngram-mod

ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`

Mapping from the old upstream config:
| legacy flag | old value | new canonical key |
|-------------|----------|-------------------|
| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |

MTP remains intentionally disabled.

### Flash-attention + sampling

Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
`NO_MMAP`) are unchanged and pass straight through.

## MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

| flag | meaning | our use |
|------|---------|---------|
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |

The current deployment uses `--override-tensor` (not `--fit` or
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
apply.

## Rollback

The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-default-small.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
`--spec-ngram-mod-*` flags. The model file is not changed either way.

## See Also

- [[dealign-ornith-crack]] — the Ornith Q6_K model run
- [[Dankserver System Specs]] — the host
- [[deployment]] — service-scoped git deploy

## Citations

- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
  MoE offload guidance and the `--split-mode graph` + CPU-offload warning
- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
  build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
  q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
  `--split-mode graph` MoE note (PR 1080)