serverAppConfigs/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md

---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
  - ikllama
  - llamacpp
  - cuda
  - dankserver
  - moe
  - ornith
  - kv-cache
  - hadamard
timestamp: 2026-08-26
---

# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

> **Active 2026-08-26.** `frank-small` switched from upstream
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
> RTX 3090. This page documents the build, the mandatory FA build flags, and
> the MoE / CPU-offload runtime arguments.

The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
model runbook.

## Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

## Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
13.2 toolkit is installed via **pip** inside the vllm venv at
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
`.so` aliases and static libs CMake needs.

```bash
# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8
```

### ⚠️ MANDATORY: always include the FA all-quants flags

> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
> BOTH:
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
>   into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
>   a subset and **errors at runtime** telling you to recompile when a combo
>   (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
>   the GPU, so this is the flag that matters here.
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
>   IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
>   FA kernels. Relevant because the MoE experts run on the CPU; the KV
>   cache that backs CPU-side attention uses the IQK FA path.
>
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).

### CUDA stage note

`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
CCCL CUDA version skew by defining
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
a strict header lint). Do not re-run configure without that flag or you get
a hard compile error.

## Runtime configuration

Files: `scripts/serve-frank-small.sh`, `scripts/frank-small.env`,
`scripts/frank-small.service` in the repo. The service just calls
the serve script; the env file holds all tunables.

### KV cache: K default f16 / V q6_0 + Hadamard (V only)

- `CACHE_TYPE_K=f16` — K cache at the default full-precision f16
  (passes `--cache-type-k f16`). No Hadamard on K.
- `CACHE_TYPE_V=q6_0` — V cache at 6-bit k-quant
  (passes `--cache-type-v q6_0`).
- `V_CACHE_HADAMARD=1` — `--v-cache-hadamard` on V only. The Hadamard
  transform preserves V quality at 6-bit; K is full-precision f16 and does
  not need it (`K_CACHE_HADAMARD=0`). Implemented for both CUDA
  (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. Flash-attention is **on
  by default**.

> **KV cache type history on this build (`b166e269`):**
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
>   `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV`; `iq5_*` only
>   exist as weight quants. `--cache-type-k iq5_k` throws `Invalid cache
>   type`.
> - `q8_KV` **crashes** with a `ggml view` size assert
>   (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
>   QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
> - The deployment settled on K at default **f16** (full precision, no
>   Hadamard) with V at **q6_0 + Hadamard**.

With 3 parallel slots at 163840 context the KV cache is large; keeping K at
full precision f16 while quantizing only V (with Hadamard) is the chosen
budget split. The GPU-side expert set stays reduced to fit (see the expert
section).

### Split mode: none (single GPU)

- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
  single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
  single device:

  > `llama error loading model: It is not possible to use split mode 'graph'
  > with less than 2 devices`

  So `graph` is **not** a harmless no-op on one GPU — it is a hard load
  error (this bit us on the first deploy and loop the service). It also still
  works with the `--override-tensor` expert pins.

> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
> single-GPU host you must use `none` (or `layer`, which also needs >1).
> Do not assume `--split-mode graph` degrades gracefully on one GPU.

### Expert offload (MoE: CPU with a partial GPU slice)

The expert-offload pattern from upstream llama.cpp carries over: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
no exps and are untouched by the regex.

- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
  experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
  24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
  full-attention set) are left off the list so their experts load to CUDA.
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
  integer (the literal `all` is rejected). 999 means "load as many tensor
  layers as possible", while the expert override still pins the listed exps
  to CPU.

> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
> disables it. `only_active_experts` is also on by default. Confirmed in the
> boot log for the running service.

To move more experts back to the GPU later, drop more block numbers from
`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
cache there is little headroom, so the default keeps only 35-40 on GPU.

### Qwen-Sharp chat template via --jinja

`--chat-template-file` alone only stores the template text; ik_llama **does
not** render a custom template as real Jinja unless `--jinja` is also given
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
fallback>`). Without `--jinja` it warns it only supports "commonly used
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
template's thinking / image / tool formatting. So the serve script passes
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).

### Speculative decoding: disabled (3 parallel slots)

`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
intentionally disabled because ik_llama **limits any speculative stage to a
a single server slot**: with `--parallel 3` a configured spec stage
hard-exits at load with

> `ERR [load_model] Speculative decoding is currently limited to a single
> server slot (-np 1).` / `n_parallel=3`

This service keeps **3 parallel slots** (one per user family via
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
need the speculation it would cost a slot for.

> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
> extra slots win.

### Flash-attention + sampling

Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
`NO_MMAP`) are unchanged and pass straight through.

## MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

| flag | meaning | our use |
|------|---------|---------|
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |

The current deployment uses `--override-tensor` (not `--fit` or
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
apply.

## Rollback

The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-small.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
spec / `--spec-type` flags. The model file is not changed either way.

## See Also

- [[dealign-ornith-crack]] — the Ornith Q6_K model run
- [[Dankserver System Specs]] — the host
- [[deployment]] — service-scoped git deploy

## Citations

- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
  MoE offload guidance and the `--split-mode graph` + CPU-offload warning
- [ik_llama.cpp serverAppConfigs-docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
  build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
- [ik_llama.cpp serverAppConfigs-docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
  q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
  `--split-mode graph` MoE note (PR 1080)