Frank
3f3b1bdb27
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja fallback). Without it the Qwen-Sharp template's thinking/image/tool formatting is silently dropped. Add USE_JINJA=1 -> --jinja. Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34 back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
248 lines
12 KiB
Markdown
248 lines
12 KiB
Markdown
---
|
|
type: reference
|
|
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
|
|
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
|
tags:
|
|
- ikllama
|
|
- llamacpp
|
|
- cuda
|
|
- dankserver
|
|
- moe
|
|
- ornith
|
|
- kv-cache
|
|
- hadamard
|
|
timestamp: 2026-08-26
|
|
---
|
|
|
|
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
|
|
|
> **Active 2026-08-26.** `frank-default-small` switched from upstream
|
|
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
|
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
|
> the MoE / CPU-offload runtime arguments.
|
|
|
|
The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
|
|
Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
|
|
Speculative decoding is **disabled** (ik_llama limits spec to a single slot;
|
|
this service keeps 3 parallel slots). The low-parameter MoE does not need
|
|
it. See [[dealign-ornith-crack]] for the model runbook.
|
|
|
|
## Why ik_llama.cpp
|
|
|
|
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
|
|
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
|
|
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
|
|
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
|
|
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
|
|
|
|
## Build recipe (CUDA 13.2 via staged pip toolkit)
|
|
|
|
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
|
|
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
|
|
13.2 toolkit is installed via **pip** inside the vllm venv at
|
|
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
|
|
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
|
|
`.so` aliases and static libs CMake needs.
|
|
|
|
```bash
|
|
# 1. Clone a pinned ik_llama.cpp
|
|
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
|
|
cd /home/dank/ik_llama.cpp
|
|
git checkout b166e269 # the version this deployment was built from
|
|
|
|
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
|
|
cmake -B build \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DGGML_CUDA=ON \
|
|
-DGGML_CUDA_FA_ALL_QUANTS=ON \
|
|
-DGGML_IQK_FA_ALL_QUANTS=ON \
|
|
-DCMAKE_CUDA_ARCHITECTURES=86 \
|
|
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
|
|
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
|
|
-DGGML_NATIVE=ON
|
|
|
|
# 3. Build
|
|
cmake --build build --config Release -j 8
|
|
```
|
|
|
|
### ⚠️ MANDATORY: always include the FA all-quants flags
|
|
|
|
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
|
|
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
|
|
> BOTH:
|
|
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
|
|
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
|
|
> a subset and **errors at runtime** telling you to recompile when a combo
|
|
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
|
|
> the GPU, so this is the flag that matters here.
|
|
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
|
|
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
|
|
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
|
|
> cache that backs CPU-side attention uses the IQK FA path.
|
|
>
|
|
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
|
|
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
|
|
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
|
|
|
|
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
|
|
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
|
|
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
|
|
|
|
### CUDA stage note
|
|
|
|
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
|
|
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
|
|
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
|
|
CCCL CUDA version skew by defining
|
|
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
|
|
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
|
|
a strict header lint). Do not re-run configure without that flag or you get
|
|
a hard compile error.
|
|
|
|
## Runtime configuration
|
|
|
|
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
|
|
`scripts/frank-default-small.service` in the repo. The service just calls
|
|
the serve script; the env file holds all tunables.
|
|
|
|
### KV cache: q8_0 + Hadamard
|
|
|
|
- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` — standard 8-bit KV cache type
|
|
(passes `--cache-type-k q8_0 --cache-type-v q8_0`).
|
|
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
|
|
`--v-cache-hadamard`. The Hadamard transform preserves KV quality at
|
|
8-bit (docs recommend it for quantized KV below Q6_0). Implemented for
|
|
both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths.
|
|
- Flash-attention is **on by default** (no `-fa` flag needed).
|
|
|
|
> ⚠️ **Do not use `q8_KV` (fast type) here.** On this commit (`b166e269`)
|
|
> the fast `q8_KV` cache type crashes at model init on this QWEN3MOE/MTP +
|
|
> multimodal arch with a `ggml view` size assertion
|
|
> (`ggml.c:5427: GGML_ASSERT(view_src == NULL || ... data_size + view_offs`)
|
|
> right after the output buffer is created. It loops the service. Plain
|
|
> `q8_0` (and `f16`) work and were smoke-tested with correct output; this
|
|
> config uses `q8_0`. This was isolated empirically by toggling cache type
|
|
> while keeping the override/hadamard flags.
|
|
|
|
These keep the 163840-slot / 491520-pool context while roughly halving KV
|
|
cache RAM vs f16.
|
|
|
|
### Split mode: none (single GPU)
|
|
|
|
- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
|
|
single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
|
|
single device:
|
|
|
|
> `llama error loading model: It is not possible to use split mode 'graph'
|
|
> with less than 2 devices`
|
|
|
|
So `graph` is **not** a harmless no-op on one GPU — it is a hard load
|
|
error (this bit us on the first deploy and loop the service). It also still
|
|
works with the `--override-tensor` expert pins.
|
|
|
|
> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
|
|
> single-GPU host you must use `none` (or `layer`, which also needs >1).
|
|
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
|
|
|
|
### Expert offload (MoE: CPU with a partial GPU slice)
|
|
|
|
The expert-offload pattern from upstream llama.cpp carries over: the
|
|
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
|
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
|
|
On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA**
|
|
(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of
|
|
that CPU range stays on CPU.
|
|
|
|
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
|
|
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
|
|
24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention +
|
|
35-40 set) are left off the list so their experts load to CUDA.
|
|
- `OVERRIDE_BUFT=CPU`.
|
|
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
|
integer (the literal `all` is rejected). 999 means "load as many tensor
|
|
layers as possible", while the expert override still pins the listed exps
|
|
to CPU.
|
|
|
|
To move more experts back to the GPU later, drop more block numbers from
|
|
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
|
|
first and keep ~2-3 GiB margin for KV/compute).
|
|
|
|
### Qwen-Sharp chat template via --jinja
|
|
|
|
`--chat-template-file` alone only stores the template text; ik_llama **does
|
|
not** render a custom template as real Jinja unless `--jinja` is also given
|
|
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
|
|
fallback>`). Without `--jinja` it warns it only supports "commonly used
|
|
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
|
|
template's thinking / image / tool formatting. So the serve script passes
|
|
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
|
|
|
|
### Speculative decoding: disabled (3 parallel slots)
|
|
|
|
`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
|
|
intentionally disabled because ik_llama **limits any speculative stage to a
|
|
a single server slot**: with `--parallel 3` a configured spec stage
|
|
hard-exits at load with
|
|
|
|
> `ERR [load_model] Speculative decoding is currently limited to a single
|
|
> server slot (-np 1).` / `n_parallel=3`
|
|
|
|
This service keeps **3 parallel slots** (one per user family via
|
|
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
|
|
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
|
|
need the speculation it would cost a slot for.
|
|
|
|
> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
|
|
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
|
|
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
|
|
> extra slots win.
|
|
|
|
### Flash-attention + sampling
|
|
|
|
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
|
|
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
|
|
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
|
|
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
|
|
`NO_MMAP`) are unchanged and pass straight through.
|
|
|
|
## MoE / hybrid-arg reference (not in upstream llama.cpp)
|
|
|
|
Several ik_llama knobs matter for a CPU-MoE offload host:
|
|
|
|
| flag | meaning | our use |
|
|
|------|---------|---------|
|
|
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
|
|
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
|
|
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
|
|
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
|
|
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
|
|
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
|
|
|
|
The current deployment uses `--override-tensor` (not `--fit` or
|
|
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
|
|
apply.
|
|
|
|
## Rollback
|
|
|
|
The previous runtime `upstream llama.cpp b10121` and its binary
|
|
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
|
rollback to the old stack, change `LLAMA_SERVER=` in
|
|
`scripts/frank-default-small.env` back to that path and
|
|
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
|
|
spec / `--spec-type` flags. The model file is not changed either way.
|
|
|
|
## See Also
|
|
|
|
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
|
|
- [[Dankserver System Specs]] — the host
|
|
- [[deployment]] — service-scoped git deploy
|
|
|
|
## Citations
|
|
|
|
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
|
|
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
|
|
- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
|
|
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
|
|
- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
|
|
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
|
|
`--split-mode graph` MoE note (PR 1080) |