Frank
5829a0bfe8
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
262 lines
12 KiB
Markdown
262 lines
12 KiB
Markdown
---
|
|
type: reference
|
|
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
|
|
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
|
tags:
|
|
- ikllama
|
|
- llamacpp
|
|
- cuda
|
|
- dankserver
|
|
- moe
|
|
- ornith
|
|
- kv-cache
|
|
- hadamard
|
|
timestamp: 2026-08-26
|
|
---
|
|
|
|
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
|
|
|
> **Active 2026-08-26.** `frank-default-small` switched from upstream
|
|
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
|
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
|
> the MoE / CPU-offload runtime arguments.
|
|
|
|
The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
|
|
3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
|
|
encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
|
|
Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
|
|
model runbook.
|
|
|
|
## Why ik_llama.cpp
|
|
|
|
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
|
|
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
|
|
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
|
|
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
|
|
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
|
|
|
|
## Build recipe (CUDA 13.2 via staged pip toolkit)
|
|
|
|
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
|
|
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
|
|
13.2 toolkit is installed via **pip** inside the vllm venv at
|
|
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
|
|
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
|
|
`.so` aliases and static libs CMake needs.
|
|
|
|
```bash
|
|
# 1. Clone a pinned ik_llama.cpp
|
|
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
|
|
cd /home/dank/ik_llama.cpp
|
|
git checkout b166e269 # the version this deployment was built from
|
|
|
|
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
|
|
cmake -B build \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DGGML_CUDA=ON \
|
|
-DGGML_CUDA_FA_ALL_QUANTS=ON \
|
|
-DGGML_IQK_FA_ALL_QUANTS=ON \
|
|
-DCMAKE_CUDA_ARCHITECTURES=86 \
|
|
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
|
|
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
|
|
-DGGML_NATIVE=ON
|
|
|
|
# 3. Build
|
|
cmake --build build --config Release -j 8
|
|
```
|
|
|
|
### ⚠️ MANDATORY: always include the FA all-quants flags
|
|
|
|
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
|
|
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
|
|
> BOTH:
|
|
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
|
|
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
|
|
> a subset and **errors at runtime** telling you to recompile when a combo
|
|
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
|
|
> the GPU, so this is the flag that matters here.
|
|
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
|
|
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
|
|
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
|
|
> cache that backs CPU-side attention uses the IQK FA path.
|
|
>
|
|
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
|
|
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
|
|
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
|
|
|
|
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
|
|
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
|
|
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
|
|
|
|
### CUDA stage note
|
|
|
|
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
|
|
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
|
|
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
|
|
CCCL CUDA version skew by defining
|
|
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
|
|
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
|
|
a strict header lint). Do not re-run configure without that flag or you get
|
|
a hard compile error.
|
|
|
|
## Runtime configuration
|
|
|
|
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
|
|
`scripts/frank-default-small.service` in the repo. The service just calls
|
|
the serve script; the env file holds all tunables.
|
|
|
|
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
|
|
|
|
- `CACHE_TYPE_K=f16` — K cache at the default full-precision f16
|
|
(passes `--cache-type-k f16`). No Hadamard on K.
|
|
- `CACHE_TYPE_V=q6_0` — V cache at 6-bit k-quant
|
|
(passes `--cache-type-v q6_0`).
|
|
- `V_CACHE_HADAMARD=1` — `--v-cache-hadamard` on V only. The Hadamard
|
|
transform preserves V quality at 6-bit; K is full-precision f16 and does
|
|
not need it (`K_CACHE_HADAMARD=0`). Implemented for both CUDA
|
|
(`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. Flash-attention is **on
|
|
by default**.
|
|
|
|
> **KV cache type history on this build (`b166e269`):**
|
|
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
|
|
> `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV`; `iq5_*` only
|
|
> exist as weight quants. `--cache-type-k iq5_k` throws `Invalid cache
|
|
> type`.
|
|
> - `q8_KV` **crashes** with a `ggml view` size assert
|
|
> (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
|
|
> QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
|
|
> - The deployment settled on K at default **f16** (full precision, no
|
|
> Hadamard) with V at **q6_0 + Hadamard**.
|
|
|
|
With 3 parallel slots at 163840 context the KV cache is large; keeping K at
|
|
full precision f16 while quantizing only V (with Hadamard) is the chosen
|
|
budget split. The GPU-side expert set stays reduced to fit (see the expert
|
|
section).
|
|
|
|
### Split mode: none (single GPU)
|
|
|
|
- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
|
|
single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
|
|
single device:
|
|
|
|
> `llama error loading model: It is not possible to use split mode 'graph'
|
|
> with less than 2 devices`
|
|
|
|
So `graph` is **not** a harmless no-op on one GPU — it is a hard load
|
|
error (this bit us on the first deploy and loop the service). It also still
|
|
works with the `--override-tensor` expert pins.
|
|
|
|
> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
|
|
> single-GPU host you must use `none` (or `layer`, which also needs >1).
|
|
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
|
|
|
|
### Expert offload (MoE: CPU with a partial GPU slice)
|
|
|
|
The expert-offload pattern from upstream llama.cpp carries over: the
|
|
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
|
the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
|
|
Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
|
|
MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
|
|
are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
|
|
quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
|
|
no exps and are untouched by the regex.
|
|
|
|
- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
|
|
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
|
|
24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
|
|
full-attention set) are left off the list so their experts load to CUDA.
|
|
- `OVERRIDE_BUFT=CPU`.
|
|
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
|
integer (the literal `all` is rejected). 999 means "load as many tensor
|
|
layers as possible", while the expert override still pins the listed exps
|
|
to CPU.
|
|
|
|
> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
|
|
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
|
|
> disables it. `only_active_experts` is also on by default. Confirmed in the
|
|
> boot log for the running service.
|
|
|
|
To move more experts back to the GPU later, drop more block numbers from
|
|
`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
|
|
headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
|
|
cache there is little headroom, so the default keeps only 35-40 on GPU.
|
|
|
|
### Qwen-Sharp chat template via --jinja
|
|
|
|
`--chat-template-file` alone only stores the template text; ik_llama **does
|
|
not** render a custom template as real Jinja unless `--jinja` is also given
|
|
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
|
|
fallback>`). Without `--jinja` it warns it only supports "commonly used
|
|
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
|
|
template's thinking / image / tool formatting. So the serve script passes
|
|
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
|
|
|
|
### Speculative decoding: disabled (3 parallel slots)
|
|
|
|
`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
|
|
intentionally disabled because ik_llama **limits any speculative stage to a
|
|
a single server slot**: with `--parallel 3` a configured spec stage
|
|
hard-exits at load with
|
|
|
|
> `ERR [load_model] Speculative decoding is currently limited to a single
|
|
> server slot (-np 1).` / `n_parallel=3`
|
|
|
|
This service keeps **3 parallel slots** (one per user family via
|
|
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
|
|
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
|
|
need the speculation it would cost a slot for.
|
|
|
|
> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
|
|
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
|
|
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
|
|
> extra slots win.
|
|
|
|
### Flash-attention + sampling
|
|
|
|
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
|
|
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
|
|
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
|
|
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
|
|
`NO_MMAP`) are unchanged and pass straight through.
|
|
|
|
## MoE / hybrid-arg reference (not in upstream llama.cpp)
|
|
|
|
Several ik_llama knobs matter for a CPU-MoE offload host:
|
|
|
|
| flag | meaning | our use |
|
|
|------|---------|---------|
|
|
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
|
|
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
|
|
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
|
|
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
|
|
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
|
|
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
|
|
|
|
The current deployment uses `--override-tensor` (not `--fit` or
|
|
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
|
|
apply.
|
|
|
|
## Rollback
|
|
|
|
The previous runtime `upstream llama.cpp b10121` and its binary
|
|
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
|
rollback to the old stack, change `LLAMA_SERVER=` in
|
|
`scripts/frank-default-small.env` back to that path and
|
|
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
|
|
spec / `--spec-type` flags. The model file is not changed either way.
|
|
|
|
## See Also
|
|
|
|
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
|
|
- [[Dankserver System Specs]] — the host
|
|
- [[deployment]] — service-scoped git deploy
|
|
|
|
## Citations
|
|
|
|
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
|
|
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
|
|
- [ik_llama.cpp serverAppConfigs-docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
|
|
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
|
|
- [ik_llama.cpp serverAppConfigs-docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
|
|
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
|
|
`--split-mode graph` MoE note (PR 1080) |