frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13 toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the old binary is kept for rollback. Runtime changes: - KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard - --split-mode graph (single-GPU no-op, retained per operator pref) - migrate ngram speculation to canonical --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24 (legacy --spec-ngram-mod-* are rejected in ik main) - N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay pinned to CPU via --override-tensor Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory for the quantized KV cache; recipe + reasoning baked into docs/LLM/ikllama-ornith.md. Tests and both indices updated.
This commit is contained in:
Frank
@@ -0,0 +1,208 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode graph, ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- kv-cache
|
||||
- hadamard
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
||||
|
||||
> **Active 2026-08-26.** `frank-default-small` switched from upstream
|
||||
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
||||
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
||||
> the MoE / CPU-offload runtime arguments.
|
||||
|
||||
The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
|
||||
Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
|
||||
Speculation stays ngram-only (MTP intentionally disabled). See
|
||||
[[dealign-ornith-crack]] for the model runbook.
|
||||
|
||||
## Why ik_llama.cpp
|
||||
|
||||
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
|
||||
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
|
||||
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
|
||||
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
|
||||
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
|
||||
|
||||
## Build recipe (CUDA 13.2 via staged pip toolkit)
|
||||
|
||||
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
|
||||
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
|
||||
13.2 toolkit is installed via **pip** inside the vllm venv at
|
||||
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
|
||||
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
|
||||
`.so` aliases and static libs CMake needs.
|
||||
|
||||
```bash
|
||||
# 1. Clone a pinned ik_llama.cpp
|
||||
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
|
||||
cd /home/dank/ik_llama.cpp
|
||||
git checkout b166e269 # the version this deployment was built from
|
||||
|
||||
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
|
||||
cmake -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_FA_ALL_QUANTS=ON \
|
||||
-DGGML_IQK_FA_ALL_QUANTS=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 \
|
||||
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
|
||||
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
|
||||
-DGGML_NATIVE=ON
|
||||
|
||||
# 3. Build
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
### ⚠️ MANDATORY: always include the FA all-quants flags
|
||||
|
||||
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
|
||||
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
|
||||
> BOTH:
|
||||
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
|
||||
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
|
||||
> a subset and **errors at runtime** telling you to recompile when a combo
|
||||
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
|
||||
> the GPU, so this is the flag that matters here.
|
||||
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
|
||||
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
|
||||
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
|
||||
> cache that backs CPU-side attention uses the IQK FA path.
|
||||
>
|
||||
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
|
||||
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
|
||||
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
|
||||
|
||||
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
|
||||
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
|
||||
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
|
||||
|
||||
### CUDA stage note
|
||||
|
||||
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
|
||||
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
|
||||
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
|
||||
CCCL CUDA version skew by defining
|
||||
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
|
||||
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
|
||||
a strict header lint). Do not re-run configure without that flag or you get
|
||||
a hard compile error.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
|
||||
`scripts/frank-default-small.service` in the repo. The service just calls
|
||||
the serve script; the env file holds all tunables.
|
||||
|
||||
### KV cache: q8_KV + Hadamard
|
||||
|
||||
- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik
|
||||
PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`.
|
||||
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
|
||||
`--v-cache-hadamard`. The Hadamard transform preserves KV quality at
|
||||
8-bit (docs recommend it for quantized KV below Q6_0). Implemented for
|
||||
both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths.
|
||||
|
||||
These keep the 163840-slot / 491520-pool context while roughly halving KV
|
||||
cache RAM vs f16.
|
||||
|
||||
### Split-mode graph
|
||||
|
||||
- `SPLIT_MODE=graph` → `--split-mode graph`. On a single 3090 this is a
|
||||
no-op for correctness but keeps the graph scheduled across the compute
|
||||
buffers. ik recommends it for MoE when more than one device is present;
|
||||
here it is retained per operator preference alongside the explicit
|
||||
`--override-tensor` expert pins.
|
||||
|
||||
> **Caveat:** the ik_llama README warns that `--split-mode graph` combined
|
||||
> with partial GPU offload (`--cpu-moe` / `--n-cpu-moe` / `--override-tensor`)
|
||||
> can rarely produce incoherent output. The documented mitigation is
|
||||
> `-cuda graphs=0`. Smoke-test after every load; if a response is gibberish,
|
||||
> add `-cuda graphs=0` to the serve line and re-test.
|
||||
|
||||
### Expert offload (MoE on CPU)
|
||||
|
||||
The expert-offload pattern is unchanged from upstream llama.cpp: the
|
||||
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
||||
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
|
||||
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
|
||||
- `OVERRIDE_BUFT=CPU`.
|
||||
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
||||
integer (the literal `all` is rejected). 999 means "load as many tensor
|
||||
layers as possible", while the expert override still pins the exps to CPU.
|
||||
|
||||
### Speculative decoding: canonical ngram-mod
|
||||
|
||||
ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
|
||||
flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
|
||||
- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`
|
||||
|
||||
Mapping from the old upstream config:
|
||||
| legacy flag | old value | new canonical key |
|
||||
|-------------|----------|-------------------|
|
||||
| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
|
||||
| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
|
||||
| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |
|
||||
|
||||
MTP remains intentionally disabled.
|
||||
|
||||
### Flash-attention + sampling
|
||||
|
||||
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
|
||||
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
|
||||
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
|
||||
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
|
||||
`NO_MMAP`) are unchanged and pass straight through.
|
||||
|
||||
## MoE / hybrid-arg reference (not in upstream llama.cpp)
|
||||
|
||||
Several ik_llama knobs matter for a CPU-MoE offload host:
|
||||
|
||||
| flag | meaning | our use |
|
||||
|------|---------|---------|
|
||||
| `--split-mode graph` | split tensors+graph across devices; strong for MoE | retained (single-GPU no-op) |
|
||||
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
|
||||
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
|
||||
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
|
||||
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
|
||||
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
|
||||
|
||||
The current deployment uses `--override-tensor` (not `--fit` or
|
||||
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
|
||||
apply.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous runtime `upstream llama.cpp b10121` and its binary
|
||||
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
||||
rollback to the old stack, change `LLAMA_SERVER=` in
|
||||
`scripts/frank-default-small.env` back to that path and
|
||||
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
|
||||
`--spec-ngram-mod-*` flags. The model file is not changed either way.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
|
||||
- [[Dankserver System Specs]] — the host
|
||||
- [[deployment]] — service-scoped git deploy
|
||||
|
||||
## Citations
|
||||
|
||||
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
|
||||
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
|
||||
- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
|
||||
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
|
||||
- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
|
||||
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
|
||||
`--split-mode graph` MoE note (PR 1080)
|
||||
+13
-11
@@ -1,7 +1,7 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
@@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template.
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[upstream llama.cpp b10121
|
||||
B2[ik_llama.cpp build b166e269
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
ngram-mod, selected experts offloaded via override, 491520 ctx]
|
||||
A --> B --> C
|
||||
ngram-mod, q8_KV + Hadamard cache, 491520 ctx]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
I[LLM Clients / API] -->|HTTP :7770| B
|
||||
I -->|HTTP :7770| E
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
@@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_KV + Hadamard KV cache,
|
||||
split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS
|
||||
build flags
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
+2
-1
@@ -110,9 +110,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_KV + Hadamard KV cache, split-mode graph; 2026-08-26)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
||||
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
||||
|
||||
|
||||
Reference in New Issue
Block a user