dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-small: KV K default f16 (no hadamard), V q6_0 + hadamard

Browse Source 
K cache back to default f16 full precision with K_CACHE_HADAMARD=0;
V cache at q6_0 with V_CACHE_HADAMARD=1. Env, serve script, bats, and
KB updated.
This commit is contained in:
Frank
2026-08-26 05:53:01 -07:00
parent 8f860d24a6
commit 24c5eb4e28
4 changed files with 35 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/ikllama-ornith-dankserver.md
+17 -15
View File
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K q8_0 / V q5_0 with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 160k ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -105,16 +105,17 @@ Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
`scripts/frank-default-small.service` in the repo. The service just calls
the serve script; the env file holds all tunables.
### KV cache: K q8_0 / V q5_0 + Hadamard
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q5_0` — asymmetric KV precision: the
K cache at 8-bit, the V cache at 5-bit (passes `--cache-type-k q8_0
--cache-type-v q5_0`). Keeping K higher-precision than V is a common
asymmetric budget split.
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
`--v-cache-hadamard`. The Hadamard transform preserves KV quality at
quantized precision. Implemented for both CUDA (`ggml-cuda/hadamard.cu`)
and CPU (IQK) paths. Flash-attention is **on by default**.
- `CACHE_TYPE_K=f16` — K cache at the default full-precision f16
(passes `--cache-type-k f16`). No Hadamard on K.
- `CACHE_TYPE_V=q6_0` — V cache at 6-bit k-quant
(passes `--cache-type-v q6_0`).
- `V_CACHE_HADAMARD=1` — `--v-cache-hadamard` on V only. The Hadamard
transform preserves V quality at 6-bit; K is full-precision f16 and does
not need it (`K_CACHE_HADAMARD=0`). Implemented for both CUDA
(`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. Flash-attention is **on
by default**.
> **KV cache type history on this build (`b166e269`):**
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
@@ -124,12 +125,13 @@ the serve script; the env file holds all tunables.
> - `q8_KV` **crashes** with a `ggml view` size assert
> (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
> QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
> - `q6_0` (both caches) boots and served, but with `PARALLEL=3` the KV
> tripled, so the K cache went back up to `q8_0` and V down to `q5_0` for
> the chosen K/V budget split.
> - The deployment settled on K at default **f16** (full precision, no
> Hadamard) with V at **q6_0 + Hadamard**.
With 3 parallel slots at 160k context, the KV cache is ~3x a single slot's.
The GPU-side expert set is reduced to fit (see the expert section).
With 3 parallel slots at 163840 context the KV cache is large; keeping K at
full precision f16 while quantizing only V (with Hadamard) is the chosen
budget split. The GPU-side expert set stays reduced to fit (see the expert
section).
### Split mode: none (single GPU)