From 0e5c37f2f8bd403e534cdb687f9bca644b80ed0f Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 26 Aug 2026 03:23:18 -0700 Subject: [PATCH] frank-default-small: use q8_0 (not q8_KV) KV cache on ik_llama The ik_llama fast q8_KV cache type crashes at model init on this QWEN3MOE/MTP + multimodal arch at commit b166e269 with a ggml_view size assert (ggml.c:5427) right after the output buffer is created, looping the service. Empirically isolated by toggling cache type: q8_0 and f16 load and serve (smoke-tested correct output), q8_KV asserts. Switch to q8_0 + Hadamard (K/V), which still gives the 8-bit cache + Hadamard quality win. KB updated with the crash lesson. --- docs/LLM/ikllama-ornith-dankserver.md | 18 ++++++++++++++---- docs/LLM/index.md | 4 ++-- docs/index.md | 2 +- scripts/frank-default-small.env | 20 +++++++++++--------- scripts/serve-frank-small.sh | 5 ++++- 5 files changed, 32 insertions(+), 17 deletions(-) diff --git a/docs/LLM/ikllama-ornith-dankserver.md b/docs/LLM/ikllama-ornith-dankserver.md index 3323f03..10de131 100644 --- a/docs/LLM/ikllama-ornith-dankserver.md +++ b/docs/LLM/ikllama-ornith-dankserver.md @@ -1,7 +1,7 @@ --- type: reference title: ik_llama.cpp on Dankserver (Ornith Q6_K) -description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. tags: - ikllama - llamacpp @@ -104,14 +104,24 @@ Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`, `scripts/frank-default-small.service` in the repo. The service just calls the serve script; the env file holds all tunables. -### KV cache: q8_KV + Hadamard +### KV cache: q8_0 + Hadamard -- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik - PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`. +- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` — standard 8-bit KV cache type + (passes `--cache-type-k q8_0 --cache-type-v q8_0`). - `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and `--v-cache-hadamard`. The Hadamard transform preserves KV quality at 8-bit (docs recommend it for quantized KV below Q6_0). Implemented for both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. +- Flash-attention is **on by default** (no `-fa` flag needed). + +> ⚠️ **Do not use `q8_KV` (fast type) here.** On this commit (`b166e269`) +> the fast `q8_KV` cache type crashes at model init on this QWEN3MOE/MTP + +> multimodal arch with a `ggml view` size assertion +> (`ggml.c:5427: GGML_ASSERT(view_src == NULL || ... data_size + view_offs`) +> right after the output buffer is created. It loops the service. Plain +> `q8_0` (and `f16`) work and were smoke-tested with correct output; this +> config uses `q8_0`. This was isolated empirically by toggling cache type +> while keeping the override/hadamard flags. These keep the 163840-slot / 491520-pool context while roughly halving KV cache RAM vs f16. diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 3ab7e4e..6c70cdd 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -20,7 +20,7 @@ graph LR B2[ik_llama.cpp build b166e269 frank-default-small, port 7770] C[Ornith-1.5-35B-A3B-CRACK Q6_K -ngram-mod, q8_KV + Hadamard cache, 491520 ctx] +ngram-mod, q8_0 + Hadamard cache, 491520 ctx] A --> B2 --> C end @@ -57,7 +57,7 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot] - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) - [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for - frank-default-small (2026-08-26): q8_KV + Hadamard KV cache, + frank-default-small (2026-08-26): q8_0 + Hadamard KV cache, split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment) diff --git a/docs/index.md b/docs/index.md index e6e511c..43eeb81 100644 --- a/docs/index.md +++ b/docs/index.md @@ -110,7 +110,7 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) -- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_KV + Hadamard KV cache, split-mode none; 2026-08-26) +- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21) - [[Dankserver System Specs]] — the RTX 3090 box hardware diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 2a31ba8..bc2af8a 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -5,12 +5,12 @@ # 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB # RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights # pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe). -# KV cache quantized to q8_KV with Hadamard transforms on both K and V. -# --split-mode graph enabled (single GPU; retained per operator preference). +# KV cache quantized to q8_0 with Hadamard transforms on both K and V. +# Split mode none (single GPU; graph would require 2+ devices). # # Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must # ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON -# so the q8_KV + Hadamard KV cache combos compile into the CUDA/CPU FA +# so the quantized + Hadamard KV cache combos compile into the CUDA/CPU FA # kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24. # Switched from CRACK Q8_0 to Q6_K on 2026-08-25. # @@ -67,12 +67,14 @@ OVERRIDE_BUFT=CPU # OVERRIDE_EXPERT below. N_GPU_LAYERS=999 -# KV cache on GPU, both K and V at q8_KV (fast quant type on ik_llama, PR 208) -# plus Hadamard transforms on both caches to preserve quality at 8-bit. -# For quantized KV below Q6_0 the docs recommend --k-cache-hadamard; we apply -# --v-cache-hadamard to V as well. -CACHE_TYPE_K=q8_KV -CACHE_TYPE_V=q8_KV +# KV cache on GPU, both K and V at q8_0 plus Hadamard transforms on both +# caches to preserve quality at 8-bit. NOTE: ik_llama's fast q8_KV type +# crashes at model init on this MTP/multimodal arch with a ggml_view size +# assert (ggml.c:5427) at commit b166e269; plain q8_0 works. Docs recommend +# --k-cache-hadamard for quantized KV below Q6_0; we apply --v-cache-hadamard +# to V as well. Flash-attention is on by default. +CACHE_TYPE_K=q8_0 +CACHE_TYPE_V=q8_0 K_CACHE_HADAMARD=1 V_CACHE_HADAMARD=1 diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 6ee3670..98ef9f6 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -7,11 +7,14 @@ # 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; # this service uses ngram-mod instead of MTP speculation. # -# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with +# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with # GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file). +# The ik_llama fast q8_KV type crashes at init on this arch (ggml.c:5427); +# q8_0 is used instead. # # Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects # split-mode graph with fewer than 2 devices (hard load error). + # # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded # --override-tensor pattern are pinned to CPU. Selected full-attention and