dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: KV K default f16 (no hadamard), V q6_0 + hadamard

Browse Source 
K cache back to default f16 full precision with K_CACHE_HADAMARD=0;
V cache at q6_0 with V_CACHE_HADAMARD=1. Env, serve script, bats, and
KB updated.
This commit is contained in:
Frank
2026-08-26 05:53:01 -07:00
parent 8f860d24a6
commit 24c5eb4e28
4 changed files with 35 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+6 -7
View File
@@ -73,13 +73,12 @@ OVERRIDE_BUFT=CPU
# OVERRIDE_EXPERT below.
N_GPU_LAYERS=999
# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with
# Hadamard transforms. The V cache at lower precision than K is a common
# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c
# :5427). Flash-attention is on by default.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q5_0
K_CACHE_HADAMARD=1
# KV cache on GPU: K back at the default f16 (no Hadamard on K; the K
# cache is kept full-precision so it does not need the transform), and V at
# q6_0 (6-bit k-quant) with Hadamard. Flash-attention is on by default.
CACHE_TYPE_K=f16
CACHE_TYPE_V=q6_0
K_CACHE_HADAMARD=0
V_CACHE_HADAMARD=1
# Split mode: none. Dankserver has a single RTX 3090 and ik_llama requires
scripts/serve-frank-small.sh
+7 -6
View File
@@ -8,9 +8,10 @@
# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
# is disabled (ik_llama limits spec to a single slot).
#
# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
# KV cache: K at the default f16 (no Hadamard), V at q6_0 with Hadamard.
# (requires a build with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS
# =ON; see the env file). q8_KV crashes at init on this arch (ggml.c:5427);
# V uses q6_0 instead.#
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
@@ -52,10 +53,10 @@ fi
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q5_0}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=q6_0}"
: "${K_CACHE_HADAMARD:=0}"
: "${V_CACHE_HADAMARD:=0}"
: "${V_CACHE_HADAMARD:=1}"
: "${SPLIT_MODE:=none}"
: "${VERBOSITY:=4}"
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern