frank-default-small: KV K default f16 (no hadamard), V q6_0 + hadamard
K cache back to default f16 full precision with K_CACHE_HADAMARD=0; V cache at q6_0 with V_CACHE_HADAMARD=1. Env, serve script, bats, and KB updated.
This commit is contained in:
1 parent
8f860d24a6
commit
24c5eb4e28
4 files changed
+35
-32
No files matched your search
@@ -8,9 +8,10 @@
|
||||
# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
|
||||
# is disabled (ik_llama limits spec to a single slot).
|
||||
#
|
||||
# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
|
||||
# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
|
||||
# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
|
||||
# KV cache: K at the default f16 (no Hadamard), V at q6_0 with Hadamard.
|
||||
# (requires a build with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS
|
||||
# =ON; see the env file). q8_KV crashes at init on this arch (ggml.c:5427);
|
||||
# V uses q6_0 instead.#
|
||||
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
|
||||
# split-mode graph with fewer than 2 devices (hard load error).
|
||||
|
||||
@@ -52,10 +53,10 @@ fi
|
||||
: "${CTX_SIZE:=491520}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${SPEC_TYPE:=none}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q5_0}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=q6_0}"
|
||||
: "${K_CACHE_HADAMARD:=0}"
|
||||
: "${V_CACHE_HADAMARD:=0}"
|
||||
: "${V_CACHE_HADAMARD:=1}"
|
||||
: "${SPLIT_MODE:=none}"
|
||||
: "${VERBOSITY:=4}"
|
||||
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
|
||||
|
||||
Reference in new issue
Block a user