frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots triple the KV, so expert blocks 27-34 are pinned back to CPU (freed ~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model, split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md: drop the scratch-port rule (test/bench processes cannot share the single GPU with the running service anyway). Env, serve script, bats, docs updated.
This commit is contained in:
1 parent
8c41c821c4
commit
04a8fb7806
5 files changed
+84
-68
No files matched your search
@@ -5,11 +5,12 @@
|
||||
#
|
||||
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
|
||||
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
|
||||
# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
|
||||
# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
|
||||
# is disabled (ik_llama limits spec to a single slot).
|
||||
#
|
||||
# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
|
||||
# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
|
||||
# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
|
||||
# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
|
||||
# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
|
||||
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
|
||||
# split-mode graph with fewer than 2 devices (hard load error).
|
||||
|
||||
@@ -49,10 +50,10 @@ fi
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=160000}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${SPEC_TYPE:=none}"
|
||||
: "${CACHE_TYPE_K:=q6_0}"
|
||||
: "${CACHE_TYPE_V:=q6_0}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q5_0}"
|
||||
: "${K_CACHE_HADAMARD:=0}"
|
||||
: "${V_CACHE_HADAMARD:=0}"
|
||||
: "${SPLIT_MODE:=none}"
|
||||
|
||||
Reference in new issue
Block a user