fix(frank-default-large): enable flash attention by pairing V cache to q8_0

Root cause of ~1 t/s long-context decode: llama.cpp build has
GGML_CUDA_FA_ALL_QUANTS=OFF, and FA requires K type == V type.
Mixed cache q8_0/f16 silently disabled FA, collapsing decode at
200k ctx. Set V to q8_0 (halves V memory too) and retune
threads 12/1 for 16 Zen5 cores.
This commit is contained in:
Frank committed 2026-08-12 02:09:05 -07:00
1 parent f20d6ce288
commit 334b6af2ae
2 files changed
+10 -7

No files matched your search

+3 -3
View File
@@ -34,11 +34,11 @@ fi
: "${DEVICE:=ROCm0}"
: "${N_GPU_LAYERS:=all}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${THREADS:=8}"
: "${THREADS_HTTP:=8}"
: "${THREADS:=12}"
: "${THREADS_HTTP:=1}"
LLAMA_SERVER="${LLAMA_SERVER:-/home/dank/llama.cpp/build/bin/llama-server}"
MODEL="$MODEL_DIR/$MODEL_FILE"