fix(frank-default-large): enable flash attention by pairing V cache to q8_0
Root cause of ~1 t/s long-context decode: llama.cpp build has GGML_CUDA_FA_ALL_QUANTS=OFF, and FA requires K type == V type. Mixed cache q8_0/f16 silently disabled FA, collapsing decode at 200k ctx. Set V to q8_0 (halves V memory too) and retune threads 12/1 for 16 Zen5 cores.
This commit is contained in:
1 parent
f20d6ce288
commit
334b6af2ae
2 files changed
+10
-7
No files matched your search
@@ -34,11 +34,11 @@ fi
|
||||
: "${DEVICE:=ROCm0}"
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${THREADS:=8}"
|
||||
: "${THREADS_HTTP:=8}"
|
||||
: "${THREADS:=12}"
|
||||
: "${THREADS_HTTP:=1}"
|
||||
|
||||
LLAMA_SERVER="${LLAMA_SERVER:-/home/dank/llama.cpp/build/bin/llama-server}"
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
|
||||
Reference in new issue
Block a user