dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(frank-default-large): enable flash attention by pairing V cache to q8_0

Browse Source 
Root cause of ~1 t/s long-context decode: llama.cpp build has
GGML_CUDA_FA_ALL_QUANTS=OFF, and FA requires K type == V type.
Mixed cache q8_0/f16 silently disabled FA, collapsing decode at
200k ctx. Set V to q8_0 (halves V memory too) and retune
threads 12/1 for 16 Zen5 cores.
This commit is contained in:
Frank
2026-08-12 02:09:05 -07:00
parent f20d6ce288
commit 334b6af2ae
2 changed files with 10 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+7 -4
View File
@@ -20,9 +20,12 @@ SPEC_NGRAM_MOD_N_MATCH=24
DEVICE=ROCm0 DEVICE=ROCm0
# Dense model (Qwen3.6-27B, not MoE): no MoE CPU offload. Full offload. # Dense model (Qwen3.6-27B, not MoE): no MoE CPU offload. Full offload.
N_GPU_LAYERS=all N_GPU_LAYERS=all
# KV on GPU, quantized (K q8_0, V f16). # KV on GPU, quantized (K q8_0, V q8_0). V MUST match K: this llama.cpp
# build has GGML_CUDA_FA_ALL_QUANTS=OFF, and Flash Attention requires
# K type == V type. Mixed q8_0/f16 silently disables FA, collapsing
# long-context decode to ~1 t/s at 200k ctx. q8_0 V halves V memory too.
CACHE_TYPE_K=q8_0 CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16 CACHE_TYPE_V=q8_0
# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700): # Batch sizing constrained by ROCm memory at 602100 ctx (3x200700):
# KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute # KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute
# graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even # graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even
@@ -30,5 +33,5 @@ CACHE_TYPE_V=f16
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench). # User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
BATCH_SIZE=2048 BATCH_SIZE=2048
UBATCH_SIZE=512 UBATCH_SIZE=512
THREADS=8 THREADS=12
THREADS_HTTP=8 THREADS_HTTP=1
scripts/serve-llama-frank-large.sh
+3 -3
View File
@@ -34,11 +34,11 @@ fi
: "${DEVICE:=ROCm0}" : "${DEVICE:=ROCm0}"
: "${N_GPU_LAYERS:=all}" : "${N_GPU_LAYERS:=all}"
: "${CACHE_TYPE_K:=q8_0}" : "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}" : "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}" : "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}" : "${UBATCH_SIZE:=512}"
: "${THREADS:=8}" : "${THREADS:=12}"
: "${THREADS_HTTP:=8}" : "${THREADS_HTTP:=1}"
LLAMA_SERVER="${LLAMA_SERVER:-/home/dank/llama.cpp/build/bin/llama-server}" LLAMA_SERVER="${LLAMA_SERVER:-/home/dank/llama.cpp/build/bin/llama-server}"
MODEL="$MODEL_DIR/$MODEL_FILE" MODEL="$MODEL_DIR/$MODEL_FILE"