dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Remove KV cache quantization, increase batch/ubatch on frank-default-large

Browse Source 
- Remove CACHE_TYPE_K/CACHE_TYPE_V (fp16 default on GPU)
- Increase BATCH_SIZE from 2048 to 8192
- Increase UBATCH_SIZE from 512 to 2048
- Remove --cache-type-k and --cache-type-v from llama-server flags
This commit is contained in:
Frank
2026-08-13 16:50:09 -07:00
parent db3eb01495
commit c984a3e4e6
2 changed files with 7 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+5 -13
View File
@@ -20,19 +20,11 @@ SPEC_NGRAM_MOD_N_MATCH=24
DEVICE=ROCm0 DEVICE=ROCm0
# Dense model (Qwen3.6-27B, not MoE): no MoE CPU offload. Full offload. # Dense model (Qwen3.6-27B, not MoE): no MoE CPU offload. Full offload.
N_GPU_LAYERS=all N_GPU_LAYERS=all
# KV on GPU, quantized (K q8_0, V q8_0). V MUST match K: this llama.cpp # KV on GPU, fp16 (unquantized). Default ggml_type for both K and V.
# build has GGML_CUDA_FA_ALL_QUANTS=OFF, and Flash Attention requires # Batch sizing: fp16 KV is larger than q8_0, but user has verified this
# K type == V type. Mixed q8_0/f16 silently disables FA, collapsing # configuration fits. See frank-default-large-benchmark.md.
# long-context decode to ~1 t/s at 200k ctx. q8_0 V halves V memory too. BATCH_SIZE=8192
CACHE_TYPE_K=q8_0 UBATCH_SIZE=2048
CACHE_TYPE_V=q8_0
# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700):
# KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute
# graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even
# at batch 2048. Proven-fits config: batch 2048 / ubatch 512.
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
BATCH_SIZE=2048
UBATCH_SIZE=512
# Sampling settings — per HauhauCS Qwen3.6-27B model card. # Sampling settings — per HauhauCS Qwen3.6-27B model card.
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0. # Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
# Clients may override per-request; these are the server-side defaults. # Clients may override per-request; these are the server-side defaults.
scripts/serve-llama-frank-large.sh
+2 -6
View File
@@ -33,10 +33,8 @@ fi
: "${SPEC_NGRAM_MOD_N_MATCH:=24}" : "${SPEC_NGRAM_MOD_N_MATCH:=24}"
: "${DEVICE:=ROCm0}" : "${DEVICE:=ROCm0}"
: "${N_GPU_LAYERS:=all}" : "${N_GPU_LAYERS:=all}"
: "${CACHE_TYPE_K:=q8_0}" : "${BATCH_SIZE:=8192}"
: "${CACHE_TYPE_V:=q8_0}" : "${UBATCH_SIZE:=2048}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${TEMP:=1.0}" : "${TEMP:=1.0}"
: "${TOP_K:=20}" : "${TOP_K:=20}"
: "${TOP_P:=0.95}" : "${TOP_P:=0.95}"
@@ -66,8 +64,6 @@ exec "$LLAMA_SERVER" \
--device "$DEVICE" \ --device "$DEVICE" \
--gpu-layers "$N_GPU_LAYERS" \ --gpu-layers "$N_GPU_LAYERS" \
--fit off \ --fit off \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--ctx-size "$CTX_SIZE" \ --ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \ --parallel "$PARALLEL" \
--batch-size "$BATCH_SIZE" \ --batch-size "$BATCH_SIZE" \