dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: KV K+V to f16, add verbosity tunable (F16 KV diag)

Browse Source 
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
This commit is contained in:
Frank committed 2026-08-14 12:50:59 -07:00
1 parent bea52138d2
commit f70460b002
2 files changed
+12 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-small.env
+9 -2
View File
@@ -21,9 +21,16 @@ NGRAM_MOD_N_MATCH=24
# decode. Fixes the cuBLAS OOM crash class during big prefills. # decode. Fixes the cuBLAS OOM crash class during big prefills.
N_CPU_MOE=36 N_CPU_MOE=36
N_GPU_LAYERS=all N_GPU_LAYERS=all
# KV cache on GPU, K quantized to q8_0, V stays f16 (user-locked decision). # KV cache on GPU, both K and V at f16 (2026-08-14 diagnostic).
CACHE_TYPE_K=q8_0 # F16 K removes the flash-attention F16-conversion workspace (the MMA_F16
# prefill kernel needs no copy when K is already f16), trading static pool
# size for the context-proportional workspace growth that caused the
# cublasSgemm_v2 crashes. Run with VERBOSITY=4 so the load log shows the
# real allocation sizes; refit CTX_SIZE if it does not fit.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16 CACHE_TYPE_V=f16
# Diagnostic verbosity for this run; logs the KV/graph allocation sizes.
VERBOSITY=4
# Store SWA checkpoints in RAM to hold peak GPU memory down for the # Store SWA checkpoints in RAM to hold peak GPU memory down for the
# larger batch/ubatch (leaves headroom for ctx-checkpoints operation). # larger batch/ubatch (leaves headroom for ctx-checkpoints operation).
CTX_CHECKPOINTS=4 CTX_CHECKPOINTS=4
scripts/serve-llama-frank-small.sh
+3 -1
View File
@@ -30,8 +30,9 @@ fi
: "${NGRAM_MOD_N_MIN:=48}" : "${NGRAM_MOD_N_MIN:=48}"
: "${NGRAM_MOD_N_MAX:=64}" : "${NGRAM_MOD_N_MAX:=64}"
: "${NGRAM_MOD_N_MATCH:=24}" : "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}" : "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}" : "${CACHE_TYPE_V:=f16}"
: "${VERBOSITY:=3}"
: "${N_CPU_MOE:=36}" # 36 of 40 layers: MoE expert weights stay on CPU; 4 on GPU : "${N_CPU_MOE:=36}" # 36 of 40 layers: MoE expert weights stay on CPU; 4 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU : "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}" : "${BATCH_SIZE:=2048}"
@@ -66,6 +67,7 @@ exec "$LLAMA_SERVER" \
--spec-ngram-mod-n-match "$NGRAM_MOD_N_MATCH" \ --spec-ngram-mod-n-match "$NGRAM_MOD_N_MATCH" \
--cache-type-k "$CACHE_TYPE_K" \ --cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \ --cache-type-v "$CACHE_TYPE_V" \
--verbosity "$VERBOSITY" \
--ctx-size "$CTX_SIZE" \ --ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \ --parallel "$PARALLEL" \
--n-cpu-moe "$N_CPU_MOE" \ --n-cpu-moe "$N_CPU_MOE" \