dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: pin VLLM_ATTENTION_BACKEND=TURBOQUANT

Browse Source 
Gemma4's config hook force-selects TRITON_ATTN for the mixed 256/512
head dims whenever no backend is selected, and TRITON_ATTN rejects the
turboquant KV dtype (measured: 'kv_cache_dtype not supported', crash
loop). The TURBOQUANT backend accepts any head size and the
turboquant_* dtypes, and the Gemma4 MTP speculator carries the
target's selected backend through to the draft layers.
This commit is contained in:
Frank
2026-09-19 00:07:57 -07:00
parent 5e581666f2
commit 966b70db1a
2 changed files with 13 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-small.env
+8
View File
@@ -85,6 +85,14 @@ LISTEN_PORT=8000
MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=turboquant_k8v4
# Gemma4's config hook force-selects TRITON_ATTN for the mixed 256/512 head
# dims, but only when no backend is selected — and TRITON_ATTN rejects the
# turboquant KV dtype (measured: ValueError "kv_cache_dtype not supported",
# crash loop). Pin the TURBOQUANT backend explicitly: it accepts any head
# size, the turboquant dtypes, and has the Ampere path, and the Gemma4 MTP
# speculator carries the target's selected backend through to the draft
# layers, so the drafter follows.
VLLM_ATTENTION_BACKEND=TURBOQUANT
MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4