dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: --attention-backend flag (env var removed in this build)

Browse Source 
The VLLM_ATTENTION_BACKEND env var does not exist in this vLLM dev
build (warns 'Unknown vLLM environment variable' and is ignored), so
the pin landed nowhere and Gemma4's hook force-selected TRITON_ATTN
again. Pass --attention-backend TURBOQUANT as a CLI flag instead.
This commit is contained in:
Frank
2026-09-19 00:17:41 -07:00
parent 966b70db1a
commit 7e6ef1ac99
3 changed files with 18 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-small.env
+10 -8
View File
@@ -85,14 +85,16 @@ LISTEN_PORT=8000
MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=turboquant_k8v4
# Gemma4's config hook force-selects TRITON_ATTN for the mixed 256/512 head
# dims, but only when no backend is selected — and TRITON_ATTN rejects the
# turboquant KV dtype (measured: ValueError "kv_cache_dtype not supported",
# crash loop). Pin the TURBOQUANT backend explicitly: it accepts any head
# size, the turboquant dtypes, and has the Ampere path, and the Gemma4 MTP
# speculator carries the target's selected backend through to the draft
# layers, so the drafter follows.
VLLM_ATTENTION_BACKEND=TURBOQUANT
# ATTENTION_BACKEND: Gemma4's config hook force-selects TRITON_ATTN for the
# mixed 256/512 head dims whenever no backend is selected, and TRITON_ATTN
# rejects the turboquant KV dtype (measured: ValueError "kv_cache_dtype not
# supported", crash loop). Pin the TURBOQUANT backend explicitly via the
# --attention-backend flag (this vLLM build removed the VLLM_ATTENTION_
# BACKEND env var — it warns "Unknown vLLM environment variable" and
# ignores it). The TURBOQUANT backend accepts any head size and the
# turboquant_* dtypes, and the Gemma4 MTP speculator carries the target's
# selected backend through to the draft layers, so the drafter follows.
ATTENTION_BACKEND=TURBOQUANT
MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4
scripts/serve-frank-small.sh
+2
View File
@@ -44,6 +44,7 @@ fi
: "${MAX_NUM_SEQS:=16}"
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
: "${SPEC_NUM_TOKENS:=4}"
: "${ATTENTION_BACKEND:=TURBOQUANT}"
: "${ENFORCE_EAGER:=0}"
# Fail loudly rather than half-loading a model into VRAM: a missing file here
@@ -66,6 +67,7 @@ VLLM_FLAGS=(
--kv-cache-dtype "$KV_CACHE_DTYPE"
--max-num-seqs "$MAX_NUM_SEQS"
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
--attention-backend "$ATTENTION_BACKEND"
--speculative-config "$SPEC_CONFIG"
--reasoning-parser gemma4
--enable-auto-tool-choice