frank-small: --attention-backend flag (env var removed in this build)
The VLLM_ATTENTION_BACKEND env var does not exist in this vLLM dev build (warns 'Unknown vLLM environment variable' and is ignored), so the pin landed nowhere and Gemma4's hook force-selected TRITON_ATTN again. Pass --attention-backend TURBOQUANT as a CLI flag instead.
This commit is contained in:
1 parent
966b70db1a
commit
7e6ef1ac99
3 files changed
+18
-12
No files matched your search
@@ -44,6 +44,7 @@ fi
|
||||
: "${MAX_NUM_SEQS:=16}"
|
||||
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
|
||||
: "${SPEC_NUM_TOKENS:=4}"
|
||||
: "${ATTENTION_BACKEND:=TURBOQUANT}"
|
||||
: "${ENFORCE_EAGER:=0}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
@@ -66,6 +67,7 @@ VLLM_FLAGS=(
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--attention-backend "$ATTENTION_BACKEND"
|
||||
--speculative-config "$SPEC_CONFIG"
|
||||
--reasoning-parser gemma4
|
||||
--enable-auto-tool-choice
|
||||
|
||||
Reference in new issue
Block a user