frank-small: revert to bf16 KV — the only dtype that boots Gemma4
int8_per_token_head: page-size unification fails (scale tensor breaks the 256/512 head-dim ratio). turboquant_k8v4: backend validation fails (no mm-prefix attention, which Gemma4's vision/audio path requires). fp8: hardware-dead on sm_86. bf16 it is; history documented in frank-small.env, seam pins auto + no backend pin.
This commit is contained in:
1 parent
7e6ef1ac99
commit
bab8f7f839
3 files changed
+24
-27
No files matched your search
@@ -44,7 +44,6 @@ fi
|
||||
: "${MAX_NUM_SEQS:=16}"
|
||||
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
|
||||
: "${SPEC_NUM_TOKENS:=4}"
|
||||
: "${ATTENTION_BACKEND:=TURBOQUANT}"
|
||||
: "${ENFORCE_EAGER:=0}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
@@ -67,7 +66,6 @@ VLLM_FLAGS=(
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--attention-backend "$ATTENTION_BACKEND"
|
||||
--speculative-config "$SPEC_CONFIG"
|
||||
--reasoning-parser gemma4
|
||||
--enable-auto-tool-choice
|
||||
|
||||
Reference in new issue
Block a user