frank-small: revert to bf16 KV — the only dtype that boots Gemma4

int8_per_token_head: page-size unification fails (scale tensor breaks
the 256/512 head-dim ratio). turboquant_k8v4: backend validation fails
(no mm-prefix attention, which Gemma4's vision/audio path requires).
fp8: hardware-dead on sm_86. bf16 it is; history documented in
frank-small.env, seam pins auto + no backend pin.
This commit is contained in:
Frank committed 2026-09-19 00:26:11 -07:00
1 parent 7e6ef1ac99
commit bab8f7f839
3 files changed
+24 -27

No files matched your search

-2
View File
@@ -44,7 +44,6 @@ fi
: "${MAX_NUM_SEQS:=16}"
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
: "${SPEC_NUM_TOKENS:=4}"
: "${ATTENTION_BACKEND:=TURBOQUANT}"
: "${ENFORCE_EAGER:=0}"
# Fail loudly rather than half-loading a model into VRAM: a missing file here
@@ -67,7 +66,6 @@ VLLM_FLAGS=(
--kv-cache-dtype "$KV_CACHE_DTYPE"
--max-num-seqs "$MAX_NUM_SEQS"
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
--attention-backend "$ATTENTION_BACKEND"
--speculative-config "$SPEC_CONFIG"
--reasoning-parser gemma4
--enable-auto-tool-choice