frank-small: revert to bf16 KV — the only dtype that boots Gemma4

int8_per_token_head: page-size unification fails (scale tensor breaks
the 256/512 head-dim ratio). turboquant_k8v4: backend validation fails
(no mm-prefix attention, which Gemma4's vision/audio path requires).
fp8: hardware-dead on sm_86. bf16 it is; history documented in
frank-small.env, seam pins auto + no backend pin.
This commit is contained in:
Frank committed 2026-09-19 00:26:11 -07:00
1 parent 7e6ef1ac99
commit bab8f7f839
3 files changed
+24 -27

No files matched your search

+12 -11
View File
@@ -84,17 +84,18 @@ LISTEN_PORT=8000
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=turboquant_k8v4
# ATTENTION_BACKEND: Gemma4's config hook force-selects TRITON_ATTN for the
# mixed 256/512 head dims whenever no backend is selected, and TRITON_ATTN
# rejects the turboquant KV dtype (measured: ValueError "kv_cache_dtype not
# supported", crash loop). Pin the TURBOQUANT backend explicitly via the
# --attention-backend flag (this vLLM build removed the VLLM_ATTENTION_
# BACKEND env var — it warns "Unknown vLLM environment variable" and
# ignores it). The TURBOQUANT backend accepts any head size and the
# turboquant_* dtypes, and the Gemma4 MTP speculator carries the target's
# selected backend through to the draft layers, so the drafter follows.
ATTENTION_BACKEND=TURBOQUANT
KV_CACHE_DTYPE=auto
# KV quant history (measured 2026-09-19, all on this image): bf16 (auto)
# is the ONLY dtype that boots Gemma4 here. int8_per_token_head dies in
# unify_kv_cache_spec_page_size — its separate fp32 scale tensor makes the
# per-token page bytes 4160 (SWA, head_dim 256) vs 8256 (full, head_dim
# 512), not an integer ratio. turboquant_k8v4 dies in backend validation —
# the TURBOQUANT backend rejects "partial multimodal token full attention"
# (the mm-prefix path Gemma4's vision/audio tokens require), and the
# VLLM_ATTENTION_BACKEND env var does not exist in this build (the pin
# goes through --attention-backend). fp8 is hardware-dead on sm_86
# (cvt.e4m3x2 is sm_89+). Revisit on a vLLM release that documents TurboQuant
# mm-prefix support.
MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4