frank-small: turboquant_k8v4 KV — int8 cannot boot this model
int8_per_token_head keeps a separate fp32 scale tensor per (token, head); Gemma4's hybrid head dims (SWA 256 / full 512) make the per-token page bytes 4160 vs 8256, not an integer ratio, so unify_kv_cache_spec_page_size raises NotImplementedError — nine crash-loop restarts measured. TurboQuant k8v4 packs its block scales inside the tensor, keeps the 2:1 page ratio, and has a dedicated backend with an explicit Ampere path. 289,647 -> ~772k-token pool.
This commit is contained in:
1 parent
8d88b3a0a9
commit
5e581666f2
2 files changed
+28
-24
No files matched your search
@@ -98,7 +98,7 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small serves the full window with int8 KV on a dedicated card" {
|
||||
@test "frank-small serves the full window with turboquant k8v4 KV on a dedicated card" {
|
||||
local maxlen kvdtype util seqs
|
||||
maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
@@ -106,12 +106,13 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
# The model's FULL trained window (262144), not a divided one.
|
||||
[ "$maxlen" = "262144" ]
|
||||
# int8 KV (per-token, per-head scales): fp8_e4m3 would halve the pool
|
||||
# charge further, but the 3090 is sm_86 (Ampere) — no native fp8, and
|
||||
# the TRITON_ATTN backend that Gemma4 forces emits cvt.e4m3x2
|
||||
# instructions that exist only from sm_89. int8 is pure ALU dequant in
|
||||
# the same backend, so it runs.
|
||||
[ "$kvdtype" = "int8_per_token_head" ]
|
||||
# TurboQuant k8v4 KV (K 8-bit, V 4-bit, in-tensor block scales,
|
||||
# dedicated TURBOQUANT backend with an Ampere path). The int8/fp8
|
||||
# per-token-head dtypes cannot boot this model: their separate fp32
|
||||
# scale tensor breaks the page-size ratio between Gemma4's SWA
|
||||
# (head_dim 256) and full-attention (head_dim 512) layers, and fp8 is
|
||||
# hardware-dead on sm_86 anyway.
|
||||
[ "$kvdtype" = "turboquant_k8v4" ]
|
||||
# The 3090 is dedicated to this service, so the engine may take most of
|
||||
# it; the KV pool, not the seq ceiling, caps full-context concurrency.
|
||||
[ "$util" = "0.95" ]
|
||||
|
||||
Reference in new issue
Block a user