dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: int8 KV (per-token, per-head) — doubles the pool

Browse Source 
fp8 KV is hardware-dead on the 3090 (sm_86, no cvt.e4m3x2), so int8 is
the cheapest quant the TRITON_ATTN backend can run: pure ALU dequant,
same backend, no fp8 tensor cores needed. 289,647 -> ~579k-token pool,
two -> four full-context sessions. Quality verified against the bf16
baseline (needle-in-haystack at 213k + short battery) before the swap.
This commit is contained in:
Frank committed 2026-09-18 23:24:37 -07:00
1 parent ae137317ab
commit 8d88b3a0a9
2 files changed
+24 -16
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-small.env
+17 -11
View File
@@ -29,15 +29,21 @@
# for as many concurrent sessions as the card holds. # for as many concurrent sessions as the card holds.
# MAX_MODEL_LEN=262144 the model's max_position_embeddings # MAX_MODEL_LEN=262144 the model's max_position_embeddings
# GPU_MEM_UTIL=0.95 the card is dedicated to this service # GPU_MEM_UTIL=0.95 the card is dedicated to this service
# KV_CACHE_DTYPE=auto bf16 KV. fp8_e4m3 would halve the KV charge # KV_CACHE_DTYPE=int8_per_token_head int8 KV (per-token, per-head scales)
# (289,756 -> 623,666-token pool, one -> two # in the same TRITON_ATTN backend — pure ALU
# full-context sessions), but the 3090 is # dequant, no fp8 hardware needed, so it runs
# sm_86 (Ampere) and has no native fp8: the # on sm_86. fp8_e4m3 is the cheaper option but
# TRITON_ATTN backend Gemma4 forces emits # the 3090 is sm_86 (Ampere) and has no native
# cvt.e4m3x2 instructions that exist only from # fp8: the TRITON_ATTN backend Gemma4 forces
# sm_89 (Ada), so fp8 KV cannot boot on this # emits cvt.e4m3x2 instructions that exist only
# card (measured 2026-09-19; the "sm_89 # from sm_89 (Ada), so fp8 KV cannot boot on
# patch" experiment is in .AGENTTRASH). # this card (measured 2026-09-19; the "sm_89
# patch" experiment is in .AGENTTRASH). int8
# halves the KV charge (289,647 -> ~579k-token
# pool, two -> four full-context sessions) at
# the cost of a dequant in the attention path;
# quality verified against the bf16 baseline
# (needle + short battery) on 2026-09-19.
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this, # MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
# caps full-context concurrency # caps full-context concurrency
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill # MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
@@ -71,11 +77,11 @@ ALIAS=frank-small
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=8000 LISTEN_PORT=8000
# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling, # Full window, dedicated card, int8 KV (see above), roomy batch ceiling,
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers). # 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
MAX_MODEL_LEN=262144 MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95 GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=auto KV_CACHE_DTYPE=int8_per_token_head
MAX_NUM_SEQS=16 MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384 MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4 SPEC_NUM_TOKENS=4
tests/frank_small_unit.bats
+7 -5
View File
@@ -98,7 +98,7 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE" ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
} }
@test "frank-small serves the full window with bf16 KV on a dedicated card" { @test "frank-small serves the full window with int8 KV on a dedicated card" {
local maxlen kvdtype util seqs local maxlen kvdtype util seqs
maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")" maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")"
kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")" kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
@@ -106,10 +106,12 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")" seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")"
# The model's FULL trained window (262144), not a divided one. # The model's FULL trained window (262144), not a divided one.
[ "$maxlen" = "262144" ] [ "$maxlen" = "262144" ]
# bf16 KV: fp8_e4m3 would halve the pool charge, but the 3090 is # int8 KV (per-token, per-head scales): fp8_e4m3 would halve the pool
# sm_86 (Ampere) — no native fp8, and the TRITON_ATTN backend that # charge further, but the 3090 is sm_86 (Ampere) — no native fp8, and
# Gemma4 forces emits cvt.e4m3x2 instructions that exist only from sm_89. # the TRITON_ATTN backend that Gemma4 forces emits cvt.e4m3x2
[ "$kvdtype" = "auto" ] # instructions that exist only from sm_89. int8 is pure ALU dequant in
# the same backend, so it runs.
[ "$kvdtype" = "int8_per_token_head" ]
# The 3090 is dedicated to this service, so the engine may take most of # The 3090 is dedicated to this service, so the engine may take most of
# it; the KV pool, not the seq ceiling, caps full-context concurrency. # it; the KV pool, not the seq ceiling, caps full-context concurrency.
[ "$util" = "0.95" ] [ "$util" = "0.95" ]