frank-small: four FULL 262144-token slots — the max that fits the card

CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained
window. Measured fit arithmetic on the 3090: 1 full-context slot idles
at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed.
A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool
allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit
on purpose: fit maximises the TOTAL pool and would divide a smaller
window across slots at parallel>1. The slot-grid seam now pins both the
full-window division and the budget inequality so a future capacity
bump fails pre-deploy, not in the boot allocation.

Audio verified while measuring: the mmproj's audio encoder transcribes
input_audio parts verbatim (JFK WAV sample). Header comments updated.
This commit is contained in:
Frank committed 2026-09-17 02:41:11 -07:00
1 parent 87caca4894
commit dbd764a493
3 files changed
+44 -28

No files matched your search

+17 -11
View File
@@ -61,22 +61,28 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
}
@test "frank-small quantizes KV to q8 and lets --fit size the context" {
local parallel cache_k cache_v ctx
@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
local parallel cache_k cache_v ctx fixed per_slot
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
[ "$parallel" -ge 1 ]
# q8_0 KV is what fits the 24 GiB card beside 7.4 GiB of Q4_K_M weights +
# mmproj + sidecar: Gemma4's 16 kv heads x 512 dim cost 16 KiB per token
# per layer at q8, and the model's sliding-window pattern keeps the pool
# far below full-ctx x all-layers. The engine itself sizes the context:
# CTX_SIZE is unset, so the serve script passes --fit on.
[ -z "$ctx" ]
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
# Every slot must be the model's FULL trained window (262144), not a
# divided one -- CTX_SIZE is the total pool and --parallel divides it.
[ $((ctx / parallel)) -eq 262144 ]
[ $((ctx % parallel)) -eq 0 ]
[ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
grep -q -- '--fit on' "$SERVE"
# Fit budget from the measured boot arithmetic on the 3090 (24576
# MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
# so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
# (weights, projector, sidecar draft, compute buffers, CUDA context)
# is 11996-3450 = 8546 MiB. If the config asks for more slots than
# the card holds, this inequality fails before deploy instead of the
# KV allocation failing at boot.
fixed=8546
per_slot=3450
[ $((fixed + parallel * per_slot)) -le 24576 ]
# Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
# without which prefill collapses about 12x (see the CUDA lesson in
# AGENTS.md). The build-cuda tree must carry it.