frank-small: four FULL 262144-token slots — the max that fits the card
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained window. Measured fit arithmetic on the 3090: 1 full-context slot idles at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed. A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit on purpose: fit maximises the TOTAL pool and would divide a smaller window across slots at parallel>1. The slot-grid seam now pins both the full-window division and the budget inequality so a future capacity bump fails pre-deploy, not in the boot allocation. Audio verified while measuring: the mmproj's audio encoder transcribes input_audio parts verbatim (JFK WAV sample). Header comments updated.
This commit is contained in:
1 parent
87caca4894
commit
dbd764a493
3 files changed
+44
-28
No files matched your search
+17
-11
@@ -61,22 +61,28 @@ QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small quantizes KV to q8 and lets --fit size the context" {
|
||||
local parallel cache_k cache_v ctx
|
||||
@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
|
||||
local parallel cache_k cache_v ctx fixed per_slot
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
# q8_0 KV is what fits the 24 GiB card beside 7.4 GiB of Q4_K_M weights +
|
||||
# mmproj + sidecar: Gemma4's 16 kv heads x 512 dim cost 16 KiB per token
|
||||
# per layer at q8, and the model's sliding-window pattern keeps the pool
|
||||
# far below full-ctx x all-layers. The engine itself sizes the context:
|
||||
# CTX_SIZE is unset, so the serve script passes --fit on.
|
||||
[ -z "$ctx" ]
|
||||
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
|
||||
# Every slot must be the model's FULL trained window (262144), not a
|
||||
# divided one -- CTX_SIZE is the total pool and --parallel divides it.
|
||||
[ $((ctx / parallel)) -eq 262144 ]
|
||||
[ $((ctx % parallel)) -eq 0 ]
|
||||
[ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
|
||||
grep -q -- '--fit on' "$SERVE"
|
||||
# Fit budget from the measured boot arithmetic on the 3090 (24576
|
||||
# MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
|
||||
# so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
|
||||
# (weights, projector, sidecar draft, compute buffers, CUDA context)
|
||||
# is 11996-3450 = 8546 MiB. If the config asks for more slots than
|
||||
# the card holds, this inequality fails before deploy instead of the
|
||||
# KV allocation failing at boot.
|
||||
fixed=8546
|
||||
per_slot=3450
|
||||
[ $((fixed + parallel * per_slot)) -le 24576 ]
|
||||
# Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
|
||||
# without which prefill collapses about 12x (see the CUDA lesson in
|
||||
# AGENTS.md). The build-cuda tree must carry it.
|
||||
|
||||
Reference in new issue
Block a user