frank-small: pool 540672 (3 x 180224) so the card keeps ~2.7 GiB free

Measured after the first 3-slot deploy: 3 x 196,608 booted (n_slots=3,
n_ctx_slot=196608, multimodal loaded) but used 22,972/24,576 MiB - 1,153 MiB
free, outside the 2-3 GiB band. The two measurements now recorded in
frank-small.env price the vision tower at ~1.4 GiB (non-KV 7,390 -> 8,859
MiB), so the pool drops 49,152 tokens (16,384/slot, 1,176 MiB) to land ~2.7
GiB free. The bats budget gate is recalibrated to the measured 9 GiB non-KV
cost, which now rejects the 3 x 196,608 shape instead of blessing it.
This commit is contained in:
Frank committed 2026-10-02 03:17:35 -07:00
1 parent a02850203d
commit 7fe881de01
4 files changed
+44 -37

No files matched your search

+5 -5
View File
@@ -37,11 +37,11 @@ fi
: "${ALIAS:=frank-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
# ctx-size is the POOL llama.cpp divides across slots: 589824 / 3 gives
# 196608 per slot — three independent conversations, each with a 192k window
# (the native 262144 no longer fits three times on a 24 GiB card; see
# frank-small.env for the arithmetic that lands ~2.5 GiB of card free).
: "${CTX_SIZE:=589824}"
# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
# 180224 per slot — three independent conversations (the native 262144 no
# longer fits three times on a 24 GiB card once the vision tower is loaded;
# frank-small.env carries the two measurements that set this number).
: "${CTX_SIZE:=540672}"
: "${PARALLEL:=3}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=q8_0}"