frank-small: pool 540672 (3 x 180224) so the card keeps ~2.7 GiB free
Measured after the first 3-slot deploy: 3 x 196,608 booted (n_slots=3, n_ctx_slot=196608, multimodal loaded) but used 22,972/24,576 MiB - 1,153 MiB free, outside the 2-3 GiB band. The two measurements now recorded in frank-small.env price the vision tower at ~1.4 GiB (non-KV 7,390 -> 8,859 MiB), so the pool drops 49,152 tokens (16,384/slot, 1,176 MiB) to land ~2.7 GiB free. The bats budget gate is recalibrated to the measured 9 GiB non-KV cost, which now rejects the 3 x 196,608 shape instead of blessing it.
This commit is contained in:
1 parent
a02850203d
commit
7fe881de01
4 files changed
+44
-37
No files matched your search
@@ -37,11 +37,11 @@ fi
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 589824 / 3 gives
|
||||
# 196608 per slot — three independent conversations, each with a 192k window
|
||||
# (the native 262144 no longer fits three times on a 24 GiB card; see
|
||||
# frank-small.env for the arithmetic that lands ~2.5 GiB of card free).
|
||||
: "${CTX_SIZE:=589824}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
|
||||
# 180224 per slot — three independent conversations (the native 262144 no
|
||||
# longer fits three times on a 24 GiB card once the vision tower is loaded;
|
||||
# frank-small.env carries the two measurements that set this number).
|
||||
: "${CTX_SIZE:=540672}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
|
||||
Reference in new issue
Block a user