frank-default-large: split 0.53,0.47 - provably-safe max (34 blocks)

0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
This commit is contained in:
Frank
2026-08-17 01:21:21 -07:00
parent 193034fdb8
commit 2b96043740
+10 -8
View File
@@ -25,14 +25,16 @@ LISTEN_PORT=7770
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 0.54,0.46 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum.
# Split math (exact GGUF sizes, 2026-08-17): layer pool 27720 MB + MTP
# head; llama.cpp assigns RPC0 the blocks while cumulative weight <= the
# fraction of the pool. cum(34) = 53.83%, cum(35) = 55.31%. Measured:
# 0.50 -> 32 blocks, 0.55 -> 36 blocks (9th FA block 35 -> KV 9216 MiB
# -> cudaMalloc OOM by ~740 MiB). FA blocks at 3,7,...,63; KV stays
# 8192 MiB (8 FA) at 35 blocks. Predicted free ~0.65 GiB.
TENSOR_SPLIT=0.54,0.46
# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33) - the maximum
# provably-safe split, measured 2026-08-17 (two failed attempts):
# - 0.50: 32 blocks, KV 8192 MiB (8 FA) -> healthy, 1477 MiB free.
# - 0.54: weights 35 blocks but KV jumps to 9216 MiB (9th FA block) ->
# cudaMalloc OOM by ~740 MiB. The KV-per-device count rounds the
# split fraction over 66 layers, independent of the weight boundary;
# the flip to 9 FA sits at f ~= 0.538, just below the weight
# threshold for block 35. 0.53 keeps 8 FA under every rounding rule.
# GGUF block scan: FA at 3,7,...,63; linear blocks 431 MB, FA 410 MB.
TENSOR_SPLIT=0.53,0.47
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).