From 193034fdb87ae46ae3321c362b6a0969f939ae0d Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 17 Aug 2026 00:58:06 -0700 Subject: [PATCH] frank-default-large: split 0.54,0.46 - 35 blocks on 3090 (0.55 overran to 36 -> OOM) 0.55 measured: 36 blocks (0-35) -> 9th FA block, KV 9216 MiB, cudaMalloc OOM by ~740 MiB, crash loop. Exact GGUF math: layer pool 27720 MB + MTP head; split assigns blocks while cumulative <= fraction x pool; cum(34)= 53.83%, cum(35)=55.31%. 0.54 lands 35 blocks (0-34), 8 FA layers, KV 8192 MiB, ~0.66 GiB free. --- scripts/frank-default-large.env | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 0a0cb37..7226efc 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -25,13 +25,14 @@ LISTEN_PORT=7770 RPC_SERVERS=192.168.0.235:50052 # Model split fractions, one per device in order [RPC0, iGPU]. -# 0.55,0.45 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum -# that fits its 24 GB card at the 262144 f16 KV pool. VRAM math from the -# live load (2026-08-17): RPC0 buffer ~14.4 GiB + KV 8192 MiB (8 full- -# attention layers) + RS/compute/overhead -> ~0.8 GiB free. Block 35 is a -# full-attention layer: adding it costs +1 GiB KV -> OOM. GGUF block scan: -# FA at 3,7,...,63 (every 4th, offset 3). -TENSOR_SPLIT=0.55,0.45 +# 0.54,0.46 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum. +# Split math (exact GGUF sizes, 2026-08-17): layer pool 27720 MB + MTP +# head; llama.cpp assigns RPC0 the blocks while cumulative weight <= the +# fraction of the pool. cum(34) = 53.83%, cum(35) = 55.31%. Measured: +# 0.50 -> 32 blocks, 0.55 -> 36 blocks (9th FA block 35 -> KV 9216 MiB +# -> cudaMalloc OOM by ~740 MiB). FA blocks at 3,7,...,63; KV stays +# 8192 MiB (8 FA) at 35 blocks. Predicted free ~0.65 GiB. +TENSOR_SPLIT=0.54,0.46 # Context pool. 262144 total = 1 parallel slot x 262144 per slot # (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).