diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 0a0cb37..7226efc 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -25,13 +25,14 @@ LISTEN_PORT=7770 RPC_SERVERS=192.168.0.235:50052 # Model split fractions, one per device in order [RPC0, iGPU]. -# 0.55,0.45 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum -# that fits its 24 GB card at the 262144 f16 KV pool. VRAM math from the -# live load (2026-08-17): RPC0 buffer ~14.4 GiB + KV 8192 MiB (8 full- -# attention layers) + RS/compute/overhead -> ~0.8 GiB free. Block 35 is a -# full-attention layer: adding it costs +1 GiB KV -> OOM. GGUF block scan: -# FA at 3,7,...,63 (every 4th, offset 3). -TENSOR_SPLIT=0.55,0.45 +# 0.54,0.46 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum. +# Split math (exact GGUF sizes, 2026-08-17): layer pool 27720 MB + MTP +# head; llama.cpp assigns RPC0 the blocks while cumulative weight <= the +# fraction of the pool. cum(34) = 53.83%, cum(35) = 55.31%. Measured: +# 0.50 -> 32 blocks, 0.55 -> 36 blocks (9th FA block 35 -> KV 9216 MiB +# -> cudaMalloc OOM by ~740 MiB). FA blocks at 3,7,...,63; KV stays +# 8192 MiB (8 FA) at 35 blocks. Predicted free ~0.65 GiB. +TENSOR_SPLIT=0.54,0.46 # Context pool. 262144 total = 1 parallel slot x 262144 per slot # (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).