dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-large: split 0.54,0.46 - 35 blocks on 3090 (0.55 overran to 36 -> OOM)

Browse Source 
0.55 measured: 36 blocks (0-35) -> 9th FA block, KV 9216 MiB, cudaMalloc
OOM by ~740 MiB, crash loop. Exact GGUF math: layer pool 27720 MB + MTP
head; split assigns blocks while cumulative <= fraction x pool; cum(34)=
53.83%, cum(35)=55.31%. 0.54 lands 35 blocks (0-34), 8 FA layers, KV
8192 MiB, ~0.66 GiB free.
This commit is contained in:
Frank
2026-08-17 00:58:06 -07:00
parent accf90b6fa
commit 193034fdb8
1 changed files with 8 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+8 -7
View File
@@ -25,13 +25,14 @@ LISTEN_PORT=7770
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 0.55,0.45 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum
# that fits its 24 GB card at the 262144 f16 KV pool. VRAM math from the
# live load (2026-08-17): RPC0 buffer ~14.4 GiB + KV 8192 MiB (8 full-
# attention layers) + RS/compute/overhead -> ~0.8 GiB free. Block 35 is a
# full-attention layer: adding it costs +1 GiB KV -> OOM. GGUF block scan:
# FA at 3,7,...,63 (every 4th, offset 3).
TENSOR_SPLIT=0.55,0.45
# 0.54,0.46 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum.
# Split math (exact GGUF sizes, 2026-08-17): layer pool 27720 MB + MTP
# head; llama.cpp assigns RPC0 the blocks while cumulative weight <= the
# fraction of the pool. cum(34) = 53.83%, cum(35) = 55.31%. Measured:
# 0.50 -> 32 blocks, 0.55 -> 36 blocks (9th FA block 35 -> KV 9216 MiB
# -> cudaMalloc OOM by ~740 MiB). FA blocks at 3,7,...,63; KV stays
# 8192 MiB (8 FA) at 35 blocks. Predicted free ~0.65 GiB.
TENSOR_SPLIT=0.54,0.46
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).