dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-large: split 0.55,0.45 = 35 blocks on the 3090 (true max)

Browse Source 
GGUF block scan (2026-08-17): FA at 3,7,...,63; linear blocks 383 MiB,
FA blocks 446 MiB. 35 blocks (0-34) keep 8 FA layers on RPC0 -> KV stays
8192 MiB, predicted ~0.8 GiB free. Block 35 is FA: +1 GiB KV -> OOM.
Measured at 0.50: 22650/24576 used, 1477 free (the earlier 714 MiB
projection was conservative).
This commit is contained in:
Frank
2026-08-17 00:34:58 -07:00
parent 53ada02a63
commit accf90b6fa
1 changed files with 7 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+7 -6
View File
@@ -25,12 +25,13 @@ LISTEN_PORT=7770
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 0.50,0.50 = ~33 of 66 layers on the 3090 - the maximum that fits its
# 24 GB card at the 262144 f16 KV pool (VRAM math 2026-08-16, from live
# journal buffers: 33 layers -> ~23.9 GB used, 714 MiB margin; 34 layers
# -> 244 MiB margin; 35 -> OOM). The 3090 is 2-3x faster per layer, so
# maxing its share minimizes the per-step device time in the pipeline.
TENSOR_SPLIT=0.50,0.50
# 0.55,0.45 = 35 of 65 blocks on the 3090 (blocks 0-34) - the maximum
# that fits its 24 GB card at the 262144 f16 KV pool. VRAM math from the
# live load (2026-08-17): RPC0 buffer ~14.4 GiB + KV 8192 MiB (8 full-
# attention layers) + RS/compute/overhead -> ~0.8 GiB free. Block 35 is a
# full-attention layer: adding it costs +1 GiB KV -> OOM. GGUF block scan:
# FA at 3,7,...,63 (every 4th, offset 3).
TENSOR_SPLIT=0.55,0.45
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).