dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: PARALLEL=1, CTX 262144, split 0.50 (max 3090 weight)

Browse Source 
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
This commit is contained in:
Frank
2026-08-16 23:53:09 -07:00
parent 7fe271e65d
commit 53ada02a63
1 changed files with 11 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+11 -9
View File
@@ -25,16 +25,17 @@ LISTEN_PORT=7770
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 0.40,0.60 = 40% of layers on the 3090 (~26 layers) - the maximum that
# fits its 24 GB card with 2x200700 f16 KV (see the KV math in
# docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md). The 3090 is 2-3x
# faster per layer, so shifting weight to it is also a speed win.
TENSOR_SPLIT=0.40,0.60
# 0.50,0.50 = ~33 of 66 layers on the 3090 - the maximum that fits its
# 24 GB card at the 262144 f16 KV pool (VRAM math 2026-08-16, from live
# journal buffers: 33 layers -> ~23.9 GB used, 714 MiB margin; 34 layers
# -> 244 MiB margin; 35 -> OOM). The 3090 is 2-3x faster per layer, so
# maxing its share minimizes the per-step device time in the pipeline.
TENSOR_SPLIT=0.50,0.50
# Context pool. 401400 total = 2 parallel slots x 200700 per slot
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).
CTX_SIZE=401400
PARALLEL=2
CTX_SIZE=262144
PARALLEL=1
# Speculative decoding. Matches production frank-default-large: MTP (bundled
# head, n_max 4 per the published tuning curve) with the ngram-mod fallback.
@@ -51,7 +52,8 @@ SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
# KV on GPU, unquantized (f16). 16 full-attention layers x 2 (K,V) x 4 heads
# x 256 dim x 2 bytes = 64 KiB/token -> 401400 tokens = 24.5 GiB total.
# x 256 dim x 2 bytes = 64 KiB/token -> 262144 tokens = 16.6 GiB total,
# held in proportion to the full-attention layers each device holds.
# Keep K == V so Flash Attention stays enabled.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16