dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: refit ctx 800000->700000 for dual-f16 KV pool

Browse Source 
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
This commit is contained in:
Frank
2026-08-14 12:55:44 -07:00
parent f70460b002
commit f96aa2b692
1 changed files with 5 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+5 -3
View File
@@ -4,9 +4,11 @@ MODEL_DIR=/home/dank/models
ALIAS=frank-default-small
HOST=0.0.0.0
PORT=7770
# 4 parallel slots x 200000 ctx each = 800000 total context pool.
# (Increased from 3×200k to add a 4th parallel stream.)
CTX_SIZE=800000
# 4 parallel slots x 175104 ctx each = 700416 total context pool.
# (2026-08-14: refitted from 800000 when the KV cache went dual-f16; the
# f16/f16 pool costs 15640 MiB at 800768 cells and did not fit the 3090's
# 24 GiB with the image-warmup graph. 700416 cells leaves ~2.3 GiB free.)
CTX_SIZE=700000
PARALLEL=4
# Spec: ngram-mod. n_min=48, n_max=64, n_match=24 (defaults). 4 expert
# layers back on GPU; 36 on CPU. Batch 8192 / ubatch 3072.