dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: 2×114688 ctx (112k per-slot), ubatch 1024

Browse Source 
Lower per-slot context from 131,072 (128k) to 114,688 (112k) to free
~1 GiB of KV, which funds the ubatch 512 -> 1024 increase (prefill
throughput). The per-slot window is the product contract, so models.yml
contextWindow follows to 114,688.

- CTX_SIZE: 262144 -> 229376 (2 × 114,688)
- UBATCH_SIZE: 512 -> 1024
- Test 194: per-slot 131072 -> 114688
- KV budget test: 131072 -> 114688, ubatch 512 -> 1024
This commit is contained in:
Frank
2026-10-06 16:38:58 -07:00
parent 87721ab0fe
commit f2575a1018
2 changed files with 22 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-secondary.env
+11 -12
View File
@@ -57,23 +57,22 @@ ALIAS=frank-secondary
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# TWO conversations at Frank's 128k cap (2026-10-06): --ctx-size is the
# POOL llama.cpp divides across slots; 262144 / 2 = 131,072 per
# conversation. Frank accepted the tight fit explicitly (arithmetic above);
# the per-slot window is the product contract, so models.yml stays at
# 131,072.
CTX_SIZE=262144
# TWO conversations at Frank's 112k cap (2026-10-06 evening): --ctx-size is
# the POOL llama.cpp divides across slots; 229376 / 2 = 114,688 per
# conversation. Lowering from 128k (131,072) frees ~1 GiB of KV, which
# funds the ubatch 512 -> 1024 increase (prefill throughput). The
# per-slot window is the product contract, so models.yml follows to
# 114,688.
CTX_SIZE=229376
PARALLEL=2
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
# on the 9B the 1024->2048 rung measured +4.6 % prefill for 616 MiB, and
# Frank's 2026-10-06 order spends the other direction: ubatch 512 claws
# back ~0.3 GiB toward the second slot at some prefill cost. The pair is a
# measurement, not a law: ubatch 1024 is the first A/B back if a future
# boot lands with headroom (loud failure at graph capture if it does not
# the 1024 rung was the first A/B back once the per-slot context dropped
# to 112k (2026-10-06 evening) and freed ~1 GiB of KV. The pair is a
# measurement, not a law (loud failure at graph capture if it does not
# fit).
BATCH_SIZE=4096
UBATCH_SIZE=512
UBATCH_SIZE=1024
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in