dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-small: 491520 context pool (163840 per slot x 3)

Browse Source 
--ctx-size is the total pool divided among --parallel slots, so with
PARALLEL=3 the 160k target must be a 491520 pool (163840/slot) to
actually deliver 160k context per slot. Env, serve default, bats updated.
This commit is contained in:
Frank
2026-08-26 05:48:28 -07:00
parent 04a8fb7806
commit 8f860d24a6
3 changed files with 10 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+6 -5
View File
@@ -43,11 +43,12 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama
# limits speculative decoding to a single slot, so SPEC stays none (MTP is
# also disabled). The 3 slots triple the KV cache, so enough expert weights
# are CPU-pinned for the cache to fit on the 24 GB card.
CTX_SIZE=160000
# Three parallel slots sharing a 491520 total context pool (491520/3 =
# 163840 per slot — the requested 160k). With PARALLEL=3 ik_llama limits
# speculative decoding to a single slot, so SPEC stays none (MTP is also
# disabled). The 3 slots + 480k pool make the KV cache large, so expert
# weights are CPU-pinned for the cache to fit on the 24 GB card.
CTX_SIZE=491520
PARALLEL=3
# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at
scripts/serve-frank-small.sh
+1 -1
View File
@@ -49,7 +49,7 @@ fi
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=160000}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=q8_0}"