dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: full native 262144 context per slot (524288 total)

Browse Source 
llama.cpp splits -c across -np slots; to give each of the 2 slots the full
262144 native context, set CTX_SIZE=524288. KV ~32 GiB; fits the 96 GiB
unified pool. Update env default, serve-script fallback, and the deployment
doc line.
This commit is contained in:
Frank
2026-08-18 08:01:17 -07:00
parent 819b9d448a
commit a08be1641f
3 changed files with 12 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+8 -6
View File
@@ -39,12 +39,14 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
# Serving: 256K context (native max, n_ctx_train = 262144), 2 slots, f16 KV,
# context checkpoints, prompt cache. Native MTP depth-4. The Strix Halo is
# unified memory; the addressable pool is 96 GiB (VRAM is a pointer). KV at
# 262144 x 2 slots is ~32 GiB (64 KiB/cell, 16 full-attn layers); total
# sits well inside the 96 GiB pool.
CTX_SIZE=262144
# Serving: full native 256K context PER SLOT (n_ctx_train = 262144), 2
# slots. -c is the TOTAL context; llama.cpp splits it across the -np slots,
# so to give each slot the full 262144 we set CTX_SIZE = 2 x 262144 =
# 524288. f16 KV, context checkpoints, prompt cache, native MTP depth-4.
# The Strix Halo is unified memory; the addressable pool is 96 GiB (VRAM is
# a pointer). KV at 524288 x 64 KiB/cell is ~32 GiB; total sits inside the
# 96 GiB pool.
CTX_SIZE=524288
PARALLEL=2
BATCH_SIZE=2048
UBATCH_SIZE=2048
scripts/serve-frank-large.sh
+1 -1
View File
@@ -37,7 +37,7 @@ fi
: "${ALIAS:=frank-default-large}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=52625}"
: "${CTX_SIZE:=262144}"
: "${CTX_SIZE:=524288}"
: "${PARALLEL:=2}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=2048}"