dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: 200000 context per slot (400000 total, 2 slots)

Browse Source 
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.

Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
This commit is contained in:
Frank
2026-08-19 10:38:53 -07:00
parent 9fd82dc6e1
commit 8e4d63b68d
6 changed files with 32 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+8 -8
View File
@@ -39,14 +39,14 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
# Serving: full native 256K context PER SLOT (n_ctx_train = 262144), 2
# slots. -c is the TOTAL context; llama.cpp splits it across the -np slots,
# so to give each slot the full 262144 we set CTX_SIZE = 2 x 262144 =
# 524288. f16 KV, context checkpoints, prompt cache, native MTP depth-4.
# The Strix Halo is unified memory; the addressable pool is 96 GiB (VRAM is
# a pointer). KV at 524288 x 64 KiB/cell is ~32 GiB; total sits inside the
# 96 GiB pool.
CTX_SIZE=524288
# Serving: 200000 context PER SLOT (user request 2026-08-19; native max
# n_ctx_train = 262144), 2 slots. -c is the TOTAL context; llama.cpp
# splits it across the -np slots, so CTX_SIZE = 2 x 200000 = 400000.
# f16 KV, context checkpoints, prompt cache, native MTP depth-4.
# The Strix Halo is unified memory; rocm-smi reports the 64 GiB budget.
# KV at 400000 x 64 KiB/cell is ~24 GiB. The ~8 GiB saved versus 524288
# adds OOM headroom (see the 2026-08-18 CTX_CHECKPOINTS 32 -> 16 change).
CTX_SIZE=400000
PARALLEL=2
BATCH_SIZE=2048
UBATCH_SIZE=2048