dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: full 262144 context, 1 slot

Browse Source 
User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
This commit is contained in:
Frank
2026-08-19 11:17:31 -07:00
parent 8e4d63b68d
commit 048e7eaaba
6 changed files with 22 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+9 -9
View File
@@ -39,15 +39,15 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
# Serving: 200000 context PER SLOT (user request 2026-08-19; native max
# n_ctx_train = 262144), 2 slots. -c is the TOTAL context; llama.cpp
# splits it across the -np slots, so CTX_SIZE = 2 x 200000 = 400000.
# f16 KV, context checkpoints, prompt cache, native MTP depth-4.
# The Strix Halo is unified memory; rocm-smi reports the 64 GiB budget.
# KV at 400000 x 64 KiB/cell is ~24 GiB. The ~8 GiB saved versus 524288
# adds OOM headroom (see the 2026-08-18 CTX_CHECKPOINTS 32 -> 16 change).
CTX_SIZE=400000
PARALLEL=2
# Serving: full native 262144 context (n_ctx_train), 1 slot (user request
# 2026-08-19: parallel slots degraded performance). -c is the TOTAL
# context; with PARALLEL=1 it equals the per-slot context. Concurrent
# requests queue on the single slot; the server never preempts a running
# generation. f16 KV, context checkpoints, prompt cache, native MTP
# depth-4. The Strix Halo is unified memory; rocm-smi reports the 64 GiB
# budget. KV at 262144 x 64 KiB/cell is ~16 GiB.
CTX_SIZE=262144
PARALLEL=1
BATCH_SIZE=2048
UBATCH_SIZE=2048
CACHE_TYPE_K=f16