dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Reduce large model context to 64K

Browse Source
This commit is contained in:
Frank
2026-08-25 04:13:00 -07:00
parent b0eb571321
commit 3cf6353050
4 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+2 -2
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: 131072 context (128K below the model's 262144 maximum), 1 slot. f16
# Serving: 65536 context (192K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=131072
CTX_SIZE=65536
PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).