dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce large model context to 128K

Browse Source
This commit is contained in:
Frank
2026-08-25 04:09:12 -07:00
parent ee3c3de750
commit b0eb571321
4 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+2 -2
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: 196608 context (64K below the model's 262144 maximum), 1 slot. f16
# Serving: 131072 context (128K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=196608
CTX_SIZE=131072
PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).