dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce large model context to 64K

Browse Source
This commit is contained in:
Frank
2026-08-25 04:13:00 -07:00
parent b0eb571321
commit 3cf6353050
4 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/deployment.md
+1 -1
View File
@@ -145,7 +145,7 @@ The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
is embedded in the main GGUF. K/V are f16, 131072 context (128K below the
is embedded in the main GGUF. K/V are f16, 65536 context (192K below the
native 262144 maximum; 1 slot, concurrent requests queue, the server never
preempts a running generation),
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
docs/LLM/eemin-rocmi4-mtp.md
+1 -1
View File
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings
- 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV,
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation.
scripts/frank-default-large.env
+2 -2
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: 131072 context (128K below the model's 262144 maximum), 1 slot. f16
# Serving: 65536 context (192K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=131072
CTX_SIZE=65536
PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).
tests/frank_large_config.bats
+1 -1
View File
@@ -40,7 +40,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
}
@test "frank-default-large fits the model and KV cache in unified memory" {
grep -q '^CTX_SIZE=131072$' "$ENV_FILE"
grep -q '^CTX_SIZE=65536$' "$ENV_FILE"
grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE"
}