dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce large model context to 64K

Browse Source
This commit is contained in:
Frank committed 2026-08-25 04:13:00 -07:00
1 parent b0eb571321
commit 3cf6353050
4 files changed
+5 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/deployment.md
+1 -1
View File
@@ -145,7 +145,7 @@ The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all (`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
is embedded in the main GGUF. K/V are f16, 131072 context (128K below the is embedded in the main GGUF. K/V are f16, 65536 context (192K below the
native 262144 maximum; 1 slot, concurrent requests queue, the server never native 262144 maximum; 1 slot, concurrent requests queue, the server never
preempts a running generation), preempts a running generation),
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k `-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
docs/LLM/eemin-rocmi4-mtp.md
+1 -1
View File
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings ### Serving settings
- 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV, - 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation. queue; the server never preempts a running generation.
scripts/frank-default-large.env
+2 -2
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=7770 LISTEN_PORT=7770
# Serving: 131072 context (128K below the model's 262144 maximum), 1 slot. f16 # Serving: 65536 context (192K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0. # KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache. # This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation. # Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=131072 CTX_SIZE=65536
PARALLEL=1 PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in # Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts). # scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).
tests/frank_large_config.bats
+1 -1
View File
@@ -40,7 +40,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
} }
@test "frank-default-large fits the model and KV cache in unified memory" { @test "frank-default-large fits the model and KV cache in unified memory" {
grep -q '^CTX_SIZE=131072$' "$ENV_FILE" grep -q '^CTX_SIZE=65536$' "$ENV_FILE"
grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE" grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE"
} }