dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce large model context to 128K

Browse Source
This commit is contained in:
Frank committed 2026-08-25 04:09:12 -07:00
1 parent ee3c3de750
commit b0eb571321
4 files changed
+5 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/deployment.md
+1 -1
View File
@@ -145,7 +145,7 @@ The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all (`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
is embedded in the main GGUF. K/V are f16, 196608 context (64K below the is embedded in the main GGUF. K/V are f16, 131072 context (128K below the
native 262144 maximum; 1 slot, concurrent requests queue, the server never native 262144 maximum; 1 slot, concurrent requests queue, the server never
preempts a running generation), preempts a running generation),
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k `-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
docs/LLM/eemin-rocmi4-mtp.md
+1 -1
View File
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings ### Serving settings
- 196608 context (64K below the native 262144 maximum), 1 slot, f16 KV, - 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation. queue; the server never preempts a running generation.
scripts/frank-default-large.env
+2 -2
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=7770 LISTEN_PORT=7770
# Serving: 196608 context (64K below the model's 262144 maximum), 1 slot. f16 # Serving: 131072 context (128K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0. # KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache. # This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation. # Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=196608 CTX_SIZE=131072
PARALLEL=1 PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in # Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts). # scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).
tests/frank_large_config.bats
+1 -1
View File
@@ -40,7 +40,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
} }
@test "frank-default-large fits the model and KV cache in unified memory" { @test "frank-default-large fits the model and KV cache in unified memory" {
grep -q '^CTX_SIZE=196608$' "$ENV_FILE" grep -q '^CTX_SIZE=131072$' "$ENV_FILE"
grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE" grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE"
} }