Reduce frank-default-large context for KV headroom

This commit is contained in:
Frank committed 2026-08-25 04:05:33 -07:00
1 parent 7ca9dc248e
commit ee3c3de750
5 files changed
+18 -11

No files matched your search

+3 -2
View File
@@ -145,8 +145,9 @@ The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
is embedded in the main GGUF. K/V are f16, full 262144 context (1 slot,
concurrent requests queue, the server never preempts a running generation),
is embedded in the main GGUF. K/V are f16, 196608 context (64K below the
native 262144 maximum; 1 slot, concurrent requests queue, the server never
preempts a running generation),
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
context with `scripts/bench-ubatch-rocmi4.sh`), 16 context checkpoints, 8 GiB
prompt cache, and reasoning in deepseek format. The unit file is
+4 -3
View File
@@ -112,9 +112,10 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings
- 262144 context (native max), 1 slot, f16 KV, context checkpoints, prompt
cache. Concurrent requests queue; the server never preempts a running
generation.
- 196608 context (64K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation.
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
`message.reasoning_content`).