Reduce frank-default-large context for KV headroom
This commit is contained in:
1 parent
7ca9dc248e
commit
ee3c3de750
5 files changed
+18
-11
No files matched your search
@@ -145,8 +145,9 @@ The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
|
||||
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
|
||||
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
|
||||
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
|
||||
is embedded in the main GGUF. K/V are f16, full 262144 context (1 slot,
|
||||
concurrent requests queue, the server never preempts a running generation),
|
||||
is embedded in the main GGUF. K/V are f16, 196608 context (64K below the
|
||||
native 262144 maximum; 1 slot, concurrent requests queue, the server never
|
||||
preempts a running generation),
|
||||
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
|
||||
context with `scripts/bench-ubatch-rocmi4.sh`), 16 context checkpoints, 8 GiB
|
||||
prompt cache, and reasoning in deepseek format. The unit file is
|
||||
|
||||
Reference in new issue
Block a user