Reduce frank-default-large context for KV headroom
This commit is contained in:
1 parent
7ca9dc248e
commit
ee3c3de750
5 files changed
+18
-11
No files matched your search
@@ -112,9 +112,10 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Serving settings
|
||||
|
||||
- 262144 context (native max), 1 slot, f16 KV, context checkpoints, prompt
|
||||
cache. Concurrent requests queue; the server never preempts a running
|
||||
generation.
|
||||
- 196608 context (64K below the native 262144 maximum), 1 slot, f16 KV,
|
||||
context checkpoints, prompt cache. The lower limit leaves headroom for the
|
||||
model plus KV cache in the host's unified-memory budget. Concurrent requests
|
||||
queue; the server never preempts a running generation.
|
||||
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
|
||||
`message.reasoning_content`).
|
||||
|
||||
Reference in new issue
Block a user