Reduce large model context to 64K
This commit is contained in:
1 parent
b0eb571321
commit
3cf6353050
4 files changed
+5
-5
No files matched your search
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Serving settings
|
||||
|
||||
- 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV,
|
||||
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
|
||||
context checkpoints, prompt cache. The lower limit leaves headroom for the
|
||||
model plus KV cache in the host's unified-memory budget. Concurrent requests
|
||||
queue; the server never preempts a running generation.
|
||||
|
||||
Reference in new issue
Block a user