Reduce large model context to 64K

This commit is contained in:
Frank committed 2026-08-25 04:13:00 -07:00
1 parent b0eb571321
commit 3cf6353050
4 files changed
+5 -5

No files matched your search

+1 -1
View File
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings
- 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV,
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation.