Reduce large model context to 128K

This commit is contained in:
Frank committed 2026-08-25 04:09:12 -07:00
1 parent ee3c3de750
commit b0eb571321
4 files changed
+5 -5

No files matched your search

+1 -1
View File
@@ -112,7 +112,7 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings
- 196608 context (64K below the native 262144 maximum), 1 slot, f16 KV,
- 131072 context (128K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation.