frank-default-small: lower context by 32k per stream for cache headroom
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the .env, models.json (contextWindow/maxTokens 229376 per slot), and the index/deployment/ornith runbook references.
This commit is contained in:
1 parent
4462aa992c
commit
7ba1acdbc5
4 files changed
+8
-7
No files matched your search
@@ -77,7 +77,7 @@ llama-server \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--spec-type draft-mtp \
|
||||
--override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||||
-ngl all -c 786432 -np 3 \
|
||||
-ngl all -c 688128 -np 3 \
|
||||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||||
--temp 1.0 --top-p 0.95 --top-k 20
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user