frank-default-small: lower context by 32k per stream for cache headroom
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the .env, models.json (contextWindow/maxTokens 229376 per slot), and the index/deployment/ornith runbook references.
This commit is contained in:
1 parent
4462aa992c
commit
7ba1acdbc5
4 files changed
+8
-7
No files matched your search
@@ -128,7 +128,8 @@ binary with the MoE expert weights pinned to CPU by an explicit
|
||||
`--override-tensor` regex (the expert `\.ffn_(up|down|gate|gate_up)_(ch|)exps`
|
||||
pattern, buffer type `CPU`) and 7 CPU threads, native `draft-mtp`
|
||||
speculative decoding (bundled MoE MTP head, n_max 4), f16 KV, context
|
||||
786432 (3 slots of ~262K each), batch 4096 / ubatch 2048, `--no-mmap`. The
|
||||
688128 (3 slots of ~229K each, 32k headroom per stream), batch 4096 /
|
||||
ubatch 2048, `--no-mmap`. The
|
||||
unit is `scripts/frank-default-small.service`. It launches through
|
||||
`scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`. See [[dealign-ornith-crack]].
|
||||
|
||||
Reference in new issue
Block a user