large: Kairic chat profile — reasoning on/deepseek, batch 4096/2048, preserve_thinking
Local overrides to the pinned runner for production chat:
- --reasoning on --reasoning-format deepseek (thinking to
message.reasoning_content, matching the retired v3 service so the
fleet's reasoning clients work)
- --reasoning-budget -1 (unrestricted thinking)
- --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
thinking tokens in the conversation history across turns)
- batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
memory has headroom)
The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
This commit is contained in:
1 parent
7aee1df7c3
commit
40d50fed06
2 files changed
+26
-12
No files matched your search
@@ -50,8 +50,10 @@ LISTEN_PORT=52625
|
||||
# while the prompt cache is enabled.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
# Local override: the pinned runner uses 2048/512; production chat uses a
|
||||
# larger 4096/2048 batch (unified memory has headroom).
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=2048
|
||||
TARGET_THREADS=16
|
||||
BATCH_THREADS=32
|
||||
CACHE_TYPE_K=f16
|
||||
@@ -70,4 +72,10 @@ SPEC_DRAFT_P_SPLIT=0.10
|
||||
# profile (temp 0 / top-p 1 / top-k 0 / min-p 0) as the server default.
|
||||
# Clients override per request; the model card recommends Qwen live
|
||||
# values for ordinary chat (temp 0.7, top-p 0.8, top-k 20, min-p 0,
|
||||
# presence_penalty 1.5). Reasoning is off in the qualified profile.
|
||||
# presence_penalty 1.5).
|
||||
#
|
||||
# Reasoning: the qualified profile runs reasoning off, but production
|
||||
# chat runs --reasoning on --reasoning-format deepseek (thinking goes to
|
||||
# message.reasoning_content, matching the retired v3 service) with
|
||||
# --reasoning-budget -1 (unrestricted) and chat_template_kwargs
|
||||
# preserve_thinking=true (preserve thinking tokens across turns).
|
||||
Reference in new issue
Block a user