large: Kairic chat profile — reasoning on/deepseek, batch 4096/2048, preserve_thinking

Local overrides to the pinned runner for production chat:
  - --reasoning on --reasoning-format deepseek (thinking to
    message.reasoning_content, matching the retired v3 service so the
    fleet's reasoning clients work)
  - --reasoning-budget -1 (unrestricted thinking)
  - --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
    thinking tokens in the conversation history across turns)
  - batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
    memory has headroom)

The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
This commit is contained in:
Frank committed 2026-08-22 01:03:08 -07:00
1 parent 7aee1df7c3
commit 40d50fed06
2 files changed
+26 -12

No files matched your search

+11 -3
View File
@@ -50,8 +50,10 @@ LISTEN_PORT=52625
# while the prompt cache is enabled.
CTX_SIZE=262144
PARALLEL=1
BATCH_SIZE=2048
UBATCH_SIZE=512
# Local override: the pinned runner uses 2048/512; production chat uses a
# larger 4096/2048 batch (unified memory has headroom).
BATCH_SIZE=4096
UBATCH_SIZE=2048
TARGET_THREADS=16
BATCH_THREADS=32
CACHE_TYPE_K=f16
@@ -70,4 +72,10 @@ SPEC_DRAFT_P_SPLIT=0.10
# profile (temp 0 / top-p 1 / top-k 0 / min-p 0) as the server default.
# Clients override per request; the model card recommends Qwen live
# values for ordinary chat (temp 0.7, top-p 0.8, top-k 20, min-p 0,
# presence_penalty 1.5). Reasoning is off in the qualified profile.
# presence_penalty 1.5).
#
# Reasoning: the qualified profile runs reasoning off, but production
# chat runs --reasoning on --reasoning-format deepseek (thinking goes to
# message.reasoning_content, matching the retired v3 service) with
# --reasoning-budget -1 (unrestricted) and chat_template_kwargs
# preserve_thinking=true (preserve thinking tokens across turns).