large: Kairic chat profile — reasoning on/deepseek, batch 4096/2048, preserve_thinking

Local overrides to the pinned runner for production chat:
  - --reasoning on --reasoning-format deepseek (thinking to
    message.reasoning_content, matching the retired v3 service so the
    fleet's reasoning clients work)
  - --reasoning-budget -1 (unrestricted thinking)
  - --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
    thinking tokens in the conversation history across turns)
  - batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
    memory has headroom)

The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
This commit is contained in:
Frank committed 2026-08-22 01:03:08 -07:00
1 parent 7aee1df7c3
commit 40d50fed06
2 files changed
+26 -12

No files matched your search

+15 -9
View File
@@ -47,8 +47,8 @@ fi
: "${LISTEN_PORT:=52625}"
: "${CTX_SIZE:=262144}"
: "${PARALLEL:=1}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}"
: "${TARGET_THREADS:=16}"
: "${BATCH_THREADS:=32}"
: "${CACHE_TYPE_K:=f16}"
@@ -122,10 +122,15 @@ export PROMPTFORGE_ENABLE_SMALLM_GDN_OUTPUT_IU4=0
export PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0
export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0
# Pinned runner flags, verbatim (qualified 262K one-slot profile, native
# MTP depth 4, deterministic sampler, reasoning off). Sampling is
# overridden per request by clients; the card recommends Qwen live values
# (temp 0.7, top-p 0.8, top-k 20, min-p 0, presence_penalty 1.5).
# Pinned runner flags (qualified 262K one-slot profile, native MTP depth
# 4, deterministic sampler). Local overrides for production chat:
# - reasoning on + deepseek format (thinking goes to message.reasoning_content,
# matching the retired v3 service so the fleet's reasoning clients work)
# - batch 4096 / ubatch 2048 (larger than the runner's 2048/512)
# - chat_template_kwargs preserve_thinking=true (preserve thinking tokens
# in the conversation history across turns)
# Sampling is overridden per request by clients; the card recommends Qwen
# live values (temp 0.7, top-p 0.8, top-k 20, min-p 0, presence_penalty 1.5).
exec "$LLAMA_BIN" \
-m "$MODEL" \
--alias "$ALIAS" \
@@ -163,6 +168,7 @@ exec "$LLAMA_BIN" \
--top-p 1.0 \
--top-k 0 \
--min-p 0.0 \
--reasoning off \
--reasoning-format none \
--reasoning-budget -1
--reasoning on \
--reasoning-format deepseek \
--reasoning-budget -1 \
--chat-template-kwargs '{"preserve_thinking":"true"}'