frank-default-small: 128K context and thinking on

- CTX_SIZE 65536 -> 131072 (user request). KV at 128K is 8 GiB
  (64 KiB/token f16); estimated ~43 GiB used of the 64 GiB SVM pool,
  ~21 GiB headroom.
- REASONING off -> on with REASONING_FORMAT=deepseek, so <think>
  reasoning lands in message.reasoning_content (OpenAI-compatible)
  instead of being left unparsed in content.
This commit is contained in:
Frank committed 2026-08-17 23:11:13 -07:00
1 parent 64cdfc8529
commit 2e1a78aeb4
2 files changed
+14 -7

No files matched your search

+10 -4
View File
@@ -39,9 +39,11 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
# Serving defaults from the model card: 64K context, single slot, f16 KV,
# context checkpoints, prompt cache. Native MTP depth-4.
CTX_SIZE=65536
# Serving: 128K context (user request, 2026-08-18; model card default is
# 64K), single slot, f16 KV, context checkpoints, prompt cache. Native MTP
# depth-4. KV at 128K is 8 GiB (64 KiB/token); estimated total ~43 GiB of
# the 64 GiB SVM pool, leaving ~21 GiB headroom.
CTX_SIZE=131072
PARALLEL=1
BATCH_SIZE=2048
UBATCH_SIZE=2048
@@ -60,6 +62,10 @@ SPEC_DRAFT_P_SPLIT=0.10
TEMPERATURE=0.7
TOP_P=0.95
REASONING=off
# Thinking enabled (user request, 2026-08-18). deepseek format moves the
# <think> reasoning into message.reasoning_content (OpenAI-compatible);
# none would leave the thought tags unparsed in content.
REASONING=on
REASONING_FORMAT=deepseek
VERBOSITY=4