frank-primary: max-possible window (241664) + reasoning/tool contract

Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank committed 2026-10-01 03:08:52 -07:00
1 parent e8524d4046
commit 7cab15ac39
7 files changed
+210 -53

No files matched your search

+7 -1
View File
@@ -47,7 +47,10 @@ set +a
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
# classic `podman images | grep -q` pattern races — grep exits on its first
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
@@ -115,6 +118,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--enable-prefix-caching \
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
--trust-remote-code \
> "$LOG" 2>&1