frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
1 parent
e8524d4046
commit
7cab15ac39
7 files changed
+210
-53
No files matched your search
@@ -47,7 +47,10 @@ set +a
|
||||
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
||||
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
||||
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
||||
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
||||
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
||||
# classic `podman images | grep -q` pattern races — grep exits on its first
|
||||
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
||||
@@ -115,6 +118,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
||||
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
||||
--trust-remote-code \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
|
||||
Reference in new issue
Block a user