frank-primary: max-possible window (241664) + reasoning/tool contract

Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank committed 2026-10-01 03:08:52 -07:00
1 parent e8524d4046
commit 7cab15ac39
7 files changed
+210 -53

No files matched your search

+32 -26
View File
@@ -6,8 +6,8 @@
# script drives `podman run` DIRECTLY (the image entrypoint execs
# `vllm serve`), with every engine flag qualified by boot probe and carried
# in the repo env file. The seam pins what the flags alone do not say: the
# KV qualification law (11.05 GiB/rank boots bf16 at 798,280 tokens / 3.05x
# concurrency; the old 7.2 GiB ceiling was the MTP4 workspace, not VRAM),
# KV qualification law (the night ladder: 262144/245760 OOM at graph
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
# clean cache + 262k windows), the automatic-offload law (the fork plans
# the tiered-expert offload itself; no flag exists to set), the self-healing
@@ -18,7 +18,8 @@
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
# no-spec family measured fastest once the prefetch tax was removed; MTP
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
# MNBT 1024 (Frank's lock), and NO tool dialect.
# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
# tool parser qwen3_xml, preserve_thinking on).
load helpers
@@ -74,22 +75,19 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "the qualified engine shape is in the env file" {
# 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, KV_BYTES
# 11,047,010,304 PER RANK boots at "KV cache size: 798,280 tokens" and
# reports max concurrency 3.05x for 262,144-token requests — all three
# full windows fit with an 11,848-token margin. The old "KV_BYTES is
# PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's workspace
# reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank and boots
# clean.
# UTIL 0.90 is a PRODUCTION-corrected value: the first production boot
# at 0.98 died at KV allocation (torch OOM, tried 10.28 GiB with 9.46
# free — 0.82 GiB short) while the rootless bench probe at the same
# numbers had booted; util 0.90 returns ~2.4 GiB/rank of workspace and
# is the deployed value.
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
# 2026-10-01 night ladder (Frank's method: subtract until it boots, then
# one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes
# free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens /
# 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production
# (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token
# margin over the 3-window contract). UTIL stays 0.98 — the production
# unit's pinned-expert path needs the workspace the bench probe hid;
# 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the
# pool with the window.
grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.90$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
@@ -112,14 +110,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '--trust-remote-code' "$SERVE"
}
@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
# The 2339fae trio does NOT wire --enable-auto-tool-choice /
# --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
# dropped them). tool_choice "auto" gets a 400 here — tool-using agents
# must route at frank-small or frank-large until a parser is qualified
# for this arm (a behavior change beyond the rollback, not a silent fix).
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
! grep -q -- '--tool-call-parser' "$SERVE"
@test "tools and reasoning are wired — the client contract is served" {
# 2026-10-01: the rollback-era trio served NO tool dialect and left
# thinking inline in `content` with a dangling `</think>` and a null
# `reasoning` field (measured: content 'We need answer...Brief.\n</think>
# \n\n391') — the malformed thinking block clients choke on. The working
# values are the ones ecaad36 shipped for this model and this fork's
# registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
# preserve_thinking keeps prior thinking blocks across turns.
grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
grep -q -- '--enable-auto-tool-choice' "$SERVE"
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
grep -q 'preserve_thinking' "$ENV_FILE"
grep -q 'enable_thinking' "$ENV_FILE"
}
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {