frank-primary: env-driven spec/compilation/sampling knobs
Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.
The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.
Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
This commit is contained in:
1 parent
ea1132aa0a
commit
d57174c8d3
3 files changed
+73
-14
No files matched your search
@@ -17,8 +17,11 @@
|
||||
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
||||
# CANNOT boot this model
|
||||
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
|
||||
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
|
||||
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
||||
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
|
||||
# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
|
||||
# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
|
||||
# retirement of MTP compared the wrong pair — see frank-primary.env.
|
||||
# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
||||
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
|
||||
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
||||
@@ -152,6 +155,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
||||
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
||||
${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
|
||||
${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
|
||||
${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
|
||||
--trust-remote-code \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
|
||||
Reference in new issue
Block a user