frank-primary: env-driven spec/compilation/sampling knobs

Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.

The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.

Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
This commit is contained in:
Frank committed 2026-10-02 16:37:52 -07:00
1 parent ea1132aa0a
commit d57174c8d3
3 files changed
+73 -14

No files matched your search

+36
View File
@@ -154,6 +154,42 @@ FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98
# --- optional engine knobs (unset == the flag is absent) ---------------------
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
# draft_sample_method=probabilistic (greedy drafts are rejected more often under
# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
# --expert-precise-memory spends that same budget, so an MTP arm must be tested
# with precise memory BOTH on and off.
# MTP3: '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
FRANK_PRIMARY_SPEC=
# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
# capture-ladder law: the piecewise cudagraph ladder must contain
# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
# land on a captured graph size (3 is not in the default ladder, 4 is —
# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
# MTP3 at SEQS 3: '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
FRANK_PRIMARY_COMPCONFIG=
# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
# top_k 20 and the engine WARNS about it on every boot, so a client that sends
# no sampling fields gets creativity it never asked for. This is the one knob
# that changes ANSWERS, not speed.
# '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
FRANK_PRIMARY_GEN_CONFIG=
# --- client contract: reasoning split, tools, preserve_thinking ----------------
# Without --reasoning-parser the engine leaves the thinking text INLINE in
# `content` with a dangling `</think>` and a null `reasoning` field (measured