From d57174c8d39987a9aee92a45ae7a942a33521eea Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 2 Oct 2026 16:37:52 -0700 Subject: [PATCH] frank-primary: env-driven spec/compilation/sampling knobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three engine knobs that were impossible to set without editing the run line: FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG (--compilation-config) and FRANK_PRIMARY_GEN_CONFIG (--override-generation-config). Each holds the whole JSON value; empty or unset means the flag is absent, so the shipped shape E argv is unchanged. The seam re-pins the law that justified the omission: "speculative decode is ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP. fp8+spec was never measured against fp8+no-spec on one shape, the no-spec incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor flash-next recipe ships MTP3. Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is exempt from bats' errexit, so the pre-precise-memory `! grep --expert-precise-memory` never ran), and the knob test asserts by RENDERING the flag from the sourced env instead of grepping for an assignment. --- scripts/frank-primary.env | 36 +++++++++++++++++++++++++++++ scripts/serve-frank-primary.sh | 10 +++++++-- tests/frank_primary_unit.bats | 41 ++++++++++++++++++++++++---------- 3 files changed, 73 insertions(+), 14 deletions(-) diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 3109345..2c5fe5f 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -154,6 +154,42 @@ FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 FRANK_PRIMARY_UTIL=0.98 +# --- optional engine knobs (unset == the flag is absent) --------------------- +# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds +# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP +# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the +# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came +# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the +# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the +# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our +# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass +# draft_sample_method=probabilistic (greedy drafts are rejected more often under +# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace: +# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and +# --expert-precise-memory spends that same budget, so an MTP arm must be tested +# with precise memory BOTH on and off. +# MTP3: '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' +FRANK_PRIMARY_SPEC= +# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The +# capture-ladder law: the piecewise cudagraph ladder must contain +# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15; +# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default +# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can +# miss its own step size. Pinning the ladder is ALSO the candidate fix for the +# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must +# land on a captured graph size (3 is not in the default ladder, 4 is — +# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2). +# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law. +# MTP3 at SEQS 3: '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}' +FRANK_PRIMARY_COMPCONFIG= +# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON. +# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 / +# top_k 20 and the engine WARNS about it on every boot, so a client that sends +# no sampling fields gets creativity it never asked for. This is the one knob +# that changes ANSWERS, not speed. +# '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}' +FRANK_PRIMARY_GEN_CONFIG= + # --- client contract: reasoning split, tools, preserve_thinking ---------------- # Without --reasoning-parser the engine leaves the thinking text INLINE in # `content` with a dangling `` and a null `reasoning` field (measured diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index 8bf4b6e..69c2b30 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -17,8 +17,11 @@ # shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2 # CANNOT boot this model # (torch.fx ConstraintViolationError on query_start_loc, measured at both -# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative -# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry +# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode +# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG +# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02 +# retirement of MTP compared the wrong pair — see frank-primary.env. +# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows # with depth), prefix caching, expert prefetch OFF (36% precision measured), # TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead @@ -152,6 +155,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \ --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \ --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \ + ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \ + ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \ + ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \ --trust-remote-code \ > "$LOG" 2>&1 diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 6b7f3db..fdd0f7f 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -140,7 +140,6 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE" - ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE" } @@ -152,7 +151,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE" grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE" grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE" - ! grep -q -- '--speculative-config' "$SERVE" + # Speculative decode, the cudagraph capture ladder and the sampling + # defaults are all ENV-DRIVEN knobs (unset -> the flag is absent, which is + # the shipped shape today). JSON values carry no shell-special characters + # that this quoting cannot carry. + grep -q -- '${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"}' "$SERVE" + grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE" + grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE" grep -q -- '--tensor-parallel-size 2' "$SERVE" grep -q -- '--enable-prefix-caching' "$SERVE" grep -q -- '--trust-remote-code' "$SERVE" @@ -199,20 +204,32 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # --enable-expert-offload flag in this arm; the legality of the pin is # the unit's unbounded LimitMEMLOCK. ! grep -q -- '--enable-expert-offload' "$SERVE" - ! grep -q -- '--expert-precise-memory' "$SERVE" grep -q '^LimitMEMLOCK=infinity$' "$UNIT" grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE" } -@test "speculative decode is ABSENT BY MEASUREMENT" { - # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole - # spec family against no-spec at matched geometry: no-spec measured - # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried - # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves - # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures - # never completed; until one shows a real win, no spec is the incumbent. - ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE" - ! grep -q -- '--speculative-config' "$SERVE" +@test "speculative decode is an ENV KNOB — the old retirement verdict was invalid" { + # The 2026-10-02 c1248d4 retirement of MTP compared the WRONG PAIR: + # fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance) against the bf16 no-spec boot + # (36.4-39.4), then bf16 proved unbootable and fp8 came back WITHOUT MTP — + # so fp8+spec was never compared against fp8+no-spec on the same shape. The + # no-spec incumbent runs ~30 tok/s, under the fleet's 35 tok/s target, while + # EVERY vendor flash-next recipe ships MTP3, and our MTP4 run used the + # default GREEDY draft sampling (probabilistic is the other fork option and + # is what the recipes use under temp>0). So spec is a knob, not a law: the + # shipped value is whatever a matched-pair re-measurement wins. + grep -q 'THE RETIREMENT VERDICT WAS INVALID' "$ENV_FILE" + # The knobs are empty in the shipped env, so the `${VAR:+…}` wiring renders + # NO spec/compilation flag — asserted by rendering, not by grepping the + # assignment (shape E today has neither flag in its argv). + run bash -c 'set -a; . "$1" >/dev/null; set +a + printf %s "${FRANK_PRIMARY_SPEC:+--speculative-config $FRANK_PRIMARY_SPEC}${FRANK_PRIMARY_COMPCONFIG:+--compilation-config $FRANK_PRIMARY_COMPCONFIG}"' _ "$ENV_FILE" + [ -z "$output" ] + # The capture-ladder law: the piecewise graph ladder must contain + # num_speculative_tokens + 1, so an MTP3 shape captures [4, 8, 12] and the + # 2026-10-02 SEQS=2 failure (query_start_loc specialized to the constant 3) + # is why SEQS stays a multiple-compatible value. + grep -q 'num_speculative_tokens + 1' "$ENV_FILE" } @test "KV dtype is fp8 — the incumbent, and the only one that boots" {