frank-primary: env-driven spec/compilation/sampling knobs

Three engine knobs that were impossible to set without editing the run line: FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG (--compilation-config) and FRANK_PRIMARY_GEN_CONFIG (--override-generation-config). Each holds the whole JSON value; empty or unset means the flag is absent, so the shipped shape E argv is unchanged. The seam re-pins the law that justified the omission: "speculative decode is ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP. fp8+spec was never measured against fp8+no-spec on one shape, the no-spec incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor flash-next recipe ships MTP3. Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is exempt from bats' errexit, so the pre-precise-memory `! grep --expert-precise-memory` never ran), and the knob test asserts by RENDERING the flag from the sourced env instead of grepping for an assignment.
2026-10-02 16:37:52 -07:00
parent ea1132aa0a
commit d57174c8d3
3 changed files with 73 additions and 14 deletions
@@ -154,6 +154,42 @@ FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98

+# --- optional engine knobs (unset == the flag is absent) ---------------------
+# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
+# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
+# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
+# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
+# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
+# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
+# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
+# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
+# draft_sample_method=probabilistic (greedy drafts are rejected more often under
+# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
+# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
+# --expert-precise-memory spends that same budget, so an MTP arm must be tested
+# with precise memory BOTH on and off.
+#   MTP3:  '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
+FRANK_PRIMARY_SPEC=
+# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
+# capture-ladder law: the piecewise cudagraph ladder must contain
+# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
+# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
+# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
+# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
+# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
+# land on a captured graph size (3 is not in the default ladder, 4 is —
+# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
+# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
+#   MTP3 at SEQS 3:  '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
+FRANK_PRIMARY_COMPCONFIG=
+# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
+# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
+# top_k 20 and the engine WARNS about it on every boot, so a client that sends
+# no sampling fields gets creativity it never asked for. This is the one knob
+# that changes ANSWERS, not speed.
+#   '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
+FRANK_PRIMARY_GEN_CONFIG=
+
 # --- client contract: reasoning split, tools, preserve_thinking ----------------
 # Without --reasoning-parser the engine leaves the thinking text INLINE in
 # `content` with a dangling `</think>` and a null `reasoning` field (measured
@@ -17,8 +17,11 @@
 # shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
 # CANNOT boot this model
 # (torch.fx ConstraintViolationError on query_start_loc, measured at both
-# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
-# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
+# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
+# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
+# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
+# retirement of MTP compared the wrong pair — see frank-primary.env.
+# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
 # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
 # with depth), prefix caching, expert prefetch OFF (36% precision measured),
 # TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
@@ -152,6 +155,9 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
+  ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
+  ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
+  ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
  --trust-remote-code \
  > "$LOG" 2>&1

@@ -140,7 +140,6 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
-    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }

@@ -152,7 +151,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
-    ! grep -q -- '--speculative-config' "$SERVE"
+    # Speculative decode, the cudagraph capture ladder and the sampling
+    # defaults are all ENV-DRIVEN knobs (unset -> the flag is absent, which is
+    # the shipped shape today). JSON values carry no shell-special characters
+    # that this quoting cannot carry.
+    grep -q -- '${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"}' "$SERVE"
+    grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
+    grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
    grep -q -- '--tensor-parallel-size 2' "$SERVE"
    grep -q -- '--enable-prefix-caching' "$SERVE"
    grep -q -- '--trust-remote-code' "$SERVE"
@@ -199,20 +204,32 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # --enable-expert-offload flag in this arm; the legality of the pin is
    # the unit's unbounded LimitMEMLOCK.
    ! grep -q -- '--enable-expert-offload' "$SERVE"
-    ! grep -q -- '--expert-precise-memory' "$SERVE"
    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
 }

-@test "speculative decode is ABSENT BY MEASUREMENT" {
-    # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
-    # spec family against no-spec at matched geometry: no-spec measured
-    # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
-    # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
-    # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
-    # never completed; until one shows a real win, no spec is the incumbent.
-    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
-    ! grep -q -- '--speculative-config' "$SERVE"
+@test "speculative decode is an ENV KNOB — the old retirement verdict was invalid" {
+    # The 2026-10-02 c1248d4 retirement of MTP compared the WRONG PAIR:
+    # fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance) against the bf16 no-spec boot
+    # (36.4-39.4), then bf16 proved unbootable and fp8 came back WITHOUT MTP —
+    # so fp8+spec was never compared against fp8+no-spec on the same shape. The
+    # no-spec incumbent runs ~30 tok/s, under the fleet's 35 tok/s target, while
+    # EVERY vendor flash-next recipe ships MTP3, and our MTP4 run used the
+    # default GREEDY draft sampling (probabilistic is the other fork option and
+    # is what the recipes use under temp>0). So spec is a knob, not a law: the
+    # shipped value is whatever a matched-pair re-measurement wins.
+    grep -q 'THE RETIREMENT VERDICT WAS INVALID' "$ENV_FILE"
+    # The knobs are empty in the shipped env, so the `${VAR:+…}` wiring renders
+    # NO spec/compilation flag — asserted by rendering, not by grepping the
+    # assignment (shape E today has neither flag in its argv).
+    run bash -c 'set -a; . "$1" >/dev/null; set +a
+        printf %s "${FRANK_PRIMARY_SPEC:+--speculative-config $FRANK_PRIMARY_SPEC}${FRANK_PRIMARY_COMPCONFIG:+--compilation-config $FRANK_PRIMARY_COMPCONFIG}"' _ "$ENV_FILE"
+    [ -z "$output" ]
+    # The capture-ladder law: the piecewise graph ladder must contain
+    # num_speculative_tokens + 1, so an MTP3 shape captures [4, 8, 12] and the
+    # 2026-10-02 SEQS=2 failure (query_start_loc specialized to the constant 3)
+    # is why SEQS stays a multiple-compatible value.
+    grep -q 'num_speculative_tokens + 1' "$ENV_FILE"
 }

@test "KV dtype is fp8 — the incumbent, and the only one that boots" {