frank-primary: bf16 no-spec cutover (262144x3, gather 32x64, MNBT 1024)

Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired
to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache
unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank
(engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token
requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth).
Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM;
spec decode absent by measurement; bf16 trade documented (clean cache +
262k windows for ~14% decode at 4064 ctx).

scripts/frank-primary.env

@@ -1,37 +1,44 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
+# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
-# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
+# campaign's verdict, Frank's call): same digest-pinned engine and payload as
-# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
+# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
-# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
+# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
-# digest-pinned image the 2026-09-29 tuning campaign measured — serving
+# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
+# restorable from the tag.
-# the MTP4 drafter on the two R9700s.
 #
-# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
+# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
-#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
+#   * MAXLEN 262144 x SEQS 3 (786,432 tokens) at bf16 KV boots at
-#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
+#     KV_BYTES 11,047,010,304 PER RANK: the engine measured "KV cache size:
-#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
+#     798,280 tokens; Maximum concurrency for 262,144 tokens per request:
-#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
+#     3.05x". The pool holds all three full windows with an 11,848-token
-#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
+#     margin. Measured decode at the winner shape: 39.38/38.79/36.42 tok/s
-#     free; 7.2 GiB boots).
+#     (166/1087/4064 ctx).
-#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
+#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
-#     — the stock flag quantizes them together, which is exactly what was
+#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
-#     requested here). Accepted by the attention path; qualified with a
+#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
-#     completion probe.
+#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
+#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
+#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
+#     the gather win) — traded deliberately for clean cache + 262k windows.
+#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
+#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
+#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
+#     era). Do not drop it.
+#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
+#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
+#     and arena. 1024 keeps the workspace floor so the arena stays max.
 #   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
 #     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste; prefetch off is the tuned
+#     roughly half the gather traffic was waste. Never re-add
-#     incumbent and the +40-51% decode came from deleting the flag. Never
+#     --expert-prefetch without a fresh A/B.
-#     re-add --expert-prefetch without a fresh A/B.
 #   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
 #     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics (repetition loops, early halts, invalid measurements).
+#     numerics.
-# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
-# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
 # prefixed FRANK_PRIMARY_ so none can leak.
+#
 
 # --- runtime ----------------------------------------------------------------
 # Digest-pinned: the floating :latest tag moved once already. Local alias
@@ -53,9 +60,11 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
 # --- qualified engine shape ---------------------------------------------------
-FRANK_PRIMARY_MAXLEN=200000
+FRANK_PRIMARY_MAXLEN=262144
 FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=7730934784
+FRANK_PRIMARY_KV_BYTES=11047010304
-FRANK_PRIMARY_KV_DTYPE=fp8
+FRANK_PRIMARY_KV_DTYPE=auto
-FRANK_PRIMARY_SPEC=4
+FRANK_PRIMARY_MNBT=1024
+FRANK_PRIMARY_GATHER_CHUNKS=32
+FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98

scripts/serve-frank-primary.sh

@@ -11,12 +11,14 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
+# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
+# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
+# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
+# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
+# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
-# precision — the tuned incumbent deletes it), TunableOp off.
+# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
+# expert prefetch OFF (36% precision measured), TunableOp off.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
@@ -100,17 +102,19 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --cap-add SYS_PTRACE \
   --device /dev/kfd --device /dev/dri --device /dev/mem \
   -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
+  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
   "$FRANK_PRIMARY_IMAGE" /models \
   --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
   --tensor-parallel-size 2 \
   --max-model-len "$FRANK_PRIMARY_MAXLEN" \
   --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
   --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
   --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
   --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
   --trust-remote-code \
   > "$LOG" 2>&1
 

tests/frank_primary_unit.bats

@@ -2,25 +2,23 @@
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
 # This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
-# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
+# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
-# reversed the failed GLM/DFlash2 experiment): the serve script drives
+# script drives `podman run` DIRECTLY (the image entrypoint execs
-# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
+# `vllm serve`), with every engine flag qualified by boot probe and carried
-# engine flag qualified by boot probe and carried in the repo env file. The
+# in the repo env file. The seam pins what the flags alone do not say: the
-# seam pins what the flags alone do not say: the KV qualification law
+# KV qualification law (11.05 GiB/rank boots bf16 at 798,280 tokens / 3.05x
-# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
+# concurrency; the old 7.2 GiB ceiling was the MTP4 workspace, not VRAM),
-# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
+# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
-# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
+# clean cache + 262k windows), the automatic-offload law (the fork plans
-# together), the automatic-offload law (the fork plans the tiered-expert
+# the tiered-expert offload itself; no flag exists to set), the self-healing
-# offload itself; no flag exists to set), the self-healing law (the run
+# law (the run returns before the engine is up, so the serve script must
-# returns before the engine is up, so the serve script must wait on the
+# wait on the container for Restart=on-failure to mean anything), the
-# container for Restart=on-failure to mean anything), the memlock law that
+# memlock law that makes the pinned host rows legal, the deployer's
-# makes the pinned host rows legal, the deployer's image/payload gates with
+# image/payload gates with the false-green MainPID check, and the measured
-# the false-green MainPID check, and the measured laws: expert prefetch
+# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
-# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
+# no-spec family measured fastest once the prefetch tax was removed; MTP
-# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
+# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
-# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
+# MNBT 1024 (Frank's lock), and NO tool dialect.
-# carried the prefetch tax), and NO tool dialect (the GLM era added
-# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
 
 load helpers
 
@@ -76,19 +74,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }
 
 @test "the qualified engine shape is in the env file" {
-    # 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
+    # 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, util 0.98,
-    # 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
+    # KV_BYTES 11,047,010,304 PER RANK boots at "KV cache size: 798,280
-    # reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
+    # tokens" and reports max concurrency 3.05x for 262,144-token requests
-    # PER RANK — sizing it to the global requirement OOMs the worker
+    # — all three full windows fit with an 11,848-token margin. The old
-    # (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
+    # "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's
-    # full-length 200k contexts do NOT fit: four clients share the ~730k
+    # workspace reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank
-    # token pool and each runs below 200k when all are deep.
+    # and boots clean.
-    grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
@@ -97,7 +98,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
     grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
     grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-    grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
+    ! grep -q -- '--speculative-config' "$SERVE"
     grep -q -- '--tensor-parallel-size 2' "$SERVE"
     grep -q -- '--enable-prefix-caching' "$SERVE"
     grep -q -- '--trust-remote-code' "$SERVE"
@@ -125,28 +129,46 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
 }
 
-@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
+@test "speculative decode is ABSENT BY MEASUREMENT" {
-    # 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
+    # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
-    # tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
+    # spec family against no-spec at matched geometry: no-spec measured
-    # baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
+    # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
-    # 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
+    # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
-    # (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
+    # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
-    # the prefetch tax and is retired with it. ngram-mod never booted on this
+    # never completed; until one shows a real win, no spec is the incumbent.
-    # model (fork bug: PLE inputs prepared only on the MTP path).
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
+    ! grep -q -- '--speculative-config' "$SERVE"
-    grep -q -- '--speculative-config' "$SERVE"
 }
 
-@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
+@test "bf16 KV is PRESENT — clean cache won the trade" {
-    # --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
+    # --kv-cache-dtype auto = bf16 attention KV. Doubles attention-KV
-    # them together; a V-only quant needs a fork patch). The GLM-era MLA
+    # bytes/token (measured 14,047 vs 7,926) and costs ~14% decode vs fp8
-    # concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
+    # at 4064 ctx (49.25 -> 36.42 with the gather win) — traded deliberately
-    # to this arm: qwen's attention path accepted the dtype and a completion
+    # for clean cache and 262k windows (Frank's call). The fp8 shape is
-    # probe qualified it 2026-09-29.
+    # restorable from git tag frank-primary-fp8-mtp-20261001.
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
     grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
 }
 
+@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
+    # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
+    # sweep's biggest lever, +18.7% decode vs default geometry, and the win
+    # GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
+    # g64x128 is past the peak; g8x16 mild. They are container ENV (the
+    # fork's knobs), not engine flags — passed through with -e.
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+}
+
+@test "MNBT is locked at 1024" {
+    # Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s);
+    # 1024-4096 are interchangeable on decode; 12288 costs 2.5% plus arena.
+    # 1024 keeps the workspace floor so the expert arena stays max.
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+}
+
 @test "expert prefetch is absent BY LAW, not by omission" {
     # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
     # prefetched slabs (36% precision) — roughly half the gather traffic was