frank-primary: bf16 no-spec cutover (262144x3, gather 32x64, MNBT 1024)

Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired
to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache
unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank
(engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token
requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth).
Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM;
spec decode absent by measurement; bf16 trade documented (clean cache +
262k windows for ~14% decode at 4064 ctx).

scripts/serve-frank-primary.sh

@@ -11,12 +11,14 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
+# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
+# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
+# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
+# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
+# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
+# expert prefetch OFF (36% precision measured), TunableOp off.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
@@ -100,17 +102,19 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --cap-add SYS_PTRACE \
   --device /dev/kfd --device /dev/dri --device /dev/mem \
   -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
+  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
   "$FRANK_PRIMARY_IMAGE" /models \
   --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
   --tensor-parallel-size 2 \
   --max-model-len "$FRANK_PRIMARY_MAXLEN" \
   --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
   --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
   --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
   --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
   --trust-remote-code \
   > "$LOG" 2>&1