frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)

Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).

Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.

Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.

Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).

Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.

Rollback: 2339fae (qwen FP8 + MTP4).

scripts/serve-frank-primary.sh

@@ -1,27 +1,30 @@
 #!/usr/bin/env bash
 # serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
+# Intel/GLM-5.3-Flash-W4A16-AutoRound, 2x R9700, dankserver :7770).
+#
+# 2026-09-30 cutover from orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4.
+# Rollback: commit 2339fae carries the whole pre-cutover scripts/ tree; restore
+# it and deploy. The qwen payload and the R9V state dirs stay on disk.
 #
 # The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
+# `vllm serve "$@"`, so the run line below IS the engine argv.
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# Why this payload runs the offload path at all: 170 GiB of int4 experts cannot
+# fit in 64 GiB of VRAM, so the engine keeps 12,096 experts per rank in pinned
+# host memory and serves them through a 2,383-slot VRAM LRU arena. The MoE
+# kernel is libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod), reached because the
+# checkpoint's quant_method "auto-round" is overridden to "inc" by the fork.
+#
+# Shape: MAXLEN 131,072 x 2 seqs, bf16 KV, util 0.98, MTP OFF, prefetch OFF.
+# Measured 2026-09-30: decode 9.9-10.0 tok/s, prefill 918 tok/s warm. The
+# 35 tok/s target is NOT met — the bound is the fork's expert-gather read path
+# (8.2 GB/s effective vs the link's 22.74 GB/s); see frank-primary.env and the
+# runbook's arithmetic section.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget for the 73.64 GiB/rank of pinned
+# host expert rows. A plain shell session also works, but the unit is the
+# fleet-parity boot-persistent shape.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -44,8 +47,9 @@ set +a
 : "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
 : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
 : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
+: "${FRANK_PRIMARY_NBATCH:?FRANK_PRIMARY_NBATCH must be set}"
+: "${FRANK_PRIMARY_TOOLPARSER:?FRANK_PRIMARY_TOOLPARSER must be set}"
+: "${FRANK_PRIMARY_REASONINGPARSER:?FRANK_PRIMARY_REASONINGPARSER must be set}"
 # Capture podman's output BEFORE grepping: under `set -o pipefail`, the
 # classic `podman images | grep -q` pattern races — grep exits on its first
 # match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
@@ -106,14 +110,27 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --tensor-parallel-size 2 \
   --max-model-len "$FRANK_PRIMARY_MAXLEN" \
   --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_NBATCH" \
   --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
-  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --enable-expert-offload \
+  --expert-precise-memory \
   --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  --enable-auto-tool-choice \
+  --tool-call-parser "$FRANK_PRIMARY_TOOLPARSER" \
+  --reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER" \
   --trust-remote-code \
   > "$LOG" 2>&1
 
+# NOT passed, deliberately (each measured on this payload — see the env file):
+#   --kv-cache-dtype             fp8 disables the sparse MLA backend's rope-free
+#                                Triton path, which is the only MLA decode on
+#                                gfx12 (aiter's MLA op is gated on CDNA>2).
+#   --speculative-config         MTP head loads and accepts well, but buys zero
+#                                decode and collapses prefill on an
+#                                offload-bound shape.
+#   --expert-prefetch*           36% precision; deleting it bought +40-51%.
+#   --kv-cache-memory-bytes      the plan's own reserve covers this shape.
+
 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
 # Restart=on-failure is real. (systemd stop goes through ExecStop in the