frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)

Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group 128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB). It is the only W4A16 checkpoint this fork can serve with expert offload: its quant_method "auto-round" is overridden to "inc", which routes the MoE to libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16 build of the same model resolves to TritonWNA16Experts, which has no offload recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory). Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29 qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it disables the sparse MLA backend's rope-free Triton path, the only MLA decode gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload --expert-precise-memory, prefix caching, glm47 tool + reasoning parsers, MTP and expert prefetch absent by measurement. Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB = 2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank). THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry (R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves 0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at 22.74 GB/s — the same structural gather-path gap the qwen campaign found. Growing the arena 13.08 -> 14.51 GiB changed decode by nothing. Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode unchanged and prefill collapses 918 -> 255 — a verification forward gathers the union of the batch's experts), NBATCH 8192 (identical prefill). Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve model that under-counts GLM's per-sequence KDA state; not needed by this shape), runbook, AGENTS.md row + section, both indexes, work log. Rollback: 2339fae (qwen FP8 + MTP4).
2026-09-30 04:22:11 -07:00
parent 2339faeaa4
commit f2bcf84e88
12 changed files with 668 additions and 224 deletions
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 |
 | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |

@@ -245,7 +245,7 @@ merely wedged kernel usually still answers ARP.
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
   NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -423,86 +423,62 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].

-### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
+### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE)

-The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
-prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
-the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
-with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
-PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
-**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
-n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
-template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
-`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
-over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
-llama.cpp b11030 the same day) at Frank's direction.
+The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`
+— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured)
+serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4,
+group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors,
+113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
+**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching,
+**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape,
+which itself replaced the R9V radiance fork on 2026-09-29.

-**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
-run qualified the `R9V_*` config into
-`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
-file `scripts/frank-primary.env` carries ONLY production overrides
-(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
-`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
-`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
-(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
-quantizes BOTH fused hyper-connection shards, so the shard-uniformity
-check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
-overlay's env.always injects the uncensored 160-slot cache geometry, which
-the q8 80-slot tiered cache rejects). Never copy tuned values from the
-state into the repo — it would drift from the qualification.
+**The payload is chosen for the offload path, not for the quant.** 170 GiB of
+int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and
+the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16
+GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's
+`quant_method` is `auto-round`, which the fork overrides to `inc`. A
+compressed-tensors W4A16 checkpoint of the same model resolves to
+`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to
+serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
+12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.

-**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
-single dict — one method per engine, NO chaining — so the llama.cpp
-`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
-(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
-architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
-single method, not a chainable one.
+**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
+prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is
+a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB
+of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry
+(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the
+fork's gather read path delivers **~8.2 GB/s against a link measured at
+22.74 GB/s** — the same structural gap the qwen campaign found. Growing the
+arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so
+residency stopped being the binding term; 0.89 hit rate would be needed for 35.

-**Serving = the fork's own launcher through a docker→podman shim.** The R9V
-tooling is docker-pinned; the shim at
-`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
-three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
-the image by its OCI manifest digest while podman keys by config digest but
-preserves the manifest digest in `RepoDigests` — the shim answers
-`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
-refs to `repo@sha256:X`; (2) podman inspect answers
-`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
-(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
-and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
-/`ServerVersion` info fields — the shim answers them.
+**MTP works here and is still rejected.** The head loads
+(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted
+length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was
+unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**:
+a verification forward gathers the *union* of the batch's experts, so spec
+decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB
+of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA
+backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2,
+so the MTP draft step demands an op that cannot exist on gfx12
+(`no attribute 'rocm_aiter_mla_decode_fwd'`).

 **Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
-54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
-quadlet would be capped by the user manager's limit).
-The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
-(ready + paths), the pinned image (RepoDigests), the PLE byte count
-(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
-clears a stale container, then execs the fork's `scripts/launch.sh` through
-the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
-without the wait a crashed engine is a green unit; `docker wait` cannot stop
-the container child, hence `ExecStop=podman stop -t 120
-frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
-PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
-restarts the unit and health-gates with the MainPID check (the false-green
-stray law).
+`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it),
+`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives
+`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the
+digest-pinned image, the payload's `config.json` and its exact 34-shard count
+and the GPU device nodes, clears the stale container identities, then **blocks
+on `podman wait`** so a crash is a nonzero ExecStart. The deployer's
+`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts
+the unit and health-gates with the MainPID check (the false-green stray law).
+Boot cost ~7 min warm (~15 min cold); health returns 503 while loading.

-Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
-payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
-battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
-length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
-mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
-round-trip + template probe re-verified on :7770 after the deploy. No
-throughput bench was run (Frank declined); the probe-era 11 tok/s decode
-was residency-bound (halved expert headroom at double the bytes/expert),
-not the production number.
-
-Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
-Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
-`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
-and the template keys, redeploy (state + `restore-config-20260927.env` +
-payload intact on disk) — see
-[[frank-primary-r9v-radiance-qwen38-uncensored]].
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
+Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and
+deploy; the qwen payload and the R9V state dirs stay on disk.

 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -851,7 +827,7 @@ Active:
 |---------|---------|
 | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
 | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
+| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

@@ -1217,8 +1217,8 @@ deploy_llm_small() {
 # (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
 # sha256 verification was the fetch script's job at download time, never
 # re-hashed here.
-LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+LLM_PRIMARY_CONTAINER=frank-primary-glm
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"

 deploy_llm_primary() {
    # Refuse rather than restart-loop: a unit whose image or payload is
@@ -1271,7 +1271,8 @@ deploy_llm_primary() {
    # races conmon's --rm cleanup, and the orphan holds the port and GPUs.
    # Stop the unit, then remove the containers BEFORE waiting on the port.
    # The legacy identities (the retired R9V container and the ad hoc
-    # experiment container qwen-fp8) hold the same GPUs and die here too.
+    # experiment container qwen-fp8, the ad hoc glm-probe) hold the same GPUs
+    # and die here too.
    run_privileged systemctl stop frank-primary.service 2>/dev/null || true
    local ctr
    for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
@@ -1,17 +1,24 @@
 #!/usr/bin/env bash
 # Payload inventory/verification for frank-primary:
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
-# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
-# became frank-primary on 2026-09-29.
+# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128,
+# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv /
+# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the
+# int4 payload that became frank-primary on 2026-09-30.
 #
-#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
+#   bash scripts/download-frank-primary.sh --verify-only   (the default)
 #   bash scripts/download-frank-primary.sh --fetch
 #
-# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
-# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
-# manifest cannot be built without the authed endpoint. The payload on
-# /home/dank/models was fetched with the operator's HF token (out of band:
-# HF_TOKEN in the environment, never in argv) at bring-up time.
+# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at
+# bring-up). The LFS oids are still not verified byte-exactly here — the tree
+# API masks them — so integrity rests on the weight-map index, the shard count
+# and the loader's own tensor-shape checks at boot.
+#
+# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which
+# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod
+# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A
+# compressed-tensors W4A16 checkpoint of the same model (e.g.
+# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which
+# has no offload recipe, and refuses to serve at all on this host.
 #
 # Modes:
 #   --fetch       resume any missing/truncated shard via `hf download` with
@@ -24,16 +31,17 @@
 #                 (slow, ~10 min for 173 GiB).
 set -euo pipefail

-REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
-EXPECTED_SHARDS=131
+REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound"
+DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound"
+EXPECTED_SHARDS=34

 mode="verify"
 [[ "${1:-}" == "--fetch" ]] && mode="fetch"

 if [[ "$mode" == "fetch" ]]; then
-    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
-    export HF_TOKEN
+    # The repo is public, so no token is required; if one is present it is taken
+    # from the environment and never from argv (a -H header is readable in
+    # /proc/<pid>/cmdline — the 2026-09-16 leak).
    mkdir -p "$DEST"
    hf download "$REPO" --local-dir "$DEST"
 fi
@@ -81,7 +89,7 @@ if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
    echo "hashing every shard (slow)..."
    cd "$DEST"
    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
-    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
+    sha256sum model-*-of-00034.safetensors > /dev/null  # presence check; content hash list printed on demand
 fi

 [[ "$fail" -eq 0 ]] || exit 1
@@ -1,34 +1,53 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
-# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
-# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
-# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
-# digest-pinned image the 2026-09-29 tuning campaign measured — serving
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
-# the MTP4 drafter on the two R9700s.
+# 2026-09-30 cutover from orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (MTP4) to
+# Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet has served
+# through the fork's W4A16 expert-offload path. ROLLBACK: the pre-cutover qwen
+# state is commit 2339fae ("frank-primary: fix the SIGPIPE race in both
+# image-digest gates") — restore scripts/ from it and deploy; the qwen payload
+# (173 GiB) and the R9V state dirs are untouched on disk.
 #
-# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
-#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
-#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
-#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
-#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
-#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
-#     free; 7.2 GiB boots).
-#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
-#     — the stock flag quantizes them together, which is exactly what was
-#     requested here). Accepted by the attention path; qualified with a
-#     completion probe.
-#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste; prefetch off is the tuned
-#     incumbent and the +40-51% decode came from deleting the flag. Never
-#     re-add --expert-prefetch without a fresh A/B.
+# QUALIFIED 2026-09-30 by boot probe + bench against the live engine:
+#   * THE 35 tok/s TARGET IS NOT MET: decode 9.9-10.0 tok/s, prefill 918 tok/s
+#     (warm, 2.6k-token prompt) at 131,072 ctx x 2 seqs. See the runbook's
+#     "Why 35 tok/s is out of reach on this hardware" — the bound is the fork's
+#     expert-gather read path (8.2 GB/s effective vs the link's 22.74 GB/s),
+#     the same structural gap the 2026-09-29 qwen campaign measured.
+#   * Arena 14.51 GiB = 2,383 slots x 6.234 MiB per rank; 12,096 host-backed
+#     experts per rank, 73.64 GiB pinned. Live per-layer telemetry
+#     (R4D_LRU_TELEMETRY=1, 2026-09-30): hit rate 0.53-0.66, ~500-730 installs
+#     vs ~2,000 routed per layer over the window.
+#   * --expert-precise-memory is ON: it replaces the 6.25 GiB default
+#     compile-headroom figure with the registry's fitted 5.46 GiB, buying
+#     1.43 GiB of cache (13.08 -> 14.51 GiB). Decode measured flat across that
+#     change (10.03 -> 9.88 tok/s), which is the evidence that the hit rate
+#     saturates near 0.6 and residency is no longer the binding term.
+#   * KV dtype stays bf16/AUTO — fp8 KV is NOT legal on this model. It disables
+#     _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend (the
+#     predicate requires a non-fp8 cache), and that rope-free Triton path is the
+#     only MLA decode available on gfx12: aiter's MLA decode op is registered
+#     under @if_aiter_supported, i.e. gated on CDNA>2. With fp8 KV the MTP draft
+#     step dies with "'_OpNamespace' 'vllm' object has no attribute
+#     'rocm_aiter_mla_decode_fwd'".
+#   * MTP4 spec decode is ABSENT DELIBERATELY, unlike the qwen shape. It WORKS
+#     here (the MTP head loads, architecture resolves to Glm5NextMTPModel, and
+#     acceptance is excellent: mean accepted length 3.76-4.90, per-position
+#     0.92/0.85/0.69/0.58, 69-97%) — but it buys nothing on an offload-bound
+#     deployment: measured decode 9.6-11.2 tok/s (vs 10.0 without) while prefill
+#     COLLAPSED 918 -> 255 tok/s. The verification forward gathers the union of
+#     the batch's experts, so spec decode does not amortize a residency-bound
+#     gather the way it amortizes compute. It also costs 2.6 GiB of arena.
+#     Reinstate only with a fresh A/B on a non-offload-bound shape.
+#   * PREFETCH IS ABSENT BY LAW, unchanged from the qwen campaign: expert-cache
+#     telemetry measured the prefetcher at 36% precision and deleting the flag
+#     bought +40-51% decode. Never re-add --expert-prefetch without a fresh A/B.
 #   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
 #     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
 #     numerics (repetition loops, early halts, invalid measurements).
-# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
-# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
+#   * KV is NOT capped explicitly: the fork's plan reserve (1.76 GiB at 131,072)
+#     already covers what max_model_len needs for this shape. It does NOT for an
+#     MTP shape — see the runbook's fork-bug section and
+#     scripts/patches/gpu_worker-kv-cap.patch.
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
 # prefixed FRANK_PRIMARY_ so none can leak.
@@ -36,26 +55,57 @@
 # --- runtime ----------------------------------------------------------------
 # Digest-pinned: the floating :latest tag moved once already. Local alias
 # qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-# bytes in rootless storage.
+# bytes in rootless storage. Same image as the qwen cutover — the W4A16 path
+# (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM) ships in it.
 FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea

 # --- payload ----------------------------------------------------------------
-FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
-FRANK_PRIMARY_MODEL_SHARDS=131
+# Intel AutoRound W4A16: symmetric int4, group 128, packing "auto_round:auto_gptq".
+# quantization_config.quant_method is "auto-round", which the fork overrides to
+# "inc" (inc.py:364) — that is what routes the MoE to R4dW4a16MoEMethod instead
+# of the compressed-tensors path's Triton experts, which has no offload recipe.
+# 34 numbered shards + model_extra_{conv,tensors}.safetensors, 113,074 tensors.
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound
+FRANK_PRIMARY_MODEL_SHARDS=34

 # --- identity ---------------------------------------------------------------
 # The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
-FRANK_PRIMARY_CONTAINER=frank-primary-qwen
+FRANK_PRIMARY_CONTAINER=frank-primary-glm
 FRANK_PRIMARY_SERVED_NAME=frank-primary
 FRANK_PRIMARY_PUBLISH=7770:7770
-# Containers removed before every start: the retired R9V identity and the ad
-# hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+# Containers removed before every start: the pre-cutover qwen identity (it holds
+# the same port and GPUs), the retired R9V identity, and the ad hoc experiment
+# identity.
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"

 # --- qualified engine shape ---------------------------------------------------
-FRANK_PRIMARY_MAXLEN=200000
-FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=7730934784
-FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_SPEC=4
+# 131,072 x 2 seqs. The window is NOT chosen for capacity: the engine's own
+# plan reserve is one max_model_len request, and below ~92k tokens the fork's
+# reserve model (13,600 B/token + a small mamba term) falls short of what the
+# engine actually needs for this model's per-sequence KDA state, which makes the
+# boot refuse ("2.07 GiB needed vs 1.74 GiB available"). 131,072 clears it with
+# the plan's own cap intact. KV cache as built: 138,561 tokens, 1.06x
+# concurrency for a 131,072-token request.
+FRANK_PRIMARY_MAXLEN=131072
+FRANK_PRIMARY_SEQS=2
+# 0.99 fails boot ("Free memory on device cuda:1 ..."); 0.98 leaves the
+# compile/capture headroom the plan models.
 FRANK_PRIMARY_UTIL=0.98
+# Prefill chunk. 2,048 is the qualified value; the expert union a chunk routes
+# to saturates quickly, so the chunk size is an install-amortization knob, not a
+# residency one. Measured 2026-09-30: NBATCH 8,192 boots (arena 12.51 GiB) and
+# leads to the SAME warm prefill (922 vs 918 tok/s) — no reason to pay its
+# 2 GiB of profile buffer.
+FRANK_PRIMARY_NBATCH=2048
+# GLM's own tool-call dialect. Without --enable-auto-tool-choice the endpoint
+# rejects tool_choice "auto" outright ("\"auto\" tool choice requires
+# --enable-auto-tool-choice and --tool-call-parser to be set"), which breaks
+# every tool-using agent on the endpoint. Verified 2026-09-30: a get_weather
+# prompt returns tool_calls with finish_reason "tool_calls".
+FRANK_PRIMARY_TOOLPARSER=glm47
+# GLM's reasoning dialect. Without it the model's thinking leaks into `content`
+# (measured: content began with the answer, then "Let me verify: ... = 391"
+# followed by a bare "</think") and the reasoning field comes back empty. With
+# it, the chain of thought lands in `reasoning` — the vLLM field name the fleet
+# documents (llama.cpp endpoints use reasoning_content).
+FRANK_PRIMARY_REASONINGPARSER=glm47
@@ -1,26 +1,24 @@
-# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
-# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
-# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
-# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
-# launcher/shim/setup-state machinery is retired and the state dirs stay on
-# disk as rollback).
+# frank-primary.service — tcclaviger RDNA4 vLLM endpoint
+# (Intel/GLM-5.3-Flash-W4A16-AutoRound int4, 131,072 x 2 seqs, bf16 KV, TP=2,
+# expert offload to pinned host RAM, 34-shard payload) on the 2x R9700 pair,
+# dankserver :7770. 2026-09-30 cutover from orcarouter
+# Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4; commit 2339fae is the pre-cutover
+# state and the rollback.
 #
 # Same port, same served alias; the harness models.yml id is unchanged.
 # The serve script drives `podman run` directly (the image entrypoint execs
 # `vllm serve`) and then waits on the container, so a crashed engine is a
 # nonzero ExecStart and Restart=on-failure is real.
 #
-# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
+# WHY A ROOT UNIT (not a rootless quadlet): the expert-offload shape pins
+# 73.64 GiB per rank of host RAM (the expert host rows). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
 # login session on this host carries 8 MiB and a user quadlet is capped by the
 # user manager's own hard limit — raising that means restarting
 # user@1000.service and bouncing every user container on the host. A root unit
 # grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
 # stays in dank's rootless podman storage.
 [Unit]
-Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
+Description=frank-primary — tcclaviger vLLM GLM-5.3-Flash int4 W4A16 on 2x R9700
 After=network-online.target
 Wants=network-online.target

@@ -35,9 +33,9 @@ Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-pr
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
 # explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# ~15 min cold (measured 2026-09-29), faster warm.
+ExecStop=/usr/bin/podman stop -t 120 frank-primary-glm
+# Boot cost: int4 payload page-in plus torch compile/graph capture —
+# ~7 min measured 2026-09-30 on a warm page cache, longer cold.
 TimeoutStartSec=1800
 Restart=on-failure
 RestartSec=10
@@ -0,0 +1,65 @@
+# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
+#
+# STATUS: NOT part of the deployed frank-primary shape. It exists because the
+# fork's plan under-models this model's KV, and it is the enabler for a future
+# MTP attempt (the production GLM shape runs without MTP and boots fine on the
+# plan's own reserve).
+#
+# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
+# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
+# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
+# cache at that reserve, so the plan's own arithmetic can make a legal
+# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
+#   - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB  -> boots
+#   - MTP4,  max_model_len 131072: reserve 1.74 GiB <  need 2.07 GiB  -> refuses
+#   - MTP4,  max_model_len 262144: reserve 3.43 GiB <  need 3.62 GiB  -> refuses
+#     ("To serve at least one request with the model's max seq len ...")
+# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
+# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
+# where the KV reserve would eat a third of the expert arena.
+#
+# THE FIX: let an explicit operator KV request raise the cap. The compile and
+# graph-capture headroom still comes off the top, so the blast radius is one
+# flag the operator chose.
+#
+# APPLY (into a host dir that is then bind-mounted read-only over the image):
+#   IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+#   D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
+#   podman run --rm --entrypoint cat $IMAGE \
+#     /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
+#     > $D/vllm/v1/worker/gpu_worker.py
+#   ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
+#   # then add to the podman run:
+#   #   -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
+#   #   -e CLAV_KV_RESERVE_FLOOR_GIB=<needed>   (or pass --kv-cache-memory-bytes)
+#
+# Base file sha256 (the pristine file out of the pinned image):
+#   1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
+# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
+# these bytes exactly):
+#   80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
+#
+--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
+@@ -824,6 +824,21 @@
+         expert_cfg = self.vllm_config.offload_config.expert
+         if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
+             cap = int(expert_cfg.planned_kv_reserve_bytes)
+            # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
+            # 13600 B/token + a small mamba term, but this model also carries a
+            # per-sequence KDA state and (under MTP) the draft's KV, so the
+            # reserve can sit BELOW what max_model_len legitimately needs -- and
+            # the cap then makes a legal config unbootable (measured: need
+            # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
+            # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
+            # is the operator's word: let it raise the cap. The compile/capture
+            # headroom still comes off the top.
+            _explicit = self.cache_config.kv_cache_memory_bytes
+            _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
+            if _explicit:
+                cap = max(cap, int(_explicit))
+            if _floor:
+                cap = max(cap, int(float(_floor) * (1 << 30)))
+             if self.available_kv_cache_memory_bytes > cap:
+                 logger.info_once(
+                     "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
@@ -1,27 +1,30 @@
 #!/usr/bin/env bash
 # serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
+# Intel/GLM-5.3-Flash-W4A16-AutoRound, 2x R9700, dankserver :7770).
+#
+# 2026-09-30 cutover from orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4.
+# Rollback: commit 2339fae carries the whole pre-cutover scripts/ tree; restore
+# it and deploy. The qwen payload and the R9V state dirs stay on disk.
 #
 # The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
+# `vllm serve "$@"`, so the run line below IS the engine argv.
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# Why this payload runs the offload path at all: 170 GiB of int4 experts cannot
+# fit in 64 GiB of VRAM, so the engine keeps 12,096 experts per rank in pinned
+# host memory and serves them through a 2,383-slot VRAM LRU arena. The MoE
+# kernel is libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod), reached because the
+# checkpoint's quant_method "auto-round" is overridden to "inc" by the fork.
+#
+# Shape: MAXLEN 131,072 x 2 seqs, bf16 KV, util 0.98, MTP OFF, prefetch OFF.
+# Measured 2026-09-30: decode 9.9-10.0 tok/s, prefill 918 tok/s warm. The
+# 35 tok/s target is NOT met — the bound is the fork's expert-gather read path
+# (8.2 GB/s effective vs the link's 22.74 GB/s); see frank-primary.env and the
+# runbook's arithmetic section.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget for the 73.64 GiB/rank of pinned
+# host expert rows. A plain shell session also works, but the unit is the
+# fleet-parity boot-persistent shape.
 set -euo pipefail

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -44,8 +47,9 @@ set +a
 : "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
 : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
 : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
+: "${FRANK_PRIMARY_NBATCH:?FRANK_PRIMARY_NBATCH must be set}"
+: "${FRANK_PRIMARY_TOOLPARSER:?FRANK_PRIMARY_TOOLPARSER must be set}"
+: "${FRANK_PRIMARY_REASONINGPARSER:?FRANK_PRIMARY_REASONINGPARSER must be set}"
 # Capture podman's output BEFORE grepping: under `set -o pipefail`, the
 # classic `podman images | grep -q` pattern races — grep exits on its first
 # match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
@@ -106,14 +110,27 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
  --tensor-parallel-size 2 \
  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_NBATCH" \
  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
-  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --enable-expert-offload \
+  --expert-precise-memory \
  --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  --enable-auto-tool-choice \
+  --tool-call-parser "$FRANK_PRIMARY_TOOLPARSER" \
+  --reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER" \
  --trust-remote-code \
  > "$LOG" 2>&1

+# NOT passed, deliberately (each measured on this payload — see the env file):
+#   --kv-cache-dtype             fp8 disables the sparse MLA backend's rope-free
+#                                Triton path, which is the only MLA decode on
+#                                gfx12 (aiter's MLA op is gated on CDNA>2).
+#   --speculative-config         MTP head loads and accepts well, but buys zero
+#                                decode and collapses prefill on an
+#                                offload-bound shape.
+#   --expert-prefetch*           36% precision; deleting it bought +40-51%.
+#   --kv-cache-memory-bytes      the plan's own reserve covers this shape.
+
 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
 # Restart=on-failure is real. (systemd stop goes through ExecStop in the
@@ -0,0 +1,224 @@
+---
+type: runbook
+title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - glm5.3
+  - w4a16
+  - int4
+  - expert-offload
+  - expert-cache
+  - autoround
+  - vllm
+timestamp: 2026-09-30
+---
+
+# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+
+> **Question this page answers.** Can the fleet serve the 320B-class
+> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
+> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
+> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
+> reachable** with this payload on this hardware. The bound is not the model,
+> the quant, or the link: it is the fork's expert-gather read path, and it is
+> measured, not inferred.
+
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
+same hardware, the same gather-path wall, found first on qwen),
+[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
+[[VLLM Index]], [[LLM/index|LLM fleet]].
+
+## The deployed shape
+
+| term | value |
+| --- | --- |
+| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
+| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
+| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
+| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
+| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
+| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
+| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
+
+```mermaid
+graph LR
+    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
+    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
+    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
+    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
+    D --> F[73.64 GiB pinned host rows / rank]
+    E --> G[decode step]
+    F -->|8.2 GB/s effective| G
+```
+
+## Why this payload and not another
+
+The W4A16 fast path on this fork is reachable **only** through the INC
+(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
+for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
+their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
+offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
+cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
+
+The chain that makes Intel's checkpoint work, with the exact evidence:
+
+```
+[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
+[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
+[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
+[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
+[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
+                           host 73.64 GiB pinned, host rows prepacked on the GPU
+```
+
+`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
+format-agnostic but requires the experts class to expose the hot/cold split.
+
+## The numbers, and the wall
+
+Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
+
+| depth | prefill | decode |
+| --- | --- | --- |
+| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
+| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
+| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
+
+Decode is flat across context and across every arena change — the signature of a
+per-step cost that is not KV- or residency-driven at the margin.
+
+### The arithmetic that bounds it
+
+Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
+**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.
+
+| hit rate | miss traffic | step @ 8.2 GB/s | decode |
+| --- | --- | --- | --- |
+| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
+| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
+| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |
+
+Two independent measurements pin the missed term:
+
+- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
+  gives the hit rate directly — the LRU is *working*:
+  `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
+  read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
+- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
+  pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
+  36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.
+
+So the step is bounded by the fork's gather read path, and the only other lever
+is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
+73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
+routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
+(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
+need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.
+
+## Measured arms (all 2026-09-30, one boot each)
+
+| arm | decode | prefill | verdict |
+| --- | --- | --- | --- |
+| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
+| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
+| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
+| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
+
+### Why MTP is rejected here (it is not that it fails)
+
+MTP **works** on GLM in this fork: the head loads, the architecture resolves to
+`Glm5NextMTPModel`, and acceptance is excellent —
+
+```
+SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
+Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
+Avg Draft acceptance rate: 69-97%
+```
+
+— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
+prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
+verification forward gathers the **union** of the batch's experts, so spec
+decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
+of arena. On a compute-bound (fully-resident) shape this would invert; here it
+does not.
+
+## Served surface (verified 2026-09-30 against the live engine)
+
+| capability | state | evidence |
+| --- | --- | --- |
+| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
+| completion | yes | `bench_glm.py` runs above |
+| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
+| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
+| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
+
+Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
+requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
+breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
+newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
+
+## Fork traps found (for the engine authors)
+
+1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model.** The
+   `ROCM_AITER_MLA_SPARSE` backend's `_use_rocm_sparse_triton()` requires a
+   non-fp8 KV cache, and that rope-free Triton path is the **only** MLA decode
+   available on gfx12: aiter's MLA decode op is registered under
+   `@if_aiter_supported`, which is gated on **CDNA > 2**
+   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
+   the MTP draft step demands an op that cannot exist here:
+   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
+   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
+   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
+   CDNA check, not the env var.)
+2. **The plan's KV reserve under-models this model, and the cap enforces the
+   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
+   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
+   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
+   `--max-model-len` becomes unbootable:
+   `ValueError: To serve at least one request with the model's max seq len
+   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
+   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
+   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
+   fixed) — where the reserve would eat a third of the arena.
+   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
+   request raise the cap; **it is not part of the deployed shape**, which boots
+   on the plan's own reserve.
+3. **A window is forced, not chosen.** The production `max_model_len 131072` is
+   what the reserve model requires for this payload (below ~92k the boot
+   refuses); it is not a capacity decision.
+4. **Expert prefetch remains poison** — unchanged from the qwen campaign
+   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
+
+## Operations
+
+```bash
+# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
+# dirs are untouched on disk):
+git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
+./scripts/deploy.sh dankserver --service frank-primary
+
+# payload inventory (34 shards + 2 extras, weight-map index):
+bash scripts/download-frank-primary.sh --verify-only
+
+# live telemetry (needs the profiler window; with_stack=false is mandatory —
+# with stack the stop path hangs and the trace buffer grows to tens of GB):
+podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+
+# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
+```
+
+Boot gate: the engine refuses a legal-looking shape in two measured ways — the
+KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
+64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
+what killed the first two boots at util 0.95.
+
+## Rollback targets
+
+| state | where |
+| --- | --- |
+| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
+| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
@@ -1,10 +1,13 @@
 # LLM Fleet — Knowledge Base

-Documentation for the LLM inference fleet on the two hosts (2026-09-27):
-**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
-serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
-PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
-shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
+Documentation for the LLM inference fleet on the two hosts (2026-09-30):
+**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
+digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
+Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
+Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
+of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
+bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
+Dankclaw runs two llama.cpp endpoints —
 stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
 Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
@@ -22,11 +25,11 @@ graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[R9V radiance-vLLM fork, docker-pinned
-via docker-podman shim
-frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next Q8_0
-TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
+        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+podman run, frank-primary, port 7770]
+        C[GLM-5.3-Flash W4A16 int4
+TP=2, expert offload 73.64 GiB/rank pinned,
+14.51 GiB LRU arena, 131072 ctx, vision, tools]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -58,20 +61,26 @@ frank-small, port 7771]

 ## Active Deployment

- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  runbook (2026-09-27, root systemd unit, port 7770): the R9V
-  radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
-  image bundle `2dac17a2`, driven through a docker→podman shim) serving
-  the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
-  blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
-  (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
-  131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
-  `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
-  inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
-  `reasoning`, parser `qwen3_coder`. The tuned shape lives in the
-  on-host q8xl setup state. Rollback:
-  [[frank-primary-r9v-radiance-qwen38-uncensored]]
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
+  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
+  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
+  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
+  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
+  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
+  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
+  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
+  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
+  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
+  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
+  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
+  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
+  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; payload and R9V state dirs intact on disk, git tag
+  frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
+ [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
  (same-day swap to the Q8_0 payload; rollback target, state + payload
  intact on disk): the R9V radiance-vLLM fork serving
  Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -169,6 +169,18 @@ The LLM inference fleet on the two hosts. Full index:
  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
  slab size)
+- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
+  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
+  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
+  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
+  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
+  checkpoint of the same model reaches TritonWNA16Experts, which has no
+  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
+  35 tok/s target is NOT met** — the step is bounded by the fork's gather
+  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
+  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
+  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
+  `glm47` tool/reasoning parser wiring
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
@@ -1,5 +1,47 @@
 # Bundle Change Log

+## 2026-09-30
+- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
+  the 35 tok/s target, and the shortfall is measured rather than inferred.**
+  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
+    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
+    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
+    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
+    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
+    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
+    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
+    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
+    refused to serve at all. That checkpoint was purged with the other retired
+    payloads.
+  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
+    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
+    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
+    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
+  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
+    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
+    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
+  - **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a
+    `with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per
+    layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective
+    against a link measured at 22.74 GB/s** — the same structural gather-read-path
+    gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB
+    (+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped
+    being the binding term; 35 tok/s would need a 0.89 hit rate.
+  - Measured and rejected: **MTP4** (the head loads, arch resolves to
+    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255,
+    because a verification forward gathers the *union* of the batch's experts);
+    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
+    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
+    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
+  - Fork bug found: the plan's KV reserve under-models this model's
+    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
+    that reserve, making a legal `--max-model-len` unbootable. Shipped
+    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
+    request raise the cap) — NOT needed by the deployed shape.
+  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
+    state dirs are intact on disk.
+

 ## 2026-09-29
 - **frank-large tuning investigation: a measured NEGATIVE result, nothing
@@ -1,19 +1,22 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-29
-# cutover that replaced the R9V radiance fork): the serve script drives
-# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
-# engine flag qualified by boot probe on 2026-09-29 and carried in the repo
-# env file. The seam pins what the flags alone do not say: the qualification
-# law (600k ctx / 3 sessions / q8-q8 KV were MEASURED to fit — 729,545 tokens
-# of capacity at 7.2 GiB per rank — and KV_BYTES is PER RANK), the self-healing
+# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-30
+# cutover to GLM-5.3-Flash W4A16 int4, which replaced the qwen FP8 + MTP4
+# shape): the serve script drives `podman run` DIRECTLY (the image entrypoint
+# execs `vllm serve`), with every engine flag qualified by boot probe on
+# 2026-09-30 and carried in the repo env file. The seam pins what the flags
+# alone do not say: the qualification law (131,072 x 2 seqs at bf16 KV, util
+# 0.98 is what boots; the window is forced up by the fork's KV reserve model,
+# not by capacity), the offload law (170 GiB of experts cannot fit 64 GiB of
+# VRAM — the engine MUST run with --enable-expert-offload), the self-healing
 # law (the run returns before the engine is up, so the serve script must wait
 # on the container for Restart=on-failure to mean anything), the memlock law
-# that makes the tiered-expert pin legal, the deployer's image/payload gates
-# with the false-green MainPID check, and the prefetch law (expert prefetch
-# measured at 36% precision — OFF is the incumbent; a regression that
-# reintroduces it silently costs ~40-50% decode).
+# that makes the 73.64 GiB/rank expert pin legal, the deployer's image/payload
+# gates with the false-green MainPID check, and the three measured-absent
+# flags: expert prefetch (36% precision, costs ~40-50% decode), MTP spec
+# decode (buys zero decode and collapses prefill on an offload-bound shape),
+# and fp8 KV (disables the only MLA decode path gfx12 has).

 load helpers

@@ -53,8 +56,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # outlives the unit and holds both GPUs. The container name must match
    # the env file's identity.
    grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
-    grep -q 'frank-primary-qwen' "$UNIT"
-    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
+    grep -q 'frank-primary-glm' "$UNIT"
+    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-glm$' "$ENV_FILE"
 }

@test "the serve shape is direct podman run, not a launcher or shim" {
@@ -69,32 +72,70 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }

@test "the qualified engine shape is in the env file" {
-    # 2026-09-29 boot probe: MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8
-    # fits — the engine measured "GPU KV cache size: 729,545 tokens,
-    # Maximum concurrency for 200,000 tokens per request: 3.65x" at
-    # KV_BYTES 7.2 GiB per rank. KV_BYTES is PER RANK: sizing it to the
-    # global requirement OOMs the worker (measured twice). fp8 KV is the
-    # requested q8/q8 (e4m3 for both K and V together). MTP4 spec decode,
-    # util 0.98 (0.99 fails the free-memory wall).
-    grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
+    # 2026-09-30 boot probe: 131,072 x 2 seqs at bf16 KV, util 0.98 boots and
+    # serves (measured "GPU KV cache size: 138,561 tokens"). 0.99 fails the
+    # free-memory wall. The window is not a capacity choice: the engine's own
+    # reserve model (one max_model_len request) needs a large L for this
+    # model's per-sequence KDA state, and below ~92k the boot REFUSES with
+    # "needed ... larger than the available KV cache memory". NBATCH pins the
+    # prefill chunk that was measured.
+    grep -q '^FRANK_PRIMARY_MAXLEN=131072$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_NBATCH=2048$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
 }

@test "the serve script wires the engine flags from the env" {
-    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH"' "$SERVE"
    grep -q -- '--tensor-parallel-size 2' "$SERVE"
-    grep -q 'num_speculative_tokens' "$SERVE"
    grep -q -- '--enable-prefix-caching' "$SERVE"
    grep -q -- '--trust-remote-code' "$SERVE"
 }

+@test "the endpoint advertises the GLM tool dialect" {
+    # Without --enable-auto-tool-choice the engine rejects tool_choice "auto"
+    # with a 400 ("requires --enable-auto-tool-choice and --tool-call-parser"),
+    # which breaks every tool-using agent on the endpoint.
+    grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER"' "$SERVE"
+    grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER"' "$SERVE"
+    grep -q '^FRANK_PRIMARY_TOOLPARSER=glm47$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
+}
+
+@test "expert offload is ON — the payload cannot fit VRAM without it" {
+    # 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
+    # the engine has no home for the expert tensors at all; the arena
+    # (14.51 GiB = 2,383 slots) plus 73.64 GiB/rank of pinned host rows is the
+    # only shape that boots. --expert-precise-memory is the qualified sizing.
+    grep -q -- '--enable-expert-offload' "$SERVE"
+    grep -q -- '--expert-precise-memory' "$SERVE"
+}
+
+@test "MTP spec decode is absent BY MEASUREMENT, not by omission" {
+    # 2026-09-30: the MTP head loads and accepts well (mean accepted length
+    # 3.76-4.90), but decode was unchanged (9.6-11.2 vs 10.0 tok/s) while
+    # prefill collapsed 918 -> 255 tok/s — the verification forward gathers the
+    # union of the batch's experts, so spec decode does not amortize a
+    # residency-bound gather. It also costs 2.6 GiB of arena.
+    ! grep -q -- '--speculative-config' "$SERVE"
+    ! grep -q 'speculative' "$ENV_FILE"
+}
+
+@test "fp8 KV is absent — it would disable the only MLA decode path on gfx12" {
+    # _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend requires a
+    # non-fp8 KV cache, and that rope-free Triton path is the only MLA decode
+    # available on gfx12: aiter's MLA decode op is registered under
+    # @if_aiter_supported, gated on CDNA>2. With fp8 KV the MTP draft step dies
+    # with "_OpNamespace 'vllm' object has no attribute
+    # 'rocm_aiter_mla_decode_fwd'". KV dtype therefore stays auto/bf16.
+    ! grep -q -- '--kv-cache-dtype' "$SERVE"
+    ! grep -q '^FRANK_PRIMARY_KV_DTYPE=' "$ENV_FILE"
+}
+
@test "expert prefetch is absent BY LAW, not by omission" {
    # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
    # prefetched slabs (36% precision) — roughly half the gather traffic was
@@ -149,10 +190,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
@test "the model id matches the payload's served alias wiring" {
    # The payload dir and its shard count are the fetch script's output
    # layout; the fetch script must write where the env points.
-    grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
-    grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
-    grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
+    grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
+    grep -q 'GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
+    grep -q 'Intel/GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
 }

@test "no bare HIP-reserved env var is defined" {
@@ -168,12 +209,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # storage or the payload is missing/wrong-sharded — then clears stale
    # containers (including the legacy identities), waits out the port,
    # relinks/enables/restarts the root unit and health-gates it.
-    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
+    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-glm' "$REMOTE_DEPLOY"
    grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
    grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
    grep -q 'config.json' "$REMOTE_DEPLOY"
    grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
+    grep -q 'glm-probe' "$REMOTE_DEPLOY"
    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"