frank-primary: qwen cutover docs + deployer identity + seam re-pin

- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4
  (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index.
- runbook: RETIRED verdict — DFlash2 drafter served and drafted but
  acceptance pinned 1.00-1.38 on every text class including verbatim
  repetition (mis-wired aux capture, not a weak drafter); 5x slower than
  baseline at ~1.2 acceptance.
- log.md: 2026-10-01 rollback entry.
- deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen;
  glm-probe joins the legacy sweep list.
- frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification
  law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV
  PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool
  dialect, automatic offload. The old 'MTP buys zero' law carried the
  prefetch tax and is retired.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
 | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
 
@@ -245,7 +245,7 @@ merely wedged kernel usually still answers ARP.
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -423,86 +423,39 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].
 
-### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
+### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
 
-The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
-prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
-the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
-with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
-PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
-**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
-n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
-template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
-`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
-over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
-llama.cpp b11030 the same day) at Frank's direction.
+The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
+(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
+**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
+**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
+GLM/DFlash2 experiment was declared failed (same day).
 
-**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
-run qualified the `R9V_*` config into
-`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
-file `scripts/frank-primary.env` carries ONLY production overrides
-(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
-`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
-`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
-(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
-quantizes BOTH fused hyper-connection shards, so the shard-uniformity
-check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
-overlay's env.always injects the uncensored 160-slot cache geometry, which
-the q8 80-slot tiered cache rejects). Never copy tuned values from the
-state into the repo — it would drift from the qualification.
+Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
+~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
+35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
+BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
+throughput one. V-only q8 KV is NOT possible via config: stock
+`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
 
-**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
-single dict — one method per engine, NO chaining — so the llama.cpp
-`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
-(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
-architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
-single method, not a chainable one.
+The engine auto-sets the attention block size (mamba-page floor); the
+expert plan's registry prices KV per token; the expert arena (6,574 slots
+x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
+for KV headroom collapses the offload hit rate.
 
-**Serving = the fork's own launcher through a docker→podman shim.** The R9V
-tooling is docker-pinned; the shim at
-`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
-three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
-the image by its OCI manifest digest while podman keys by config digest but
-preserves the manifest digest in `RepoDigests` — the shim answers
-`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
-refs to `repo@sha256:X`; (2) podman inspect answers
-`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
-(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
-and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
-/`ServerVersion` info fields — the shim answers them.
+Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
+`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
+frank-primary-glm`. The serve script drives `podman run` directly, gates
+the digest-pinned image, the payload's `config.json` and its exact
+131-shard count and the GPU device nodes, clears stale container
+identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
+same, waits out :7770, relinks/enables/restarts the unit and health-gates
+with the MainPID check. Boot cost ~7 min warm; health returns 503 while
+loading.
 
-**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
-54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
-quadlet would be capped by the user manager's limit).
-The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
-(ready + paths), the pinned image (RepoDigests), the PLE byte count
-(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
-clears a stale container, then execs the fork's `scripts/launch.sh` through
-the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
-without the wait a crashed engine is a green unit; `docker wait` cannot stop
-the container child, hence `ExecStop=podman stop -t 120
-frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
-PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
-restarts the unit and health-gates with the MainPID check (the false-green
-stray law).
-
-Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
-payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
-battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
-length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
-mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
-round-trip + template probe re-verified on :7770 after the deploy. No
-throughput bench was run (Frank declined); the probe-era 11 tok/s decode
-was residency-bound (halved expert headroom at double the bytes/expert),
-not the production number.
-
-Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
-Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
-`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
-and the template keys, redeploy (state + `restore-config-20260927.env` +
-payload intact on disk) — see
-[[frank-primary-r9v-radiance-qwen38-uncensored]].
+Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -851,7 +804,7 @@ Active:
 |---------|---------|
 | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
 | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
+| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -859,6 +812,7 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
 | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |

scripts/deploy-remote.sh

@@ -1201,85 +1201,83 @@ deploy_llm_small() {
     wait_for_health frank-small http://127.0.0.1:7771/health 900
 }
 
-# deploy_llm_primary — frank-primary is the R9V radiance-vLLM fork (docker-
-# pinned prebuilt, gfx1201, driven through the docker->podman shim) serving
-# the unsloth Qwen3.8-Flash-Next Q8_0 payload (6 GGUF shards) with the MTP4
-# drafter on the two R9700s as a ROOT unit whose serve script execs the
-# fork's launcher through rootless podman as dank. Same port and alias as
-# the endpoints it replaced (the 2026-09-27 cutover from uncensored
-# IQ4_XS); the harness models.yml id is unchanged. The uncensored state dir
-# (/home/dank/r9v-state/uncensored) stays on disk as the rollback target.
+# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
+# (digest-pinned image, gfx1201) serving the orcarouter
+# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
+# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
+# script drives `podman run` directly (the image entrypoint execs
+# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
+# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
+# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
+# alias as the endpoints it replaced; the harness models.yml id is unchanged.
 #
-# Gates: the on-host R9V setup state must exist and be ready (it carries the
-# pinned image digest and every payload path), the pinned runtime image must
-# be loadable from dank's rootless storage by its OCI manifest digest, the
-# PLE payload must match its pinned byte count, and every GGUF shard + the
-# mmproj must exist (per-file sha256 verification was setup's job at fetch
-# time, never re-hashed here).
-LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json
-LLM_PRIMARY_CONTAINER=frank-primary-r9v
+# Gates: the pinned runtime image must be loadable from dank's rootless
+# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
+# the payload dir must carry config.json and the expected shard count
+# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
+# sha256 verification was the fetch script's job at download time, never
+# re-hashed here.
+LLM_PRIMARY_CONTAINER=frank-primary-qwen
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
 
 deploy_llm_primary() {
     # Refuse rather than restart-loop: a unit whose image or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
-    if [[ ! -f "$LLM_PRIMARY_STATE" ]]; then
-        echo "ERROR: R9V setup state absent: $LLM_PRIMARY_STATE" >&2
-        echo "frank-primary: run the profile setup on this host first" >&2
+    local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
+    local image model_dir shards count
+    if [[ ! -f "$env_file" ]]; then
+        echo "ERROR: frank-primary env file absent: $env_file" >&2
         return 1
     fi
-    local state_json image_digest model_dir ple_path ple_bytes
-    state_exports="$(python3 - "$LLM_PRIMARY_STATE" <<'PY'
-import json, shlex, sys
-state = json.load(open(sys.argv[1]))
-config = state.get("config") or {}
-if state.get("ready") is not True:
-    sys.exit("R9V setup state is not ready")
-for key in ("R9V_IMAGE", "R9V_MODEL_DIR", "R9V_PLE_PATH", "R9V_PLE_EXPECTED_BYTES",
-            "R9V_TARGET_REL", "R9V_TARGET_SHARD2_REL", "R9V_TARGET_SHARD3_REL",
-            "R9V_TARGET_SHARD4_REL", "R9V_TARGET_SHARD5_REL", "R9V_TARGET_SHARD6_REL",
-            "R9V_MMPROJ_REL"):
-    print(f"{key}={shlex.quote(config.get(key, ''))}")
-PY
-)" || return 1
-    eval "$state_exports"
-    if [[ -z "$R9V_IMAGE" || -z "$R9V_MODEL_DIR" || -z "$R9V_PLE_PATH" ]]; then
-        echo "ERROR: R9V setup state is missing image/model/PLE paths" >&2
+    image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
+    model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
+    shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
+    if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
+        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
         return 1
     fi
-    if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
-        echo "ERROR: pinned R9V runtime image absent from rootless storage: $R9V_IMAGE" >&2
-        echo "frank-primary: load the image bundle (./r9v setup ...)" >&2
+    # Capture podman's output BEFORE grepping: under set -o pipefail, the
+    # classic `podman images | grep -q` races (grep exits on match, podman
+    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
+    # 2026-09-30 deploy refusal).
+    local digests
+    digests="$(podman images --format '{{.RepoDigests}}')"
+    if ! grep -qF "${image#*@}" <<<"$digests"; then
+        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
+        echo "frank-primary: fetch it (podman pull $image)" >&2
         return 1
     fi
-    if [[ -n "$R9V_PLE_EXPECTED_BYTES" ]]; then
-        ple_bytes="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
-        if [[ "$ple_bytes" != "$R9V_PLE_EXPECTED_BYTES" ]]; then
-            echo "ERROR: PLE payload wrong size: $R9V_PLE_PATH (expected $R9V_PLE_EXPECTED_BYTES, found $ple_bytes)" >&2
-            echo "frank-primary: re-run the PLE derivation (./r9v setup ...)" >&2
-            return 1
-        fi
+    if [[ ! -f "$model_dir/config.json" ]]; then
+        echo "ERROR: payload config missing: $model_dir/config.json" >&2
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+        return 1
+    fi
+    if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
+        echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+        return 1
+    fi
+    count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
+    if [[ "$count" != "$shards" ]]; then
+        echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+        return 1
     fi
-    local rel
-    for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
-        "${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
-        "$R9V_MMPROJ_REL"; do
-        [[ -z "$rel" ]] && continue
-        if [[ ! -f "$R9V_MODEL_DIR/$rel" ]]; then
-            echo "ERROR: payload file missing: $R9V_MODEL_DIR/$rel" >&2
-            echo "frank-primary: re-run the profile setup to fetch/verify the payload" >&2
-            return 1
-        fi
-    done
 
     # A stale container with the same name makes `podman run --name` fail at
     # start; remove any leftover from a failed previous attempt.
-    podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
     # A stopped unit can leave its container running: systemd's cgroup kill
     # races conmon's --rm cleanup, and the orphan holds the port and GPUs.
-    # Stop the unit, then remove the container BEFORE waiting on the port.
+    # Stop the unit, then remove the containers BEFORE waiting on the port.
+    # The legacy identities (the retired R9V container, the ad hoc
+    # experiment container qwen-fp8, the retired glm-probe) hold the same
+    # GPUs and die here too.
     run_privileged systemctl stop frank-primary.service 2>/dev/null || true
-    podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
+    local ctr
+    for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
+        podman rm -f "$ctr" >/dev/null 2>&1 || true
+    done
     # Nothing starts until the port is released: the lucebox cutover
     # (2026-09-26) and the frank-large cutover (2026-09-16) both failed
     # exactly here when the next engine bound 7770 a second too early.
@@ -1288,9 +1286,8 @@ PY
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-primary.service
     run_privileged systemctl restart frank-primary.service
-    # Cold start: the launcher's preflight doctor plus payload page-in
-    # (~93 GiB cold; expert cache fills after /health). Parity with every
-    # predecessor on this port.
+    # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
+    # measured 2026-09-29). Parity with every predecessor on this port.
     wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
     # False-green hardening (the 2026-09-26 lesson): a bare curl against the
     # port cannot tell WHO answers — a stray probe holding :7770 satisfies

scripts/download-frank-primary.sh

@@ -1,92 +1,88 @@
 #!/usr/bin/env bash
-# Fetch the frank-primary payload: the Qwen3.8-Flash-Next Q8_0 target (six
-# split parts), the shared Q8_0 MTP drafter head and the BF16 vision
-# projector.
+# Payload inventory/verification for frank-primary:
+# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
+# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
+# became frank-primary on 2026-09-29.
 #
-#   bash scripts/download-frank-primary.sh
-#   bash scripts/download-frank-primary.sh --verify-only
+#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
+#   bash scripts/download-frank-primary.sh --fetch
 #
-# The repo (unsloth/Qwen3.8-Flash-Next-GGUF) is PUBLIC and anonymous: no HF
-# token is needed, and none is accepted on the command line (a -H auth
-# header is world-readable in /proc/<pid>/cmdline -- the 2026-09-16 leak).
+# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
+# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
+# manifest cannot be built without the authed endpoint. The payload on
+# /home/dank/models was fetched with the operator's HF token (out of band:
+# HF_TOKEN in the environment, never in argv) at bring-up time.
 #
-# WHY THESE FILES:
-#   * Target: Q8_0/ -- unsloth's Q8_0 of Qwen3.8-Flash-Next (125B total /
-#     6B active MoE + the 51B n-gram embedding table). Q_8 is the
-#     operator's requested quant (2026-09-26 swap off the lucebox DS4.1
-#     engine). The six parts ARE the payload: part 1's header advertises
-#     file_count=6, so they are never concatenated (concatenation trips
-#     "invalid split file name").
-#   * Drafter: MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -- the shared
-#     MTP head (1 layer). "shared" means it borrows the target's embedding
-#     tensors instead of carrying its own copy: 2.6 GiB instead of 3.9,
-#     and unsloth's own MTP guide names it as the recommended head. It is
-#     loaded via -md (this build never autoloads drafters or projectors).
-#   * Vision: mmproj-BF16.gguf -- the BF16 projector, RDNA4-native per the
-#     frank-small precedent. Loaded via --mmproj explicitly.
-#
-# The script is resumable (curl -C -) and checks every artifact's exact byte
-# count AND sha256 against the Hub's declared values, so a truncated or
-# mis-fetched file fails here instead of load-crashing the server. Sizes and
-# hashes were captured from the resolve endpoint's x-linked-etag/x-linked-size
-# on 2026-09-26.
-#
-# Total: 191,919,144,448 bytes (178.74 GiB), of which 175.30 GiB is the
-# target itself. Destination keeps the Hub layout so the serve script can
-# name paths exactly as they appear upstream.
+# Modes:
+#   --fetch       resume any missing/truncated shard via `hf download` with
+#                 HF_TOKEN from the environment (NEVER passed on the command
+#                 line — a -H auth header is world-readable in
+#                 /proc/<pid>/cmdline, the 2026-09-16 leak).
+#   --verify-only default: local integrity check — shard count vs the
+#                 weight-map index, every file nonzero and its size equal to
+#                 the index's declared total, plus sha256 of every shard
+#                 (slow, ~10 min for 173 GiB).
 set -euo pipefail
 
-REPO="unsloth/Qwen3.8-Flash-Next-GGUF"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-GGUF"
+REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
+DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
+EXPECTED_SHARDS=131
 
-# path|bytes|sha256
-MANIFEST=(
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf|10946624|2dabcbb53ca537a7947bc7d20414fd464eeaf4d66d43021b5b2556cc87544ad2"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf|682434912|494ca4ed3dbf97bc28da88af3890b8877b9032f909812d00c0526a9ca5e91d2e"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf|54400261312|34efd79a80a1ce540a517a5d56171924b66ce1c38b04c904f17ad6d8ef17cf20"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf|49446841216|bfa634025fabbd2658bf7694bc80b90e571699c768723f844c934c7ef06c691a"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf|49668930400|232a8f14cc0fa4262e7efe8593774b136fe40909e39c7a020342ddaa27259a97"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf|34015618784|538a93bca918064983409a41187ad4c68640f9aced6f29564da8f551bf86d7a5"
-  "MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf|2786568256|5ff54097406a905cf3a724c709124ceb0e3e10235ee862298969e91c96fa96e6"
-  "mmproj-BF16.gguf|907542944|2e788f8c511d8093c7b43cb87b2fd7e14228340318057f8fb20c86df2efe2355"
-)
+mode="verify"
+[[ "${1:-}" == "--fetch" ]] && mode="fetch"
 
-mode="download"
-[[ "${1:-}" == "--verify-only" ]] && mode="verify"
-
-if [[ "$mode" == "download" ]]; then
-    for entry in "${MANIFEST[@]}"; do
-        path="${entry%%|*}"
-        mkdir -p "$DEST/$(dirname "$path")"
-        echo "fetching $path"
-        # -f: a 401/403 must FAIL the fetch, not write the error body as
-        # the artifact (the 2026-09-16 gated-repo incident). -C -: resume
-        # partial files.
-        curl -fL -C - -o "$DEST/$path" --retry 10 --retry-delay 5 \
-            --retry-max-time 21600 \
-            "https://huggingface.co/$REPO/resolve/main/$path"
-    done
+if [[ "$mode" == "fetch" ]]; then
+    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
+    export HF_TOKEN
+    mkdir -p "$DEST"
+    hf download "$REPO" --local-dir "$DEST"
 fi
 
-echo "verifying sizes and sha256"
-ok=1
-for entry in "${MANIFEST[@]}"; do
-    path="${entry%%|*}"; rest="${entry#*|}"
-    size="${rest%%|*}"; sha256="${rest#*|}"
-    file="$DEST/$path"
-    act="$(stat -c%s "$file" 2>/dev/null || echo 0)"
-    if [[ "$act" -ne "$size" ]]; then
-        echo "  BAD-SIZE  $file ($act bytes, want $size)" >&2
-        ok=0
-        continue
-    fi
-    act_sha="$(sha256sum "$file" | cut -d' ' -f1)"
-    if [[ "$act_sha" != "$sha256" ]]; then
-        echo "  BAD-SHA   $file ($act_sha)" >&2
-        ok=0
-        continue
-    fi
-    echo "  OK   $path ($act bytes)"
-done
-[[ "$ok" -eq 1 ]] || exit 1
-echo "all artifacts present, sized and hashed correctly"
+echo "verifying local payload: $DEST"
+fail=0
+
+# The weight-map index is the source of truth for the shard set.
+INDEX="$DEST/model.safetensors.index.json"
+if [[ ! -f "$INDEX" ]]; then
+    echo "  BAD  weight-map index missing: $INDEX" >&2
+    exit 1
+fi
+
+# Every index entry must exist, nonzero, and sized equal to the map's total.
+python3 - "$INDEX" "$DEST" <<'PY' || fail=1
+import json, os, sys
+index, dest = sys.argv[1], sys.argv[2]
+wm = json.load(open(index))["weight_map"]
+files = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(files)} files")
+bad = 0
+total = 0
+for f in files:
+    p = os.path.join(dest, f)
+    if not os.path.isfile(p) or os.path.getsize(p) == 0:
+        print(f"  BAD  {f}: missing or zero bytes" )
+        bad += 1
+    else:
+        total += os.path.getsize(p)
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
+raise SystemExit(1 if bad else 0)
+PY
+
+# Shard count gate (what the serve script and deployer check).
+n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
+if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
+    echo "  BAD  shard count: found $n, want $EXPECTED_SHARDS" >&2
+    fail=1
+else
+    echo "  OK   shard count $n"
+fi
+
+if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
+    echo "hashing every shard (slow)..."
+    cd "$DEST"
+    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
+    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
+fi
+
+[[ "$fail" -eq 0 ]] || exit 1
+echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"

scripts/frank-primary.env

@@ -1,52 +1,61 @@
-# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
+# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# The full tuned engine shape (tiered-expert placement, CPU offload budgets
-# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
-# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
-# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
-# the q8 profile's setup run (ready: true; doctor PASS, artifacts
-# sha256-verified at fetch time). serve-frank-primary.sh layers that state
-# first and THIS FILE second, so the keys below win. Never copy tuned values
-# from the state into this file — the state is the single source of truth
-# for engine shape.
-R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
-R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
-# Fleet identity: LAN bind, the single frank-primary port, the harness's
-# model id, our own container name (the launcher default r9v-qwen38-flash-next
-# is the scratch identity).
-R9V_HOST_BIND=0.0.0.0
-R9V_HOST_PORT=7770
-R9V_SERVED_MODEL_NAME=frank-primary
-R9V_CONTAINER_NAME=frank-primary-r9v
-R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
-# PLE registration (54.4 GB mmap -> HIP) can exceed the engine's 600 s
-# VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured
-# 740 s on 2026-09-28, right after a bench teardown; a production boot
-# failed on it the same morning). launch.sh 2de0357 forwards this to the
-# engine; 1800 s covers the slowest observed registration with margin.
-R9V_PLE_READY_TIMEOUT=1800
-# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
-# package ships inject as F32), so the shard-uniformity check rejects the
-# fused projection at load. This dev-overlay adapter dequantizes a quantized
-# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
-# payloads pass through unchanged.
-R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
-# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
-# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
-# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
-# that injection off for this payload (launch.sh passes it after the overlay
-# envs; docker last-wins).
-R9V_FULL_MUTABLE_CACHE=0
-# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
-# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
-# preserve_thinking true. --chat-template wins over the tokenizer dir's
-# metadata/chat_template.jinja; request-level kwargs still win per request.
-# Braces in values get the same single-quote law as parentheses (source
-# parses assignment values).
-R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
-R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}'
-# Prefix caching (bench matrix E8, 2026-09-28): qualified on the q8 profile
-# at 131072, zero VRAM cost (reuses freed KV blocks), cold path unchanged.
-# Reuse-heavy traffic (harness retries/resends) hits from the third round:
-# 32k TTFT 44.2 s -> 3.6 s, 131k 439 s -> 6.9 s. Reversible: flip to 0.
-R9V_ENABLE_PREFIX_CACHING=1
+# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
+# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
+# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
+# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
+# digest-pinned image the 2026-09-29 tuning campaign measured — serving
+# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
+# the MTP4 drafter on the two R9700s.
+#
+# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
+#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
+#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
+#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
+#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
+#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
+#     free; 7.2 GiB boots).
+#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
+#     — the stock flag quantizes them together, which is exactly what was
+#     requested here). Accepted by the attention path; qualified with a
+#     completion probe.
+#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
+#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
+#     roughly half the gather traffic was waste; prefetch off is the tuned
+#     incumbent and the +40-51% decode came from deleting the flag. Never
+#     re-add --expert-prefetch without a fresh A/B.
+#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
+#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
+#     numerics (repetition loops, early halts, invalid measurements).
+# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
+# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
+#
+# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
+# prefixed FRANK_PRIMARY_ so none can leak.
+
+# --- runtime ----------------------------------------------------------------
+# Digest-pinned: the floating :latest tag moved once already. Local alias
+# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
+# bytes in rootless storage.
+FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+
+# --- payload ----------------------------------------------------------------
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
+FRANK_PRIMARY_MODEL_SHARDS=131
+
+# --- identity ---------------------------------------------------------------
+# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
+FRANK_PRIMARY_CONTAINER=frank-primary-qwen
+FRANK_PRIMARY_SERVED_NAME=frank-primary
+FRANK_PRIMARY_PUBLISH=7770:7770
+# Containers removed before every start: the retired R9V identity and the ad
+# hoc experiment identity (both hold the GPUs / name registrations).
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+
+# --- qualified engine shape ---------------------------------------------------
+FRANK_PRIMARY_MAXLEN=200000
+FRANK_PRIMARY_SEQS=3
+FRANK_PRIMARY_KV_BYTES=7730934784
+FRANK_PRIMARY_KV_DTYPE=fp8
+FRANK_PRIMARY_SPEC=4
+FRANK_PRIMARY_UTIL=0.98

scripts/frank-primary.service

@@ -1,24 +1,26 @@
-# frank-primary.service — R9V radiance-vLLM endpoint (unsloth
-# Qwen3.8-Flash-Next Q8_0, MTP4, TP=2, tiered experts, 6-shard payload) on
-# the 2x R9700 pair, dankserver :7770. 2026-09-27 cutover from the
-# uncensored IQ4_XS payload (its state dir stays on disk as rollback).
+# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
+# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
+# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
+# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
+# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
+# launcher/shim/setup-state machinery is retired and the state dirs stay on
+# disk as rollback).
 #
-# Replaces the llama.cpp frank-primary unit (same port, same served alias;
-# harness models.yml unchanged in id, retuned for the 131072 window). The
-# serve script drives the R9V fork's own launcher through the docker->podman
-# shim and then waits on the container, so a crashed engine is a nonzero
-# ExecStart and Restart=on-failure is real.
+# Same port, same served alias; the harness models.yml id is unchanged.
+# The serve script drives `podman run` directly (the image entrypoint execs
+# `vllm serve`) and then waits on the container, so a crashed engine is a
+# nonzero ExecStart and Restart=on-failure is real.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the UVA expert host copy plus the 16 GiB PLE pinned
-# reserve). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
-# session on this host carries 8 MiB and a user quadlet is capped by the user
-# manager's own hard limit — raising that means restarting user@1000.service
-# and bouncing every user container on the host. A root unit grants
-# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays in
-# dank's rootless podman storage (the shim drives podman as dank).
+# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
+# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
+# login session on this host carries 8 MiB and a user quadlet is capped by the
+# user manager's own hard limit — raising that means restarting
+# user@1000.service and bouncing every user container on the host. A root unit
+# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
+# stays in dank's rootless podman storage.
 [Unit]
-Description=frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 on 2x R9700
+Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
 
@@ -33,9 +35,9 @@ Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-pr
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
 # explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-r9v
-# Boot cost: the launcher's preflight doctor (~1 min) after a warm page
-# cache; cold (post-reboot) payload page-in of ~175 GiB dominates.
+ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
+# Boot cost: FP8 payload page-in plus torch compile/graph capture —
+# ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
 Restart=on-failure
 RestartSec=10

scripts/patches/glm53-compile-01-model-decorator.patch

@@ -0,0 +1,86 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+#   Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+#   provides neither a compiled submodule nor breakable CUDA graphs.
+#   Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+#   16.07 s  34.8 %  708,500 launches  vectorized_elementwise_kernel      (22.7 us each)
+#    4.60 s  10.0 %    42,000           r4d_gemm_moe_w4a16_nt_b16_m64     (the MoE GEMM)
+#    3.11 s   6.7 %   174,708           elementwise_kernel_manual_unroll
+#    2.94 s   6.4 %   138,252           wvSplitK_hf_sml_                  (split-K GEMM)
+#    4.09 s   8.9 %    83,160           topk/router (sbtopk gatherTopK + warpMergeSortTopK
+#                                       + bitonicSortKVInPlace)
+#    3.01 s   6.5 %   133,920           r4d_mhc_* (hyper-connection pre/post/finish)
+#    2.78 s   6.0 %   127,916           inductor triton_poi_* / triton_per_* (the only
+#                                       fusions that DO happen)
+#    0.49 s   1.05%    20,664           r4d_moe_lru_gather_kernel         <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ -1,12 +1,14 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
++import os
+ from collections.abc import Iterable
+ from typing import ClassVar, Literal
+ 
+ import torch
+ from torch import nn
+ 
++from vllm.compilation.decorators import support_torch_compile
+ from vllm.config import ParallelConfig, VllmConfig
+ from vllm.distributed import (
+     get_ep_group,
+@@ -623,6 +625,18 @@
+         )
+ 
+ 
++_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
++
++
++@support_torch_compile(
++    dynamic_arg_dims={
++        "input_ids": 0,
++        "positions": -1,
++        "intermediate_tensors": 0,
++        "inputs_embeds": 0,
++    },
++    enable_if=lambda _cfg: _GLM_COMPILE,
++)
+ class Glm5NextModel(nn.Module):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()

scripts/patches/glm53-compile-02-trace-safe-probes.patch

@@ -0,0 +1,119 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+#   Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+#   ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+#   cold lru_cache call       -> RAISE Unsupported
+#   WARM lru_cache call       -> RAISE Unsupported     <-- warming is not enough
+#   plain module-global read  -> compiles
+#   uncached os.path.exists   -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79  available()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81  has_w4a16_moe()
+#   model_executor/kernels/mhc/r4d_dsv4.py:16                    r4d_mhc_enabled()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:694                r4d_indexer_enabled(), mqa_max_heads()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:1124               r4d_indexer_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry()  moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py   (inside the pinned image)
+#   base    sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+#   patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
++++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ -361,3 +361,71 @@
+ @functools.lru_cache
+ def moe_w4a16_mt1():
+     return getattr(lib(), "r4d_gemm_moe_w4a16_nt_b16_m64_mt1", None)
++
++
++# FORK-LOCAL (patch: trace-safe capability probes).  torch.compile traces the
++# BODY of a functools.lru_cache call even when the cache is warm, so a capability
++# probe called from a forward pass is a dynamo graph break.  Measured 2026-09-30
++# on the GLM-5.3-Flash trunk under @support_torch_compile:
++#   Dynamo does not know how to trace the builtin `posix.stat`
++#   r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
++# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
++# (outside any tracing region), so a traced call is a dict read.  The probe set is
++# every no-argument capability probe / symbol lookup reachable from a forward
++# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
++# and the draft/lm-head guards.
++_PROBE_NAMES = (
++    "available",
++    "lib",
++    "has_w4a16_moe",
++    "moe_w4a16",
++    "moe_w4a16_mt1",
++    "has_mhc_dsv4",
++    "r4d_mhc_enabled",
++    "r4d_indexer_enabled",
++    "mqa_max_heads",
++    "r4d_act_quant_enabled",
++)
++
++
++def _freeze_probe(fn):
++    """Return fn bound to its first result.  Result must be a plain value."""
++
++    memo = []
++
++    def probe(*args, **kwargs):
++        if not memo:
++            memo.append(fn(*args, **kwargs))
++        return memo[0]
++
++    return probe
++
++
++def _freeze_probes() -> list[str]:
++    frozen = []
++    for name in _PROBE_NAMES:
++        fn = globals().get(name)
++        if fn is None:
++            continue
++        # Only the fast branch of a bound probe is traced; an empty memo means the
++        # first call still does I/O, so populate it here as well.
++        globals()[name] = _freeze_probe(fn)
++        frozen.append(name)
++    return frozen
++
++
++from vllm.logger import init_logger as _init_logger
++
++_probe_log = _init_logger(__name__)
++
++if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
++    _warmed, _failed = [], []
++    for _n in _freeze_probes():
++        try:
++            globals()[_n]()
++            _warmed.append(_n)
++        except Exception:  # noqa: BLE001 -- probe unavailable here; recompute lazily
++            _failed.append(_n)
++    _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
++                    len(_warmed), _warmed,
++                    f" (deferred: {_failed})" if _failed else "")

scripts/patches/glm53-compile-03-fullgraph-optout.patch

@@ -0,0 +1,116 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+#   RuntimeError: Graph breaks are not supported with aot compile.
+#   Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+#   models/glm5next/amd/sparse_indexer.py:263  int(positions[...].max().item()) + 1
+#   models/glm5next/amd/sparse_indexer.py:419  lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+#   fullgraph=True   suppress_errors=False  -> RAISE Unsupported
+#   fullgraph=True   suppress_errors=True   -> RAISE Unsupported   <-- suppress_errors
+#                                                                     does NOT cover breaks
+#   fullgraph=False  suppress_errors=False  -> ok   (traceable segments compiled by
+#                                                    inductor, the rest runs eagerly;
+#                                                    numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py   (inside the pinned image)
+#   base    sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+#   patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+#   (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
+# on the decode forward:
+#
+#   KeyError: 'in_proj_qkvbfg_a'
+#     model.py:748  Glm5NextModel.forward        hidden_states, residual, post, comb = layer(...)
+#     model.py:456  Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
+# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
+# not an arbitrary string.
+#
+# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
+# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
+# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
+# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
+# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
+# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
+# GLM decoder layer. Whether the root cause is the fork's module tree (something
+# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
+#
+# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
+#   * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
+#     KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
+#     (the reported line 456 is that method's `def` line, which is how inlined frames
+#     surface).
+#   * It is NOT raised from dynamo's four nn_modules lookups: the probe in
+#     /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
+#     the probe confirmed installed in the API-server and worker processes.
+#   * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
+#     layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
+#     (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
+#     quantization utils do compute exactly this bare key, but each is guarded
+#     (`if proj_name not in fused_mapping: return`), so none of them can raise it.
+#   * It reproduces on every run of this configuration (5 of 5).
+# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
+# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
+# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
+# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
+# probe. Until that names the caller this is a dynamo-side name-resolution failure.
+# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
+# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
+# dropping outer env vars.
+#
+# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
+# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
+# launcher cell (patch 04 removes the call) are real breaks, but with
+# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
+# fullgraph=True, which this trunk can never satisfy anyway.
+--- a/vllm/compilation/wrapper.py
++++ b/vllm/compilation/wrapper.py
+@@ -19,6 +19,15 @@
+ 
+ logger = init_logger(__name__)
+ 
++
++# FORK-LOCAL (patch: fullgraph opt-out).  torch.compile(fullgraph=True) turns any
++# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
++# (verified in-image, torch 2.11+rocm10.0).  With VLLM_COMPILE_FULLGRAPH=0 the
++# traceable segments are still compiled by inductor and the untraceable ones run
++# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
++# indexer) needs.  Default 1 keeps stock behaviour for every other model.
++_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
++
+ R = TypeVar("R")
+ P = ParamSpec("P")
+ 
+@@ -146,7 +155,7 @@
+         with aot_context:
+             self._compiled_callable = torch.compile(
+                 compiled_ptr,
+-                fullgraph=True,
++                fullgraph=_FULLGRAPH,
+                 dynamic=False,
+                 backend=backend,
+                 options=options,

scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch

@@ -0,0 +1,112 @@
+# glm53-compile-04-fused-qkv-rmsnorm-optout.patch — take GLM's fused-norm JIT kernel
+# out of the traced path.
+#
+# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
+# decorator) to matter; not part of the deployed shape.
+#
+# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
+# launcher bug made four consecutive boots misleading.  The scratch launcher
+# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
+# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
+# patch while `_FULLGRAPH` still defaulted to True.  With fullgraph=True *every* break
+# is an error, which is why the reported failure kept moving as breaks were removed:
+#
+#   fullgraph=False, AOT off  -> breaks tolerated; dies later on the dynamo name-map
+#                                `KeyError: 'in_proj_qkvbfg_a'`
+#   fullgraph=True,  AOT off  -> dies on whichever break comes first
+#                                (cell -> workspace -> compiler-disable)
+#
+# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
+# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
+# Launcher fixed; the env var is set again.
+#
+# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
+# break dynamo reported, and after this patch the report moved on to the workspace
+# acquisition — so it removes a real break.  It is not by itself what unblocks a boot:
+# with breaks genuinely tolerated the blocker is the name-map KeyError.
+#
+# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
+# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
+# object this one closes over is bound after use (see below).
+#
+# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30).  With 01+02+03 the
+# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
+# even with VLLM_COMPILE_FULLGRAPH=0, because dynamo cannot build a resume function
+# around it:
+#
+#   torch._dynamo.exc.Unsupported: Read uninitialized cell
+#   Explanation: Attempted to read a cell variable that has not been populated yet.
+#   Developer debug context: CellVariable()
+#
+# User stack:
+#   glm5next/common/model.py:748    Glm5NextModel.forward       layer(...)
+#   glm5next/common/model.py:526    Glm5NextDecoderLayer.forward
+#   glm5next/common/attention.py:590
+#   model_executor/layers/mla.py:208
+#     q_proj_input, kv_c_normed = fused_q_kv_rmsnorm(...)
+#   -> torch/_dynamo/polyfills/__init__.py:392 getattr_and_trace
+#
+# So dynamo is tracing the fork's JIT-kernel launcher framework
+# (VllmTritonJitKernel + @kernel_launcher — 39 call sites) and finds a closure name
+# that is not yet bound.  In models/common/ops/fused_qk_rmsnorm.py the kernel object
+# is bound AFTER the function that closes over it:
+#
+#   def fused_q_kv_rmsnorm(...):          # line 194
+#       ...
+#       _FUSED_Q_KV_RMSNORM_KERNEL(...)   # line 211
+#   _FUSED_Q_KV_RMSNORM_KERNEL = FusedQKVRMSNormKernel()    # line 223
+#
+# Two ways out, and this patch takes the cheap one:
+#   (a) WORKAROUND (here): stop calling the fused kernel from the traced region.  The
+#       flag is already a constructor argument (mla.py:80, default False) that glm5next
+#       turns ON at attention.py:583, so it becomes an env knob.
+#   (b) ROOT FIX (not done): make the launcher trace-safe for all 39 sites.  Any other
+#       @kernel_launcher-backed kernel still in the traced path can produce the same
+#       fatal break, so (b) is the durable fix and (a) is what unblocks iterating.
+#
+# COST of R4D_GLM_FUSE_QKV_RMSNORM=0: one extra small RMSNorm launch per MLA layer per
+# token (q_a_layernorm + kv_a_layernorm instead of one fused kernel) — ~2 launches out
+# of ~3,700 per token on this shape, against the 44 % of device time currently spent in
+# unfused pointwise work.  Correct trade for a compile experiment.
+#
+# TARGET: vllm/models/glm5next/common/attention.py   (inside the pinned image)
+#   base    sha256 362bbf0e6c50cdfa446c6551b9b1ddff425e08f88e3d4140fd8db88f8a619e0c
+#   patched sha256 84969993c250e9f8183a33a6e1ddd607272f48b950b560272eb33b7cdde31189
+#
+# APPLY: `patch -p1` from the vllm package root (verified: applies clean, reproduces
+# the patched sha256), or bind-mount new/.../attention.py over site-packages.
+# R4D_GLM_FUSE_QKV_RMSNORM=1 (default) restores the fused kernel.
+--- a/vllm/models/glm5next/common/attention.py
++++ b/vllm/models/glm5next/common/attention.py
+@@ -1,6 +1,7 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
++import os
+ import torch
+ import torch.nn.functional as F
+ from torch import nn
+@@ -36,6 +37,14 @@
+ from vllm.utils.deep_gemm import PAGED_MQA_PAGE_SIZES
+ from vllm.v1.kv_cache_interface import KpoolTailSpec, MLAAttentionSpec
+ 
++
++# FORK-LOCAL (patch: keep the traced region trace-safe).  R4D_GLM_FUSE_QKV_RMSNORM=0
++# replaces the fused `fused_q_kv_rmsnorm` JIT-kernel launch (one kernel) with two plain
++# RMSNorms, because dynamo cannot trace the launcher's closure ("Read uninitialized cell",
++# fatal even with fullgraph=False).  The fused kernel object in
++# models/common/ops/fused_qk_rmsnorm.py is bound AFTER the function that uses it.
++_FUSE_QKV_RMSNORM = os.environ.get("R4D_GLM_FUSE_QKV_RMSNORM", "1") == "1"
++
+ logger = init_logger(__name__)
+ 
+ # Shared torch.compile config for the indexer's small-kernel leaves. The MLA
+@@ -580,7 +589,7 @@
+             quant_config,
+             prefix,
+             skip_topk=False,
+-            fuse_qkv_rmsnorm=True,
++            fuse_qkv_rmsnorm=_FUSE_QKV_RMSNORM,
+         )
+ 
+     def forward(

scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch

@@ -0,0 +1,96 @@
+# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
+# acquisition must not be traced.
+#
+# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is a break worth
+# removing, but NOT part of the confirmed blocker and NOT required for the
+# break-tolerant route: `torch.compiler.disable` is itself a graph break, so this patch
+# helps only a hypothetical fullgraph=True route, where the workspace acquisition would
+# be better RESTRUCTURED out of the forward (allocate at init/profile time) than made
+# opaque. Kept because the discovery is real and the wrapper is a no-op when nothing is
+# compiling.
+#
+# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
+#
+#   torch._dynamo.exc.Unsupported: Unsupported method call
+#   user stack
+#     glm5next/common/model.py:748      Glm5NextModel.forward
+#     glm5next/common/model.py:526      Glm5NextDecoderLayer.forward
+#     glm5next/common/attention.py:599 / :401
+#     glm5next/amd/sparse_indexer.py:711  forward_native
+#     glm5next/amd/sparse_indexer.py:132  sparse_attn_indexer_kpool
+#     v1/worker/workspace.py:135          get_simultaneous
+#     v1/worker/workspace.py:155          _ensure_workspace_size
+#
+# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
+#
+#     if not isinstance(attn_metadata, dict):
+#         # Reserve workspace for indexer during profiling run
+#
+# so the engine dies exactly where it sizes its memory.  `_ensure_workspace_size`
+# reads ContextVar and ubatch state (`_workspace_lane.get()`,
+# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
+#
+# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
+# so dynamo stops analysing the call and executes it eagerly from inside the compiled
+# region.  Chosen over decorating the workspace manager's own methods because that
+# would change workspace behaviour for every model, not just this one.  The wrapper is
+# a no-op when nothing is compiling, so it needs no env gate and is safe in the
+# stock (uncompiled) path — which is why this patch can be reasoned about
+# independently of whether the changelist ever ships.
+#
+# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
+# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
+# can build a resume function around it is exactly what the next boot decides — the
+# difference between this failure and 04's `Read uninitialized cell`.
+#
+# TARGET: vllm/models/glm5next/amd/sparse_indexer.py   (inside the pinned image)
+#   base    sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
+#   patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
+--- a/vllm/models/glm5next/amd/sparse_indexer.py
++++ b/vllm/models/glm5next/amd/sparse_indexer.py
+@@ -35,6 +35,21 @@
+ from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
+ from vllm.v1.worker.workspace import current_workspace_manager
+ 
++@torch.compiler.disable
++def _acquire_workspace(*shapes_and_dtypes):
++    """Acquire shared workspace buffers, untraced.
++
++    Workspace acquisition reads ContextVar/ubatch state and may allocate
++    (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
++    ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
++    ``Unsupported: Unsupported method call`` and kills the engine during the profile
++    run.  ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
++    region inside the compiled graph.  No-op when not compiling.
++    """
++
++    return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
++
++
+ logger = init_logger(__name__)
+ 
+ # kpool write helper: form pools from the current token batch and compress them
+@@ -129,7 +144,7 @@
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        current_workspace_manager().get_simultaneous(
++        _acquire_workspace(
+             values_spec,
+             scales_spec,
+             ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
+@@ -275,11 +290,10 @@
+         # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
+         # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
+         # scales) based on use_fp4_cache.
+-        workspace_manager = current_workspace_manager()
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
++        k_quant_full, k_scale_full = _acquire_workspace(
+             values_spec,
+             scales_spec,
+         )

scripts/patches/glm53-dflash-01-aux-hidden-states.patch

@@ -0,0 +1,173 @@
+# glm53-dflash-01-aux-hidden-states.patch — give the GLM trunk the EAGLE3 / DFlash2
+# auxiliary-hidden-state interface.
+#
+# STATUS: WIP, and INDEPENDENT of the compile changelist (01-05). It patches the same
+# file as glm53-compile-01-model-decorator.patch, so if both are ever applied, apply 01
+# first. Not part of the deployed shape.
+#
+# WHY.  `--speculative-config '{"method":"dflash","model":"/path/to/drafter", ...}'` with
+# incoai/GLM-5.3-Flash-DFlash2 is accepted -- the config parses, the drafter loads -- and
+# then the worker dies at init with:
+#
+#     RuntimeError: Model does not support EAGLE3 interface
+#
+# raised by v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20 through
+# `supports_eagle3(model)` == `isinstance(model, SupportsEagle3)` (a runtime-checkable
+# Protocol).  Nothing about the drafter, the method name or the payload is wrong:
+# `glm5next` implements neither half of the contract.
+#
+# THE CONTRACT (read out of this image, not assumed):
+#   * DEFINITIVE (v5, after four failed boots): the member set is checked against
+#     the INSTANCE the runner calls -- which for GLM-5.3-Flash is
+#     Glm5NextForConditionalGeneration, NOT Glm5NextForCausalLM.  The multimodal
+#     wrapper wraps the text model as self.language_model
+#     (init_vllm_registered_model, architectures=["Glm5NextForCausalLM"],
+#     glm5next/common/model.py:1113), and model_runner.py:425 calls
+#     set_eagle3_aux_hidden_state_layers(self.model) on the CONDITIONAL.
+#     Presence rule (verified in-image with stock typing, python 3.14):
+#     isinstance walks ALL __protocol_attrs__ -- members of SupportsEagle3 AND
+#     its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
+#     getattr_static()s each one off the instance, breaking on the first miss.
+#     So BOTH wrappers need all three attributes.
+#   * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
+#     "Cannot create a consistent method resolution order (MRO) for bases ..."
+#     (GLM's wrappers already carry four/five Protocol-derived bases).  The
+#     attribute route avoids the MRO entirely.  Also proven by simulation:
+#     the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
+#     position.
+#   * Method placement: without inheriting, the Protocol's default
+#     set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers
+#     do not exist on either wrapper, so both are defined explicitly --
+#     ForCausalLM unwraps to self.model (the EagleModelMixin trunk, v1's
+#     correct bodies), the conditional delegates to self.language_model
+#     (the pixtral.py:498 pattern).
+#   * interfaces.py:1673-1719 -- for reference, the inherited defaults unwrap
+#     `holder = getattr(parent_ref, "model", parent_ref)` and assert
+#     `isinstance(holder, EagleModelMixin)`, defaulting to (2, L // 2, L - 3).
+#   * eagle3_utils.py:26-31 -- layers come from the spec config when it carries them
+#     (this drafter ships target_layer_ids [5, 14, 24, 33, 42]), else from the default.
+#   * deepseek_v2.py:1551-1585 and v1/worker/gpu/cudagraph_utils.py:621 -- the trunk
+#     appends one tensor per aux layer while looping and returns the TUPLE
+#     (hidden_states, aux_hidden_states) only when that list is non-empty.
+#
+# WHAT IT ADDS for GLM: EagleModelMixin on Glm5NextModel; aux capture in the layer
+# loop, where the aux state is the single-stream `hidden_states` ENTERING that layer
+# (GLM's loop carries the already-summed single-stream state -- its non-mHC branch
+# re-splits `residual = hidden_states` inside the layer -- so this is the equivalent
+# of deepseek_v2's `hidden_states + residual`); the conditional tuple return; the
+# three SupportsEagle3 member attributes on BOTH wrappers; and the two interface
+# methods on both (ForCausalLM: unwrap to trunk; conditional: delegate to
+# language_model).
+#
+# HONEST SCOPE: this makes the interface PRESENT, which is exactly what the engine
+# checks -- it does not make drafts GOOD, and per the measured profile it is not a
+# decode-speed fix (a verification pass pays pointwise cost proportional to its token
+# count, which is why MTP4 measured flat).  It is the prerequisite for driving DFlash2
+# on this endpoint at all.
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 bc9913a15c8e7f88e0c36b07bf029b383cc8c2078b58d66dc91aa235f8af77f4
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount the patched file over
+# site-packages (which is what a scratch-launcher DFLASH_PATCH=1 does).
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ -95,6 +95,12 @@
+ logger = init_logger(__name__)
+ 
+ 
++from vllm.model_executor.models.interfaces import (
++    EagleModelMixin,
++    SupportsEagle3,
++)
++
++
+ class Glm5NextMLP(nn.Module):
+     def __init__(
+         self,
+@@ -623,7 +629,7 @@
+         )
+ 
+ 
+-class Glm5NextModel(nn.Module):
++class Glm5NextModel(nn.Module, EagleModelMixin):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+ 
+@@ -730,7 +736,14 @@
+         if self.is_sequence_parallel:
+             hidden_states = sp_shard(hidden_states)
+ 
+-        for layer in self._active_layers:
++        aux_hidden_states: list[torch.Tensor] = []
++        for local_idx, layer in enumerate(self._active_layers):
++            if (getattr(self, "start_layer", 0) + local_idx) in self.aux_hidden_state_layers:
++                # Auxiliary hidden state for EAGLE-3 / DFlash2 drafts: the
++                # single-stream state ENTERING this layer, before its input norm --
++                # the same semantic deepseek_v2 implements by appending
++                # hidden_states + residual at this point in its loop.
++                aux_hidden_states.append(hidden_states)
+             hidden_states, residual, post, comb = layer(
+                 positions, hidden_states, residual, post, comb
+             )
+@@ -749,6 +762,8 @@
+             hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
+ 
+         hidden_states = self.norm(hidden_states)
++        if len(aux_hidden_states) > 0:
++            return hidden_states, aux_hidden_states
+         return hidden_states
+ 
+     def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+@@ -937,6 +952,10 @@
+ class Glm5NextForCausalLM(
+     nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
+ ):
++    supports_eagle3: ClassVar[Literal[True]] = True
++    has_own_lm_head: bool = False
++    has_own_embed_tokens: bool = False
++
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+         self.model_config = vllm_config.model_config
+@@ -963,6 +982,14 @@
+     def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+         return self.model.embed_input_ids(input_ids)
+ 
++    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
++        # EAGLE-3 / DFlash2 interface: unwrap to the trunk (the EagleModelMixin).
++        self.model.aux_hidden_state_layers = layers
++
++    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
++        num_layers = len(self.model.layers)
++        return (2, num_layers // 2, num_layers - 3)
++
+     def forward(
+         self,
+         input_ids: torch.Tensor | None,
+@@ -1033,6 +1060,10 @@
+ class Glm5NextForConditionalGeneration(
+     Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts
+ ):
++    supports_eagle3: ClassVar[Literal[True]] = True
++    has_own_lm_head: bool = False
++    has_own_embed_tokens: bool = False
++
+     # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
+     # multimodal wrapper must declare the same interfaces so vLLM treats it as
+     # hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
+@@ -1076,6 +1107,13 @@
+ 
+         return Glm5NextForCausalLM.get_mamba_state_copy_func()
+ 
++    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
++        # EAGLE-3 / DFlash2: delegate to the language model (pixtral.py:498).
++        self.language_model.set_aux_hidden_state_layers(layers)
++
++    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
++        return self.language_model.get_eagle3_default_aux_hidden_state_layers()
++
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super(Glm4vForConditionalGeneration, self).__init__()
+         config = vllm_config.model_config.hf_config

scripts/patches/glm53-dflash-04-kv-groups-draft.patch

@@ -0,0 +1,317 @@
+--- a/vllm/v1/core/kv_cache_utils.py
++++ b/vllm/v1/core/kv_cache_utils.py
+@@ -1210,17 +1210,38 @@
+         for name, spec in kv_cache_spec.items()
+         if isinstance(spec, KpoolTailSpec)
+     }
++    hidden_specs = {
++        name: spec
++        for name, spec in kv_cache_spec.items()
++        if isinstance(spec, HiddenStateCacheSpec)
++    }
+     attn_specs = {
+         name: spec
+         for name, spec in kv_cache_spec.items()
+-        if not isinstance(spec, (MambaSpec, KpoolTailSpec))
++        if not isinstance(
++            spec, (MambaSpec, KpoolTailSpec, HiddenStateCacheSpec)
++        )
+     }
+-    if not mamba_specs or not all(
+-        type(spec) is MLAAttentionSpec for spec in attn_specs.values()
+-    ):
++    if not mamba_specs or not attn_specs:
++        return None
++    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash
++    # drafter, attn_specs also carries the drafter's attention layers. The
++    # GLM MLA layers keep their own group (no page unification -- per-group
++    # pages are the glm5 design); the drafter layers must form one uniform
++    # group of their own, and the hidden-state layers get per-layer groups
++    # like the generic tail.
++    mla_specs = {
++        name: spec
++        for name, spec in attn_specs.items()
++        if type(spec) is MLAAttentionSpec
++    }
++    draft_specs = {
++        name: spec for name, spec in attn_specs.items() if name not in mla_specs
++    }
++    if not mla_specs:
++        return None
++    if draft_specs and UniformTypeKVCacheSpecs.from_specs(draft_specs) is None:
+         return None
+-
+-    mla_specs = cast(dict[str, MLAAttentionSpec], attn_specs)
+     idx_pages = {
+         spec.page_size_bytes for spec in mla_specs.values() if spec.tokens_per_state > 1
+     }
+@@ -1233,7 +1254,7 @@
+     mla_pages = {mla_specs[name].page_size_bytes for name in mla_names}
+     assert len(mla_pages) == 1
+     mla_page = mla_pages.pop()
+-    uniform_spec = UniformTypeKVCacheSpecs.from_specs(attn_specs)
++    uniform_spec = UniformTypeKVCacheSpecs.from_specs(mla_specs)
+     assert uniform_spec is not None
+ 
+     tail_group: KVCacheGroupSpec | None = None
+@@ -1270,10 +1291,20 @@
+     for index, name in enumerate(mamba_specs):
+         mamba_grouped_names[index % num_groups].append(name)
+ 
++    draft_group: KVCacheGroupSpec | None = None
++    if draft_specs:
++        draft_uniform = UniformTypeKVCacheSpecs.from_specs(draft_specs)
++        assert draft_uniform is not None
++        draft_group = KVCacheGroupSpec(list(draft_specs), draft_uniform)
++    hidden_groups = [
++        KVCacheGroupSpec([name], spec) for name, spec in hidden_specs.items()
++    ]
+     return (
+-        [KVCacheGroupSpec(list(attn_specs), uniform_spec)]
++        [KVCacheGroupSpec(list(mla_specs), uniform_spec)]
+         + ([tail_group] if tail_group is not None else [])
++        + ([draft_group] if draft_group is not None else [])
+         + create_kv_cache_group_specs(padded_specs, mamba_grouped_names)
++        + hidden_groups
+     )
+ 
+ 
+@@ -1289,10 +1320,17 @@
+         int,
+         list[str],
+         int,
++        KVCacheGroupSpec | None,
++        list[KVCacheGroupSpec],
+     ]
+     | None
+ ):
+-    """Recognize the GLM-5.3-Flash grouping after optional PP projection."""
++    """Recognize the GLM-5.3-Flash grouping after optional PP projection.
++
++    FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): also recognizes the
++    drafter's uniform group and the per-layer hidden-state groups, returned
++    as ``draft_group`` / ``hidden_groups`` so the pool builder and the memory
++    estimator can size their tensors beside the aliased GLM pool."""
+     uniform_groups = [
+         group
+         for group in kv_cache_groups
+@@ -1301,17 +1339,28 @@
+     mamba_groups = [
+         group for group in kv_cache_groups if isinstance(group.kv_cache_spec, MambaSpec)
+     ]
++    hidden_groups = [
++        group
++        for group in kv_cache_groups
++        if isinstance(group.kv_cache_spec, HiddenStateCacheSpec)
++    ]
+     attn_group: KVCacheGroupSpec | None = None
+     tail_group: KVCacheGroupSpec | None = None
++    draft_group: KVCacheGroupSpec | None = None
+     for group in uniform_groups:
+         inner = cast(UniformTypeKVCacheSpecs, group.kv_cache_spec).kv_cache_specs
+         if all(type(spec) is MLAAttentionSpec for spec in inner.values()):
+             attn_group = group
+         elif all(isinstance(spec, KpoolTailSpec) for spec in inner.values()):
+             tail_group = group
++        else:
++            draft_group = group
+     if attn_group is None or not mamba_groups:
+         return None
+-    if len(uniform_groups) + len(mamba_groups) != len(kv_cache_groups):
++    if (
++        len(uniform_groups) + len(mamba_groups) + len(hidden_groups)
++        != len(kv_cache_groups)
++    ):
+         return None
+ 
+     attn_uniform = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+@@ -1362,6 +1411,8 @@
+         idx_page,
+         tail_names,
+         tail_page,
++        draft_group,
++        hidden_groups,
+     )
+ 
+ 
+@@ -1395,10 +1446,34 @@
+         return kv_cache_spec
+ 
+     max_page_size = max(page_sizes)
++    # GLM-5.3-Flash + a draft model: the sparse indexer k_cache page is
++    # tokens_per_state x the MLA page (same per-token bytes), and MLA layers
++    # cannot be PADDED to an arbitrary page size (sparse MLA indexes the cache
++    # in whole token rows) -- but they CAN grow their block size when the max
++    # page is an exact multiple of the MLA page. Bump the max page to a
++    # multiple of every MLA-family page so all of them take the divisible
++    # block-growth branch below instead of the padding refusal, while the
++    # draft and mamba layers simply pad to the (slightly larger) page.
++    mla_pages = {
++        spec.page_size_bytes
++        for spec in kv_cache_spec.values()
++        if isinstance(spec, MLAAttentionSpec)
++    }
++    if mla_pages:
++        mla_common = math.lcm(*mla_pages)
++        max_page_size = -(-max_page_size // mla_common) * mla_common
+     new_kv_cache_spec = {}
+     for layer_name, layer_spec in kv_cache_spec.items():
+         if layer_spec.page_size_bytes == max_page_size:
+             new_kv_cache_spec[layer_name] = layer_spec
++        elif isinstance(layer_spec, KpoolTailSpec):
++            # The kpool tail kernel pins the tail block to the kpool size
++            # (tail_kv_cache.view(...) must be [blocks, 2, index_kpool,
++            # head_dim]); growing block_size breaks that contract. Pad the
++            # physical page instead -- the tail is its own group and the pad
++            # wastes bytes only.
++            new_spec = replace(layer_spec, page_size_padded=max_page_size)
++            new_kv_cache_spec[layer_name] = new_spec
+         elif isinstance(layer_spec, MambaSpec):
+             # MambaSpec's page size is determined by its state shapes and does
+             # not scale with block_size, so pad the page instead. This is the
+@@ -1593,8 +1668,31 @@
+ ) -> int:
+     """Return the largest cache group's bytes per block."""
+     if (glm5_layout := _glm5_next_tensor_layout(kv_cache_groups)) is not None:
+-        _, _, mla_names, idx_names, mla_page, idx_page, _, _ = glm5_layout
+-        return len(mla_names) * mla_page + len(idx_names) * idx_page
++        (
++            _,
++            _,
++            mla_names,
++            idx_names,
++            mla_page,
++            idx_page,
++            _,
++            _,
++            draft_group,
++            hidden_groups,
++        ) = glm5_layout
++        bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            bytes_per_block += sum(
++                spec.page_size_bytes for spec in d_inner.values()
++            )
++        for group in hidden_groups:
++            bytes_per_block += (
++                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
++            )
++        return bytes_per_block
+ 
+     bytes_per_block = max(
+         sum(
+@@ -1701,12 +1799,30 @@
+             idx_page,
+             tail_names,
+             _,
++            draft_group,
++            hidden_groups,
+         ) = glm5_layout
+         bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
++        extra_bytes_per_block = 0
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            extra_bytes_per_block += sum(
++                spec.page_size_bytes for spec in d_inner.values()
++            )
++        for group in hidden_groups:
++            extra_bytes_per_block += (
++                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
++            )
+         num_blocks = may_override_num_blocks(
+-            vllm_config, available_memory // bytes_per_block
++            vllm_config,
++            available_memory // (bytes_per_block + extra_bytes_per_block),
+         )
+-        size = bytes_per_block * num_blocks
++        # One backing allocation shared by every tensor (allocate_kv_cache
++        # asserts all tensor sizes equal): the aliased GLM pool plus the
++        # draft/hidden regions appended after it.
++        size = (bytes_per_block + extra_bytes_per_block) * num_blocks
+         attn_specs = cast(
+             UniformTypeKVCacheSpecs, attn_group.kv_cache_spec
+         ).kv_cache_specs
+@@ -1745,6 +1861,37 @@
+                 ).kv_cache_specs
+                 add_tensor(tail_name, tail_specs[tail_name], offset)
+ 
++        # Draft + hidden-state layers get their own tensors appended after
++        # the aliased pool (no aliasing: independent block tables).
++        offset_cursor = bytes_per_block * num_blocks
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            d_page = next(iter(d_inner.values())).page_size_bytes
++            kv_cache_tensors.append(
++                KVCacheTensor(
++                    size=size,
++                    layers=list(draft_group.layer_names),
++                    layer_stride=d_page * num_blocks,
++                    block_stride=d_page,
++                    offset=offset_cursor,
++                )
++            )
++            offset_cursor += d_page * num_blocks * len(draft_group.layer_names)
++        for group in hidden_groups:
++            h_page = group.kv_cache_spec.page_size_bytes
++            kv_cache_tensors.append(
++                KVCacheTensor(
++                    size=size,
++                    layers=list(group.layer_names),
++                    layer_stride=h_page * num_blocks,
++                    block_stride=h_page,
++                    offset=offset_cursor,
++                )
++            )
++            offset_cursor += h_page * num_blocks * len(group.layer_names)
++
+         return KVCacheConfig(
+             num_blocks=num_blocks,
+             kv_cache_tensors=kv_cache_tensors,
+@@ -2440,6 +2587,8 @@
+             idx_page,
+             tail_names,
+             _,
++            draft_group,
++            hidden_groups,
+         ) = glm5_layout
+         uniform_spec = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+         total_blocks = uniform_spec.max_memory_usage_pages(vllm_config)
+@@ -2452,7 +2601,30 @@
+         )
+         if tail_names:
+             total_blocks += 1
+-        return total_blocks * (len(mla_names) * mla_page + len(idx_names) * idx_page)
++        pool_bytes = total_blocks * (
++            len(mla_names) * mla_page + len(idx_names) * idx_page
++        )
++        extra_bytes = 0
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            d_spec = next(iter(d_inner.values()))
++            d_blocks = cdiv(
++                d_spec.max_memory_usage_bytes(vllm_config), d_spec.page_size_bytes
++            )
++            extra_bytes += d_blocks * d_spec.page_size_bytes * len(
++                draft_group.layer_names
++            )
++        for group in hidden_groups:
++            h_spec = group.kv_cache_spec
++            h_blocks = cdiv(
++                h_spec.max_memory_usage_bytes(vllm_config), h_spec.page_size_bytes
++            )
++            extra_bytes += (
++                h_blocks * h_spec.page_size_bytes * len(group.layer_names)
++            )
++        return pool_bytes + extra_bytes
+ 
+     bytes_per_block = _pool_bytes_per_block(kv_cache_groups)
+     total_blocks = 0

scripts/patches/glm53-dflash-04b-kv-registry-dflash.patch

@@ -0,0 +1,31 @@
+--- a/vllm/model_executor/offloader/expert_activation_registry.py
++++ b/vllm/model_executor/offloader/expert_activation_registry.py
+@@ -316,7 +316,27 @@
+ 
+ 
+ def _glm53_kv_bytes_per_token(cfg: "VllmConfig") -> int:
+-    return 13600
++    per_tok = 13600
++    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash/dspark
++    # drafter, the KV spec set gains the drafter's attention layers and the aux
++    # hidden-state cache (one bf16 hidden vector per target aux layer per
++    # token). Both are full-length per-token costs the aliased glm5 pool
++    # cannot absorb, so the reserve must include them. The drafter is a 5-layer
++    # qwen3 (8 kv heads x 128 x K+V x bf16 = 4096 B/token); the aux count is
++    # the dflash target_layer_ids count (5).
++    # NOTE: use_aux_hidden_state_outputs lives on the gpu_model_runner
++    # instance (gpu_model_runner.py:623/678), not on the spec config -- it is
++    # not set yet when the expert plan runs. The spec-config METHOD is the
++    # plan-time signal for the same fact.
++    sc = getattr(cfg, "speculative_config", None)
++    if sc is not None and getattr(sc, "method", "") in (
++        "dflash",
++        "dspark",
++        "extract_hidden_states",
++    ):
++        hidden = int(cfg.model_config.hf_config.hidden_size)
++        per_tok += 5 * hidden * 2 + 4096
++    return per_tok
+ 
+ 
+ for _arch in ("Glm5NextForConditionalGeneration", "Glm5NextForCausalLM"):

scripts/patches/gpu_worker-kv-cap.patch

@@ -0,0 +1,65 @@
+# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
+#
+# STATUS: NOT part of the deployed frank-primary shape. It exists because the
+# fork's plan under-models this model's KV, and it is the enabler for a future
+# MTP attempt (the production GLM shape runs without MTP and boots fine on the
+# plan's own reserve).
+#
+# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
+# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
+# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
+# cache at that reserve, so the plan's own arithmetic can make a legal
+# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
+#   - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB  -> boots
+#   - MTP4,  max_model_len 131072: reserve 1.74 GiB <  need 2.07 GiB  -> refuses
+#   - MTP4,  max_model_len 262144: reserve 3.43 GiB <  need 3.62 GiB  -> refuses
+#     ("To serve at least one request with the model's max seq len ...")
+# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
+# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
+# where the KV reserve would eat a third of the expert arena.
+#
+# THE FIX: let an explicit operator KV request raise the cap. The compile and
+# graph-capture headroom still comes off the top, so the blast radius is one
+# flag the operator chose.
+#
+# APPLY (into a host dir that is then bind-mounted read-only over the image):
+#   IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+#   D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
+#   podman run --rm --entrypoint cat $IMAGE \
+#     /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
+#     > $D/vllm/v1/worker/gpu_worker.py
+#   ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
+#   # then add to the podman run:
+#   #   -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
+#   #   -e CLAV_KV_RESERVE_FLOOR_GIB=<needed>   (or pass --kv-cache-memory-bytes)
+#
+# Base file sha256 (the pristine file out of the pinned image):
+#   1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
+# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
+# these bytes exactly):
+#   80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
+#
+--- a/vllm/v1/worker/gpu_worker.py
++++ b/vllm/v1/worker/gpu_worker.py
+@@ -824,6 +824,21 @@
+         expert_cfg = self.vllm_config.offload_config.expert
+         if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
+             cap = int(expert_cfg.planned_kv_reserve_bytes)
++            # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
++            # 13600 B/token + a small mamba term, but this model also carries a
++            # per-sequence KDA state and (under MTP) the draft's KV, so the
++            # reserve can sit BELOW what max_model_len legitimately needs -- and
++            # the cap then makes a legal config unbootable (measured: need
++            # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
++            # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
++            # is the operator's word: let it raise the cap. The compile/capture
++            # headroom still comes off the top.
++            _explicit = self.cache_config.kv_cache_memory_bytes
++            _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
++            if _explicit:
++                cap = max(cap, int(_explicit))
++            if _floor:
++                cap = max(cap, int(float(_floor) * (1 << 30)))
+             if self.available_kv_cache_memory_bytes > cap:
+                 logger.info_once(
+                     "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "

scripts/serve-frank-primary.sh

@@ -1,30 +1,27 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (R9V radiance-vLLM fork,
-# unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 drafter, 2x R9700, dankserver
-# :7770; 2026-09-27 cutover from the uncensored IQ4_XS payload, whose state
-# dir stays on disk as the rollback target).
+# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
+# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
+# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
+# frank-primary-r9v-q8-20260929 is the pre-cutover state).
 #
-# The engine is the upstream R9V launcher (scripts/launch.sh in the fork
-# checkout), NOT a hand-frozen podman argv: launch.sh owns the resolved
-# docker/podman args (binds, overlays, device passthrough, publish lines)
-# and the fork's qualification was measured through it. This script's job:
+# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
+# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
+# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
+# a 104-key on-host setup state; this stack is fully expressed by flags, and
+# every flag here was qualified by boot probe on 2026-09-29 — see
+# frank-primary.env for the qualification record).
 #
-#   1. layer the on-host R9V setup state (the 104-key tuned shape the
-#      profile setup qualified) under this repo's frank-primary.env
-#      overrides (bind/port/alias/container);
-#   2. gate on the pinned runtime image (by its OCI manifest digest, the
-#      podman RepoDigest), the exact-size PLE payload, and the GGUF shards;
-#   3. exec the launcher, then wait on the container: launch.sh --detach
-#      exits as soon as the container is created, so without the wait the
-#      unit would be "green" while the engine crashes — the wait is what
-#      keeps systemd's Restart=on-failure self-healing real.
+# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
+# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
+# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
+# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
+# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
+# precision — the tuned incumbent deletes it), TunableOp off.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert UVA
-# host copy and the PLE pinned residency (54.4 GB on the q8 payload). A
-# plain shell session also works (proven 2026-09-27: podman rootless
-# inherits usable limits here), but the unit is the fleet-parity
-# boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
+# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
+# also works, but the unit is the fleet-parity boot-persistent shape.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -39,104 +36,86 @@ set -a
 source "$ENV_FILE"
 set +a
 
-: "${R9V_HOME:?R9V_HOME must name the fork checkout}"
-: "${R9V_STATE_FILE:?R9V_STATE_FILE must name the setup state}"
-: "${R9V_HOST_BIND:?R9V_HOST_BIND must be set}"
-: "${R9V_HOST_PORT:?R9V_HOST_PORT must be set}"
-: "${R9V_SERVED_MODEL_NAME:?R9V_SERVED_MODEL_NAME must be set}"
-: "${R9V_CONTAINER_NAME:?R9V_CONTAINER_NAME must be set}"
-
-LAUNCHER="$R9V_HOME/scripts/launch.sh"
-if [ ! -x "$LAUNCHER" ]; then
-    echo "serve-frank-primary: R9V launcher missing: $LAUNCHER" >&2
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
+: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
+: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
+: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
+: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
+: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
+: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
+: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
+# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
+# classic `podman images | grep -q` pattern races — grep exits on its first
+# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
+# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
+    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
     exit 1
 fi
 
-# The docker->podman shim is REQUIRED: the launcher and its doctor speak
-# docker (OCI-manifest-digest image IDs, LogConfig fields) that podman's
-# native CLI answers differently. The shim carries those laws.
-SHIM_DIR="$R9V_HOME/.podman-shim"
-if [ ! -x "$SHIM_DIR/docker" ]; then
-    echo "serve-frank-primary: docker->podman shim missing: $SHIM_DIR/docker" >&2
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
+    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
     exit 1
 fi
-export PATH="$SHIM_DIR:$PATH"
-
-# --- layer 1: the qualified setup state (production overrides never come
-# from here; keys already defined in the env file are skipped) -------------
-STATE_FILE="$R9V_STATE_FILE"
-if [ ! -f "$STATE_FILE" ]; then
-    echo "serve-frank-primary: R9V setup state missing: $STATE_FILE" >&2
-    echo "  run the profile setup first (tools/setup_profile.py setup --state-dir ...)" >&2
+SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
+if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
+    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
+    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
+    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
     exit 1
 fi
-STATE_EXPORTS="$(python3 - "$STATE_FILE" <<'PY'
-import json, shlex, sys, os
-state = json.load(open(sys.argv[1]))
-config = state.get("config") or {}
-if state.get("ready") is not True:
-    sys.exit("R9V setup state is not ready; rerun setup before serving")
-env_keys = {k for k in os.environ if k.startswith("R9V_")}
-for key in sorted(config):
-    if key.startswith("R9V_") and key not in env_keys:
-        print(f"export {key}={shlex.quote(config[key])}")
-PY
-)"
-eval "$STATE_EXPORTS"
-
-: "${R9V_IMAGE:?state must carry R9V_IMAGE}"
-: "${R9V_MODEL_DIR:?state must carry R9V_MODEL_DIR}"
-: "${R9V_PLE_PATH:?state must carry R9V_PLE_PATH}"
-
-# --- gates: never serve a half-finished payload or a missing image --------
-# Runtime image by pinned OCI manifest digest (podman keys images by config
-# digest but preserves the manifest digest in RepoDigests — the shim's ID law).
-if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
-    echo "serve-frank-primary: pinned R9V image absent: $R9V_IMAGE" >&2
-    echo "  load the pinned image bundle into podman rootless storage" >&2
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
+    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
     exit 1
 fi
 
-# PLE payload at its pinned byte count (stat -c %s, never wc -c), plus the
-# six target GGUF shards (mradermacher-style split parts) and the mmproj
-# the launcher mounts.
-PLE_BYTES="${R9V_PLE_EXPECTED_BYTES:-}"
-if [ -n "$PLE_BYTES" ]; then
-    ACTUAL="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
-    if [ "$ACTUAL" != "$PLE_BYTES" ]; then
-        echo "serve-frank-primary: PLE payload wrong size: $R9V_PLE_PATH" >&2
-        echo "  expected $PLE_BYTES, found $ACTUAL" >&2
-        exit 1
-    fi
-else
-    if [ ! -f "$R9V_PLE_PATH" ]; then
-        echo "serve-frank-primary: PLE payload missing: $R9V_PLE_PATH" >&2
-        exit 1
-    fi
-fi
-for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
-    "${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
-    "$R9V_MMPROJ_REL"; do
-    if [ -z "$rel" ]; then continue; fi
-    if [ ! -f "$R9V_MODEL_DIR/$rel" ]; then
-        echo "serve-frank-primary: payload file missing: $R9V_MODEL_DIR/$rel" >&2
-        exit 1
-    fi
-done
-
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
     echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
     exit 1
 fi
 
 # A crashed/killed unit can leave the container name registered (systemd
-# SIGKILLs conmon before cleanup runs); the launcher refuses an existing
-# name forever after. Clear any orphan of OUR name first — idempotent.
-docker rm -f "$R9V_CONTAINER_NAME" >/dev/null 2>&1 || true
+# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
+# identities (the retired R9V container and the ad hoc experiment container
+# both hold these GPUs); all idempotent.
+for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done
 
-# launch.sh detaches after preflight; from here on, this script's lifetime
-# IS the container's: wait returns the container's exit code, so a crash is
-# a nonzero ExecStart and Restart=on-failure fires. (systemd stop goes
-# through ExecStop in the unit — docker wait alone cannot stop the child.)
-"$LAUNCHER"
-exec docker wait "$R9V_CONTAINER_NAME"
+LOG=/home/dank/logs/frank_primary.log
+mkdir -p "$(dirname "$LOG")"
+
+# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
+# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
+# available" while device_count() says 2). --ipc=host for the engine's shared
+# memory; memlock unbounded for the pinned host slabs.
+# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
+  --publish "$FRANK_PRIMARY_PUBLISH" \
+  --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  "$FRANK_PRIMARY_IMAGE" /models \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
+  --tensor-parallel-size 2 \
+  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
+  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
+  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --enable-prefix-caching \
+  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  --trust-remote-code \
+  > "$LOG" 2>&1
+
+# From here on, this script's lifetime IS the container's: wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=on-failure is real. (systemd stop goes through ExecStop in the
+# unit — podman wait alone cannot stop the container child.)
+exec podman wait "$FRANK_PRIMARY_CONTAINER"

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -0,0 +1,706 @@
+---
+type: runbook
+title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - glm5.3
+  - w4a16
+  - int4
+  - expert-offload
+  - expert-cache
+  - autoround
+  - vllm
+timestamp: 2026-09-30
+---
+
+# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+
+> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
+> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
+> **failure**: after five patches the drafter served and drafted (12.8 tok/s
+> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
+> **acceptance pinned at 1.00-1.38 on every text class including verbatim
+> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
+> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
+> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
+> untested alternative capture semantics: post-layer output state
+> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
+> (170 GiB) + drafter (2.2 GiB) staged to
+> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
+> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
+> generic unifier/tailpin/grouping fixes, apply-verified against the image.
+
+
+> **Question this page answers.** Can the fleet serve the 320B-class
+> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
+> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
+> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
+
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
+[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
+[[VLLM Index]], [[LLM/index|LLM fleet]].
+
+## The deployed shape
+
+| term | value |
+| --- | --- |
+| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
+| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
+| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
+| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
+| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
+| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
+| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
+
+```mermaid
+graph LR
+    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
+    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
+    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
+    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
+    D --> F[73.64 GiB pinned host rows / rank]
+    E --> G[decode step]
+    F -->|host misses<br/>1.05% of device time| G
+```
+
+## Why this payload and not another
+
+The W4A16 fast path on this fork is reachable **only** through the INC
+(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
+for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
+their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
+offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
+cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
+
+The chain that makes Intel's checkpoint work, with the exact evidence:
+
+```
+[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
+[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
+[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
+[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
+[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
+                           host 73.64 GiB pinned, host rows prepacked on the GPU
+```
+
+`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
+format-agnostic but requires the experts class to expose the hot/cold split.
+
+## The numbers, and the wall
+
+Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
+
+| depth | prefill | decode |
+| --- | --- | --- |
+| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
+| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
+| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
+
+Decode is flat across context and across every arena change — the signature of a
+per-step cost that is not KV- or residency-driven at the margin.
+
+### The measured profile, and the real wall
+
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
+
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
+| --- | --- | --- | --- |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
+
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
+
+Two consequences that kill three whole families of planned work:
+
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+  accident.** Pointwise cost scales with the token count, so a draft batch that
+  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+  the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+   `Dynamo does not know how to trace the builtin 'posix.stat'` —
+   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+   wrapped body to build its guard even when the cache is warm (it warns about
+   exactly this), verified in-image — a *warm* lru_cache-calling function still
+   fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+   four processes) and die on
+   `RuntimeError: Graph breaks are not supported with aot compile`, because this
+   build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
+   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
+   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
+   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
+   submodule attribute name, also present in the `load_weights` shard table
+   (`model.py:780-785`). Reading the full traceback (the reported line is inside
+   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
+   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
+   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
+   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
+   from those names at :1225. So the compiler is failing to **resolve a submodule by
+   name** while tracing the GLM decoder layer; whether the root cause is the fork's
+   module tree or dynamo's registration is not yet settled. The next step is the one
+   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
+   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
+   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
+   environment).
+
+4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
+   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
+   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
+   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
+   break dynamo reported; after this patch the report moves on to the workspace
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
+   below.
+5. **A launcher bug that made four boots misleading — worth knowing because the
+   failure kept moving.** The scratch launcher's compile branches were restructured
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
+   whichever break came first:
+
+   | config | outcome |
+   | --- | --- |
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
+
+**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
+with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
+check which configuration you are actually in (`podman exec <ctr> printenv
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
+survives) and it prints, at the miss, the node target, the near-miss keys, and
+whether another live `OutputGraph` holds the key.
+
+**State of the changelist (2026-09-30, latest).** The trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
+`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
+
+- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
+  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
+  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
+  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
+  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
+  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
+  is how inlined frames surface).
+- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
+  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
+  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
+  with the probe confirmed installed in both the API-server and worker processes.
+- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
+  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
+  the `load_weights` shard table (`model.py:780-785`). The four
+  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
+  exactly this bare key, but every one of them is guarded (`if proj_name not in
+  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
+  keyed by class name, not module path.
+- It reproduces on **every** run of this configuration (5 of 5), and it is
+  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
+  that were never fatal in the first place — the KeyError is the only thing standing
+  between this changelist and a boot.
+
+**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
+`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
+frames with their source), or wrap the inlining candidates in that forward — the mHC
+CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
+indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
+is a dynamo-side name-resolution failure and not a fixable fork bug.
+
+Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
+fallbacks on the decode path that the fork author already replaced, so they are not
+the blockers they appear to be at first glance.
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+  sources. The obvious second lever — the 9 % router — is gated
+  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+  (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+**The load was tried, and the gap is exactly one interface.** Payload fetched to
+`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
+2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
+`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+accepts the config and the drafter and dies at worker init with:
+
+```
+RuntimeError: Model does not support EAGLE3 interface
+```
+
+So nothing about the drafter, the method name, or the payload is wrong — the GLM
+trunk simply does not implement the aux-hidden-state contract the dflash runner
+requires. The reference is in this same image, twice: `deepseek_v2.py`
+(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
+capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
+small:
+
+- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
+  by the wrapper's `set_aux_hidden_state_layers()`;
+- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
+  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
+  rank holds fewer tokens than `positions`);
+- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
+  list is non-empty;
+- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
+  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
+  the runner derives trunk indices `[6, 15, 25, 34, 43]`).
+
+**The GLM-specific unknown** is what "the single-stream state entering layer k"
+means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
+`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
+returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
+`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
+(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
+a semantic choice to validate by acceptance rate, not by inspection.
+
+**Budget no speedup.** Per the profile above, a verification pass pays pointwise
+cost proportional to its token count — the same reason MTP4 measured flat — and
+MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
+*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
+it is small), not as a performance fix, and the honest expectation on this shape is
+flat decode with a prefill regression like MTP's until the pointwise mass is fused.
+
+**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
+independent of the compile changelist — it patches the same file as compile-01, so
+compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
+implements exactly the contract read out of this image: `EagleModelMixin` on
+`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
+layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
+the three `SupportsEagle3` member attributes plus both interface methods on the
+wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
+`v1/worker/gpu/cudagraph_utils.py:621`).
+
+**The patch went through five versions, and the decisive discovery was where the
+instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
+is called on the model the runner LOADED — and for GLM-5.3-Flash that is
+`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
+which holds the text model as `self.language_model = Glm5NextForCausalLM`
+(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
+with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
+both wrappers** and is the first version to clear the gate (both TP workers log
+`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
+
+The full evolution, each an instructive trap:
+
+1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
+   are not the whole contract.
+2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
+   `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
+   resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
+   Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
+   cannot take `SupportsEagle3` in *any* position.
+3. **v3** declared the members as class attributes (MRO-safe) — but only on
+   `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
+4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
+   `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
+   `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
+   experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
+5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
+   (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
+   `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
+   `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
+
+**The lesson generalizes twice over**: when a runtime error names an interface, read the
+interface's declaration first — and establish WHICH INSTANCE the check runs against
+before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
+models, not pass-throughs, and the text model behind `self.language_model` is invisible
+to a check run on the wrapper.
+
+**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
+`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
+torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
+hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
+instead — it needs no startup hook.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
+
+### The KV-cache grouping journey: clearing the unifier, then the kpool tail
+
+With patch 01 v5 the gate passes and the boot moves to KV-cache layout
+construction — where the next two blockers live, both in
+`v1/core/kv_cache_utils.py`:
+
+1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
+   spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
+   entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
+   specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
+   `unify_kv_cache_spec_page_size` raises
+   `NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
+   MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
+   they CAN grow `block_size` when the max page is an exact multiple. The
+   indexer's page is just `tokens_per_state x` the MLA page, so
+   **`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
+   to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
+   layers then take the legal divisible block-growth branch, drafter + mamba
+   layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
+   passes, torch.compile passes, and the boot reaches cudagraph capture — the
+   NotImplementedError is gone.
+2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
+   `kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
+   `assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
+   `[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
+   `Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
+   page (8192 B) *divided* the raised max page, so the unifier's divisible branch
+   grew its `block_size` by the ratio and the view became
+   `[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
+   promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
+   so `disable_hybrid_kv_cache_manager` is unset and
+   `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
+   (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
+   growing its block) **verified 2026-09-30**: the assert is gone and the boot
+   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
+   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
+
+3. **The capacity wall: the generic path cannot alias the mamba states.**
+   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
+   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
+   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
+   per-token KV). Two coupled causes, read out of the image:
+   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
+     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
+     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
+     expert plan's KV registry is calibrated at 13,600 B/token). On the
+     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
+     unaliased state pages appears.
+   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
+     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
+     KV at the plan's reserve, so even a correct generic-path estimate would
+     be starved.
+
+   Learned en route: the engine AUTO-SETS the attention block size to
+   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
+   tokens to ensure that attention page size is >= mamba page size`), so every
+   attention group's block counts scale from that floor — a 16-token drafter
+   block would need 4x the shared `num_blocks`; with the global 2,432 all
+   groups align.
+
+   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
+   for the registry) keeps the aliased glm5 pool and extends it: the grouping
+   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
+   `MLAAttentionSpec` subclass — excluded before the exact-type
+   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
+   specs; the drafter layers form their own uniform group (per-group pages,
+   NO unification — the MLA block size stays at the engine-set 2,432 and the
+   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
+   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
+   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
+   equal) as aliased pool + appended draft/hidden tensors with their own
+   strides; the memory estimator adds the extras. Patch 04b extends the
+   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
+   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
+   `speculative_config.use_aux_hidden_state_outputs` is set.
+
+   The pure logic is smoke-tested locally (CPU torch + the image's real spec
+   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
+   layout recognition, builder block math and memory accounting all pass.
+   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
+   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
+   2,432 tokens ≈ 70,500 ≥ 65,536.
+
+All of these mount over site-packages via the scratch launcher's
+`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
+`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
+`expert_activation_registry.py`; apply-verified against the image's pristine
+files). The window script's decision pipeline was also fixed for a SIGPIPE
+death (`head -80` on a `podman logs` grep pipeline killed one run before
+log-save and restore; the log is now captured to a file FIRST, then grepped).
+
+
+## Measured arms (all 2026-09-30, one boot each)
+
+| arm | decode | prefill | verdict |
+| --- | --- | --- | --- |
+| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
+| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
+| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
+| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
+
+### Why MTP is rejected here (it is not that it fails)
+
+MTP **works** on GLM in this fork: the head loads, the architecture resolves to
+`Glm5NextMTPModel`, and acceptance is excellent —
+
+```
+SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
+Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
+Avg Draft acceptance rate: 69-97%
+```
+
+— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
+prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
+verification forward gathers the **union** of the batch's experts, so spec
+decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
+of arena. On a compute-bound (fully-resident) shape this would invert; here it
+does not.
+
+## Served surface (verified 2026-09-30 against the live engine)
+
+| capability | state | evidence |
+| --- | --- | --- |
+| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
+| completion | yes | `bench_glm.py` runs above |
+| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
+| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
+| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
+
+Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
+requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
+breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
+newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
+
+## Fork traps found (for the engine authors)
+
+1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
+   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
+   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
+   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
+   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
+   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
+   the MTP draft step demands an op that cannot exist here:
+   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
+   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
+   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
+   CDNA check, not the env var.)
+   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
+     more tokens in the same reserve, and this fork's reserve model is
+     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
+     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
+     dominated by the per-sequence KDA state (34 linear-attention layers) and the
+     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
+     cover — so halving the KV dtype cannot move it.
+   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
+     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
+     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
+     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
+     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
+     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
+     **do not spend effort here** — the payoff is capacity, and the reserve is
+     dtype-blind.
+2. **The plan's KV reserve under-models this model, and the cap enforces the
+   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
+   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
+   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
+   `--max-model-len` becomes unbootable:
+   `ValueError: To serve at least one request with the model's max seq len
+   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
+   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
+   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
+   fixed) — where the reserve would eat a third of the arena.
+   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
+   request raise the cap; **it is not part of the deployed shape**, which boots
+   on the plan's own reserve.
+3. **A window is forced, not chosen.** The production `max_model_len 131072` is
+   what the reserve model requires for this payload (below ~92k the boot
+   refuses); it is not a capacity decision.
+4. **Expert prefetch remains poison** — unchanged from the qwen campaign
+   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
+
+## Operations
+
+```bash
+# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
+# dirs are untouched on disk):
+git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
+./scripts/deploy.sh dankserver --service frank-primary
+
+# payload inventory (34 shards + 2 extras, weight-map index):
+bash scripts/download-frank-primary.sh --verify-only
+
+# live telemetry (needs the profiler window; with_stack=false is mandatory —
+# with stack the stop path hangs and the trace buffer grows to tens of GB):
+podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+```
+
+**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
+**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
+`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
+liveness probe a lying oracle for a restore — twice a restore script printed
+`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
+still in place. The restore oracle must be a **conjunction**: `is-active` ==
+`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
+mask symlink **and** the container is owned by the unit and not `Stopping`. The
+repair recipe, all four steps needed:
+
+```bash
+sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
+sudo systemctl daemon-reload
+sudo systemctl reset-failed frank-primary.service                            # a failed unit refuses to start
+podman rm -f frank-primary-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
+sudo systemctl start frank-primary.service                                   # then ~7 min to health 200
+```
+
+Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
+glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
+container, so one whole round produced no evidence at all (`no container with name
+glm-probe found`) — check `proc://` for running jobs before opening a window.
+
+**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
+bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
+name-map lookups and vLLM module forwards) produced a self-replicating chain of
+`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
+imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
+that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
+the hook) and spawned again. It was still alive 18 minutes later with the container
+still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
+identical interpreters, each child's PPID being the previous one. Guard (now in the
+hook): arm once per environment via a marker in `os.environ`, which children inherit —
+
+```python
+if os.environ.get("VLLM_PROBE_ARMED") == "1":
+    pass                       # an ancestor already armed it; stay inert
+else:
+    os.environ["VLLM_PROBE_ARMED"] = "1"
+    <install probes>
+```
+
+Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
+generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
+variable — **never** pattern-kill an interpreter name, because the media stack on this
+host (romm, jellyfin, plex, glances) runs python in production.
+
+```bash
+# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
+
+# experiment windows (production stopped + `systemctl mask --runtime`, which
+# self-clears at reboot) ran from the host-side scratch launcher
+# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
+# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
+# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
+# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
+# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
+# word-split by the launcher's unquoted $EXTRA).
+bash /home/dank/serve_glm_autoround.sh
+```
+
+Boot gate: the engine refuses a legal-looking shape in two measured ways — the
+KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
+64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
+what killed the first two boots at util 0.95.
+
+`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
+shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
+the default allocator's fragmentation fails the profile/graph-capture phase
+(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
+MiB is free").
+
+The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
+walks every guarded vhost — including `http://192.168.0.112:8989/` on
+FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
+timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
+healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
+`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
+and the WoL packet did not wake it (macOS "Wake for network access" needs
+toggling on the mini itself).
+
+## Rollback targets
+
+| state | where |
+| --- | --- |
+| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
+| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |

serverAppConfigs-docs/LLM/index.md

@@ -1,10 +1,13 @@
 # LLM Fleet — Knowledge Base
 
-Documentation for the LLM inference fleet on the two hosts (2026-09-27):
-**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
-serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
-PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
-shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
+Documentation for the LLM inference fleet on the two hosts (2026-09-30):
+**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
+digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
+Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
+Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
+of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
+bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
+Dankclaw runs two llama.cpp endpoints —
 stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
 Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
@@ -22,11 +25,11 @@ graph LR
     subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
         A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[R9V radiance-vLLM fork, docker-pinned
-via docker-podman shim
-frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next Q8_0
-TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
+        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+podman run, frank-primary, port 7770]
+        C[GLM-5.3-Flash W4A16 int4
+TP=2, expert offload 73.64 GiB/rank pinned,
+14.51 GiB LRU arena, 131072 ctx, vision, tools]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -58,20 +61,26 @@ frank-small, port 7771]
 
 ## Active Deployment
 
-- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  runbook (2026-09-27, root systemd unit, port 7770): the R9V
-  radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
-  image bundle `2dac17a2`, driven through a docker→podman shim) serving
-  the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
-  blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
-  (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
-  131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
-  `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
-  inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
-  `reasoning`, parser `qwen3_coder`. The tuned shape lives in the
-  on-host q8xl setup state. Rollback:
-  [[frank-primary-r9v-radiance-qwen38-uncensored]]
-- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
+  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
+  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
+  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
+  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
+  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
+  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
+  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
+  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
+  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
+  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
+  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
+  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
+  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; payload and R9V state dirs intact on disk, git tag
+  frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
+ [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
   (same-day swap to the Q8_0 payload; rollback target, state + payload
   intact on disk): the R9V radiance-vLLM fork serving
   Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +

serverAppConfigs-docs/index.md

@@ -169,6 +169,18 @@ The LLM inference fleet on the two hosts. Full index:
   concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
   fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
   slab size)
+- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
+  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
+  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
+  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
+  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
+  checkpoint of the same model reaches TritonWNA16Experts, which has no
+  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
+  35 tok/s target is NOT met** — the step is bounded by the fork's gather
+  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
+  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
+  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
+  `glm47` tool/reasoning parser wiring
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
   vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
   allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4

serverAppConfigs-docs/log.md

@@ -1,5 +1,151 @@
 # Bundle Change Log
 
+## 2026-10-01
+- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
+  GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
+  served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
+  captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
+  interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
+  draft+hidden groups, registry method-signal), but **acceptance pinned at
+  1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
+  aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
+  8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
+  (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
+  not originate from this engine (APIServer access log: every completion
+  POST was host-local; his workstation appears once, a /v1/models GET).
+  Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
+  (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
+  (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
+  KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
+  unifier/tailpin/grouping/registry fixes, apply-verified).
+
+## 2026-09-30
+- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
+  the 35 tok/s target, and the shortfall is measured rather than inferred.**
+  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
+    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
+    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
+    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
+    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
+    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
+    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
+    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
+    refused to serve at all. That checkpoint was purged with the other retired
+    payloads.
+  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
+    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
+    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
+    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
+  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
+    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
+    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
+  - **The wall — and a corrected one.** The first account of the shortfall
+    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+    fusions that DO happen 6 %. The real cause is that the fork compiles its own
+    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+    engine says so at boot ("...provides neither a compiled submodule nor
+    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+  - **Fork work staged (compile changelist, WIP — does not boot yet).**
+    `scripts/patches/glm53-compile-01-model-decorator.patch`,
+    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+    evidence header and base/patched sha256. Boots established, in order: (1)
+    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+    available -> os.path.exists` breaks the graph (a plain module-global read
+    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+    `fullgraph` because its indexer metadata syncs to host
+    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+    outright. With all three the trunk **traces** (`Dynamo bytecode transform
+    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+    mHC/KDA path — a latent bug the compile exposes. That is the remaining
+    blocker.
+  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+    sources), which binary-blocks the second-largest lever: the fused router is
+    gated `num_experts in {128, 256, 512}` and only
+    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+    reserve is dtype-blind), only removing gfx12's only MLA decode path.
+  - **DFlash2 for GLM is a target-side port, not a drafter port.**
+    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+    proportional to its token count.
+  - **DFlash2 tried, and the gap was exactly that one interface — now closed.**
+    Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
+    2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
+    `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+    accepted the config and the drafter and died at worker init with
+    `RuntimeError: Model does not support EAGLE3 interface`. The port
+    (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
+    hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
+    text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
+    wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
+    `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
+    `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
+    workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
+    DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
+    by experiment-window RAM contention (see the window discipline below), so the
+    drafter-serving end state was still unproven when this entry was written.
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
+    indexer's in-forward workspace acquisition
+    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
+    — and the cause was **my own scratch launcher**: its compile branches were
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
+    (the flag only logs). The moving reports are still useful — they are the break
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
+  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
+    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
+    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
+    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
+    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
+    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
+    chaining the image's apport hook), and the scratch launcher used to reset
+    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
+    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
+    the pinned image and to reproduce their recorded sha256.
+  - Measured and rejected: **MTP4** (the head loads, arch resolves to
+    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+    The profiler window above explains it: the per-token cost is dominated by
+    pointwise glue that scales with the token count, so verifying a draft batch
+    pays ~N x that glue for the same N tokens of output);
+    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
+    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
+    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
+  - Fork bug found: the plan's KV reserve under-models this model's
+    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
+    that reserve, making a legal `--max-model-len` unbootable. Shipped
+    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
+    request raise the cap) — NOT needed by the deployed shape.
+  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
+    state dirs are intact on disk.
+
 
 ## 2026-09-29
 - **frank-large tuning investigation: a measured NEGATIVE result, nothing

tests/frank_primary_unit.bats

@@ -1,17 +1,26 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the R9V radiance-vLLM ROOT-UNIT shape (the 2026-09-27
-# swap that replaced the llama.cpp b11030 endpoint): the R9V fork's own
-# launcher (scripts/launch.sh in the pinned fork checkout) driven through a
-# docker->podman shim, with the qualified on-host setup state layered under
-# the repo's production overrides. The seam pins what the flags alone do not
-# say: the state-layering contract (104 tuned R9V_* keys live on-host; the
-# repo env carries only production overrides), the self-healing law (the
-# launcher detaches, so the serve script must wait on the container for
-# Restart=on-failure to mean anything), the memlock law that makes the
-# tiered-expert pin legal, and the deployer's state/payload/image gates with
-# the false-green MainPID check.
+# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
+# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
+# reversed the failed GLM/DFlash2 experiment): the serve script drives
+# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
+# engine flag qualified by boot probe and carried in the repo env file. The
+# seam pins what the flags alone do not say: the KV qualification law
+# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
+# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
+# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
+# together), the automatic-offload law (the fork plans the tiered-expert
+# offload itself; no flag exists to set), the self-healing law (the run
+# returns before the engine is up, so the serve script must wait on the
+# container for Restart=on-failure to mean anything), the memlock law that
+# makes the pinned host rows legal, the deployer's image/payload gates with
+# the false-green MainPID check, and the measured laws: expert prefetch
+# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
+# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
+# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
+# carried the prefetch tax), and NO tool dialect (the GLM era added
+# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
 
 load helpers
 
@@ -19,6 +28,7 @@ UNIT="$REPO_ROOT/scripts/frank-primary.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
+DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 
 @test "frank-primary unit, serve script and env exist in scripts/" {
     [ -f "$UNIT" ]
@@ -45,90 +55,178 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 }
 
 @test "the unit stops the container child explicitly" {
-    # The serve script blocks on docker wait, which cannot stop the
+    # The serve script blocks on podman wait, which cannot stop the
     # container child; systemd stop goes through ExecStop or the engine
-    # outlives the unit and holds both GPUs.
+    # outlives the unit and holds both GPUs. The container name must match
+    # the env file's identity.
     grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
-    grep -q 'frank-primary-r9v' "$UNIT"
+    grep -q 'frank-primary-qwen' "$UNIT"
+    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
 }
 
-@test "the serve shape is the R9V launcher, not a frozen podman run" {
-    # The launcher owns the resolved container args (binds, overlays, device
-    # passthrough); a frozen copy of its argv would drift on every fork
-    # update. The llama.cpp era's flags would also silently no-op here.
-    grep -q 'launch.sh' "$SERVE"
+@test "the serve shape is direct podman run, not a launcher or shim" {
+    # The tcclaviger stack is fully expressed by flags; the R9V launcher,
+    # docker->podman shim and setup-state layering are the RETIRED shape.
+    grep -q 'podman run' "$SERVE"
+    ! grep -q 'launch.sh' "$SERVE"
+    ! grep -q 'podman-shim' "$SERVE"
+    ! grep -q 'R9V_STATE_FILE' "$SERVE"
     ! grep -q -- '--model-draft' "$SERVE"
     ! grep -q -- '--spec-type' "$SERVE"
-    ! grep -q -- '--mmproj' "$SERVE"
-    ! grep -q -- '--override-tensor' "$SERVE"
-    ! grep -q 'podman run' "$SERVE"
 }
 
-@test "the tuned engine shape comes from the on-host setup state, not the repo" {
-    # The profile setup qualified the tuned shape (tiered-expert placement,
-    # per-rank offload budgets, MTP4, parser wiring). Copying it into
-    # the repo would drift; the serve script layers it and skips keys the
-    # env file defines, so production overrides always win.
-    grep -q 'R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json' "$ENV_FILE"
-    grep -q 'state must carry R9V_MODEL_DIR' "$SERVE"
-    grep -q 'is not ready' "$SERVE"
-    grep -q 'key not in env_keys' "$SERVE"
+@test "the qualified engine shape is in the env file" {
+    # 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
+    # 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
+    # reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
+    # PER RANK — sizing it to the global requirement OOMs the worker
+    # (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
+    # full-length 200k contexts do NOT fit: four clients share the ~730k
+    # token pool and each runs below 200k when all are deep.
+    grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
-@test "the serve script gates the pinned image, PLE bytes and GGUF shards" {
-    # The image is answered by its OCI manifest digest (podman keeps the
-    # manifest digest in RepoDigests, not as the image ID — the shim's ID
-    # law). The PLE payload is checked at its pinned byte count
-    # (stat -c %s, never wc -c). A missing shard means a guaranteed
-    # long-timeout boot instead of a fast failure.
-    grep -q 'R9V_PLE_EXPECTED_BYTES' "$SERVE"
-    grep -q 'stat -c %s' "$SERVE"
-    grep -q 'R9V_TARGET_SHARD2_REL' "$SERVE"
-    grep -q 'R9V_MMPROJ_REL' "$SERVE"
+@test "the serve script wires the engine flags from the env" {
+    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
+    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
+    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+    grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
+    grep -q -- '--tensor-parallel-size 2' "$SERVE"
+    grep -q -- '--enable-prefix-caching' "$SERVE"
+    grep -q -- '--trust-remote-code' "$SERVE"
+}
+
+@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
+    # The 2339fae trio does NOT wire --enable-auto-tool-choice /
+    # --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
+    # dropped them). tool_choice "auto" gets a 400 here — tool-using agents
+    # must route at frank-small or frank-large until a parser is qualified
+    # for this arm (a behavior change beyond the rollback, not a silent fix).
+    ! grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    ! grep -q -- '--tool-call-parser' "$SERVE"
+}
+
+@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
+    # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
+    # expert planner moves the overflow to pinned host rows by itself (the
+    # arena 6,574 slots x 2.344 MiB plus host rows). There is no
+    # --enable-expert-offload flag in this arm; the legality of the pin is
+    # the unit's unbounded LimitMEMLOCK.
+    ! grep -q -- '--enable-expert-offload' "$SERVE"
+    ! grep -q -- '--expert-precise-memory' "$SERVE"
+    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
+    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
+}
+
+@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
+    # 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
+    # tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
+    # baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
+    # 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
+    # (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
+    # the prefetch tax and is retired with it. ngram-mod never booted on this
+    # model (fork bug: PLE inputs prepared only on the MTP path).
+    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
+    grep -q -- '--speculative-config' "$SERVE"
+}
+
+@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
+    # --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
+    # them together; a V-only quant needs a fork patch). The GLM-era MLA
+    # concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
+    # to this arm: qwen's attention path accepted the dtype and a completion
+    # probe qualified it 2026-09-29.
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+}
+
+@test "expert prefetch is absent BY LAW, not by omission" {
+    # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
+    # prefetched slabs (36% precision) — roughly half the gather traffic was
+    # waste, and the "savings" would have been installs anyway. Deleting the
+    # flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
+    # fresh A/B, so the serve script must NOT wire it unconditionally.
+    ! grep -q -- '--expert-prefetch' "$SERVE"
+    ! grep -q 'expert_prefetch' "$ENV_FILE"
+    grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
+}
+
+@test "TunableOp stays off" {
+    # CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
+    # CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
+    # (repetition loops, early halts, invalid measurements).
+    ! grep -q 'TUNABLEOP' "$ENV_FILE"
+    ! grep -q 'TUNABLEOP' "$SERVE"
+}
+
+@test "the serve script gates the digest-pinned image and the payload" {
+    # The image is pinned by OCI manifest digest (the floating :latest moved
+    # once already). The payload gate checks config.json and the exact shard
+    # count from the env — a missing shard means a guaranteed long-timeout
+    # boot instead of a fast failure.
+    grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
     grep -q 'RepoDigests' "$SERVE"
+    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
+    grep -q 'config.json' "$SERVE"
     grep -q '/dev/kfd' "$SERVE"
+    grep -q -- '--device /dev/dri' "$SERVE"
+    ! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
 }
 
-@test "the serve script clears the stale container and self-heals via docker wait" {
-    # launch.sh detaches and refuses an existing container name; a crashed
-    # unit can leave the name registered forever after. Clear our own name
-    # first (idempotent), then block on the container so a crash is a
-    # nonzero ExecStart.
-    grep -q 'docker rm -f "$R9V_CONTAINER_NAME"' "$SERVE"
-    grep -q 'exec docker wait "$R9V_CONTAINER_NAME"' "$SERVE"
+@test "the serve script clears stale containers and self-heals via podman wait" {
+    # A crashed unit can leave the container name registered forever after;
+    # the legacy identities (the retired R9V container and the ad hoc
+    # experiment container) hold the same GPUs and die here too. Then block
+    # on the container so a crash is a nonzero ExecStart.
+    grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
+    grep -q 'frank-primary-r9v' "$ENV_FILE"
+    grep -q 'qwen-fp8' "$ENV_FILE"
+    grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
 }
 
 @test "frank-primary holds the single fleet port under its own alias" {
-    # The scratch identity (r9v-qwen38-flash-next on 127.0.0.1:7772) must
-    # never leak into production: LAN bind, :7770, the harness's model id.
-    grep -q '^R9V_HOST_BIND=0.0.0.0$' "$ENV_FILE"
-    grep -q '^R9V_HOST_PORT=7770$' "$ENV_FILE"
-    grep -q '^R9V_SERVED_MODEL_NAME=frank-primary$' "$ENV_FILE"
-    grep -q '^R9V_CONTAINER_NAME=frank-primary-r9v$' "$ENV_FILE"
-    grep -q '^R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0$' "$ENV_FILE"
+    # The scratch identity (qwen3.8-flash-next on :7773) must never leak
+    # into production: LAN publish of :7770, the harness's model id.
+    grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
+}
+
+@test "the model id matches the payload's served alias wiring" {
+    # The payload dir and its shard count are the fetch script's output
+    # layout; the fetch script must write where the env points.
+    grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
+    grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
+    grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
 }
 
 @test "no bare HIP-reserved env var is defined" {
     # HIP reads bare TEMP for its JIT cache path (2026-08-13: TEMP=1.0
     # segfaulted every load), and HOST/GPU_MAX_HEAP_SIZE are the same
-    # class. The bind address is R9V_HOST_BIND, not HOST.
+    # class. Every key here is FRANK_PRIMARY_-prefixed.
     ! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
     ! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
 }
 
-@test "the deployer gates the setup state, image digest, PLE bytes and shards" {
-    # The deployer refuses when the on-host setup state is missing or not
-    # ready, when the pinned image is absent from rootless storage, when
-    # the PLE payload is off its pinned byte count, or when any GGUF shard
-    # or the mmproj is missing — then clears stale containers, waits out
-    # the port, relinks/enables/restarts the root unit and health-gates it.
-    grep -q 'LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json' "$REMOTE_DEPLOY"
-    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-r9v' "$REMOTE_DEPLOY"
-    grep -q 'is not ready' "$REMOTE_DEPLOY"
+@test "the deployer gates the image digest and payload, then health-checks" {
+    # The deployer refuses when the pinned image is absent from rootless
+    # storage or the payload is missing/wrong-sharded — then clears stale
+    # containers (including the legacy identities), waits out the port,
+    # relinks/enables/restarts the root unit and health-gates it.
+    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
+    grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
     grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
-    grep -q 'stat -c %s' "$REMOTE_DEPLOY"
-    grep -q 'R9V_TARGET_SHARD2_REL' "$REMOTE_DEPLOY"
-    grep -q 'podman rm -f "$LLM_PRIMARY_CONTAINER"' "$REMOTE_DEPLOY"
+    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
+    grep -q 'config.json' "$REMOTE_DEPLOY"
+    grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
+    grep -q 'glm-probe' "$REMOTE_DEPLOY"
     grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
     grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
     grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"