frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)

bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real error (captured from the journal this time, container fixed via journald rather than podman logs): RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked it as dynamic but your code specialized it to be a constant (3)' - a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem. The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 / 792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window, with --enforce-eager as the bypass.
frank-primary: drop --expert-precise-memory (boot crash loop with bf16)
2026-10-01 14:40:17 -07:00 · 2026-10-01 14:16:47 -07:00 · 2026-10-01 14:08:51 -07:00 · 2026-10-01 14:08:50 -07:00 · 2026-10-01 14:03:25 -07:00 · 2026-10-01 13:52:13 -07:00
32 changed files with 3846 additions and 533 deletions
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
-| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
+| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | `unsloth/Qwen3.8-27B-GGUF` **UD-Q4_K_XL** (27B dense, **vanilla — no longer abliterated**, 16.35 GiB, Apache-2.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), **4 sessions x 163840 ctx, unquantized (f16) KV** — gufo has no KV-quant lever — greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only. Swapped 2026-10-01 from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB): ~13 GiB less weights is what funds four f16 sessions, at the cost of the uncensored behaviour | 7770 |
+| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |

 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -243,9 +243,9 @@ merely wedged kernel usually still answers ARP.
 1. Stop and mask (self-healing would restart it):
   - Dankclaw `frank-large` (root unit, gufo on the iGPU):
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
+   - Dankclaw `frank-small` (root unit, SGLang on the 3090):
     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
   NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -277,19 +277,20 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-26 both `frank-small` and `frank-primary` accept images
+of 2026-10-01 both `frank-small` and `frank-primary` accept images
 (`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
-(this build never autoloads projectors); `frank-large` remains text-only
-because its stock llama.cpp deployment loads no
-vision projector (the mmproj is not loaded). Note also that
-the reasoning channel differs by runtime: llama.cpp endpoints emit
-`reasoning_content`, vLLM emits `reasoning` — academic since the
-2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
+(this build never autoloads projectors), and frank-small's MiMo payload
+bakes its vision tower INTO the checkpoint (no projector file at all);
+`frank-large` remains text-only because its gufo deployment loads no
+vision projector. Note also that
+the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
+parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
+models.yml sets `reasoningContentField` per endpoint to match.
 There is **no audio**
-input anywhere in the fleet except frank-small: that capability left with the
-FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
-projector's audio encoder (frank-small transcribes WAV verbatim even though
-`audio` is omitted from its `input` list — OMP's schema rejects it).
+input anywhere in the fleet: that capability left with the FLM NPU
+endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
+audio encoder, and left again with the Gemma4 retirement on 2026-10-01
+(the MiMo tower is vision-only).

 ### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)

@@ -397,112 +398,89 @@ no longer a one-minute flip. **Engine** —
 Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
 `Qwen3.8-27B-UD-Q8_K_XL/`.

-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)

-Mainline llama.cpp master `37b53fd` (CUDA build with
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
-port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
-(four FULL 262,144-token slots — the measured card ceiling), batch
-**8192/1024**, greedy default. Deployed flags are in
-`scripts/frank-small.env`; the serve script refuses to start if the
-weights, the projector, the sidecar or the binary are missing.
+SGLang (latest) in a **self-contained python3.12 venv**
+(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root
+systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into
+`/etc/systemd/system/`), serving
+`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense
+qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16
+AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.

-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
-chat completion returns content plus `reasoning_content`; a tool call
-returns `tool_calls`; an image request is answered from the projector;
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
-100 is WORSE at long ctx) is tabled in the runbook.
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
-Rollback: [[frank-small-qwen38-gsq-rco]].
+**The shape is memory-pinned, and the sharp edge is the mamba pool.** The
+DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of
+dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
+The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
+the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
+deferred kernel loads eat and the first long request OOMs on. Two
+262,144-token slots at the model's NATIVE window (2026-10-01 reslice from
+4 x 151,552: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and
+a compaction request — frank-primary's trigger is 183,501 tokens plus a
+32,768-token summary budget = 216,269 — now fits ONE request); a single
+request may grow to the whole pool. Prefill runs **eager**
+(`--disable-prefill-cuda-graph`) — there is no headroom for capture;
+decode graphs are on.

-### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
+**No speculative decode — a checkpoint fact.** `text_config` declares
+`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this
+INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with
+an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates
+that no `--speculative-*` flag reaches the engine.

-The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
-prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
-the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
-with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
-PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
-**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
-n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
-template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
-`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
-over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
-llama.cpp b11030 the same day) at Frank's direction.
+**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is
+12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local
+patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept).
+Details in the runbook.

-**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
-run qualified the `R9V_*` config into
-`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
-file `scripts/frank-primary.env` carries ONLY production overrides
-(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
-`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
-`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
-(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
-quantizes BOTH fused hyper-connection shards, so the shard-uniformity
-check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
-overlay's env.always injects the uncensored 160-slot cache geometry, which
-the q8 80-slot tiered cache rejects). Never copy tuned values from the
-state into the repo — it would drift from the qualification.
+Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
+chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
+llama.cpp's field name); a red-square image answers "Red"
+(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token
+requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s
+eager). At the 2 x 262,144 shape (reslice, same day): boot flags read
+`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`;
+a 200,000-token prompt — well past frank-primary's 183,501 compaction
+trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056
+tok/s eager); two concurrent 50k-token requests both answered. Decode
+**83.9 tok/s**.
+Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
+Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).

-**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
-single dict — one method per engine, NO chaining — so the llama.cpp
-`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
-(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
-architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
-single method, not a chainable one.
+### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)

-**Serving = the fork's own launcher through a docker→podman shim.** The R9V
-tooling is docker-pinned; the shim at
-`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
-three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
-the image by its OCI manifest digest while podman keys by config digest but
-preserves the manifest digest in `RepoDigests` — the shim answers
-`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
-refs to `repo@sha256:X`; (2) podman inspect answers
-`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
-(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
-and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
-/`ServerVersion` info fields — the shim answers them.
+The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
+(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
+**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
+**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
+GLM/DFlash2 experiment was declared failed (same day).

-**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
-54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
-quadlet would be capped by the user manager's limit).
-The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
-(ready + paths), the pinned image (RepoDigests), the PLE byte count
-(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
-clears a stale container, then execs the fork's `scripts/launch.sh` through
-the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
-without the wait a crashed engine is a green unit; `docker wait` cannot stop
-the container child, hence `ExecStop=podman stop -t 120
-frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
-PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
-restarts the unit and health-gates with the MainPID check (the false-green
-stray law).
+Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
+~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
+35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
+BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
+throughput one. V-only q8 KV is NOT possible via config: stock
+`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.

-Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
-payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
-battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
-length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
-mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
-round-trip + template probe re-verified on :7770 after the deploy. No
-throughput bench was run (Frank declined); the probe-era 11 tok/s decode
-was residency-bound (halved expert headroom at double the bytes/expert),
-not the production number.
+The engine auto-sets the attention block size (mamba-page floor); the
+expert plan's registry prices KV per token; the expert arena (6,574 slots
+x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
+for KV headroom collapses the offload hit rate.

-Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
-Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
-`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
-and the template keys, redeploy (state + `restore-config-20260927.env` +
-payload intact on disk) — see
-[[frank-primary-r9v-radiance-qwen38-uncensored]].
+Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
+`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
+frank-primary-glm`. The serve script drives `podman run` directly, gates
+the digest-pinned image, the payload's `config.json` and its exact
+131-shard count and the GPU device nodes, clears stale container
+identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
+same, waits out :7770, relinks/enables/restarts the unit and health-gates
+with the MainPID check. Boot cost ~7 min warm; health returns 503 while
+loading.
+
+Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.

 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -850,8 +828,8 @@ Active:
 | Service | Runbook |
 |---------|---------|
 | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
+| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` |
+| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

@@ -859,6 +837,7 @@ Retired (rollback targets and history):

 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
 | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
@@ -872,6 +851,7 @@ Retired (rollback targets and history):
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
 | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
@@ -1177,23 +1177,39 @@ deploy_llm_large() {
    wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }

-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
-# image to install and no generated unit to avoid. relink_unit points
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
-# is that unit's [Install] section.
+# deploy_llm_small — frank-small is SGLang (python3.12 venv) on the RTX 3090,
+# as a ROOT unit: the runtime is a self-contained venv, so there is no
+# container image to install and no generated unit to avoid. relink_unit
+# points /etc/systemd/system/frank-small.service at the repo file and boot
+# persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+LLM_SMALL_MODEL_DIR="${LLM_SMALL_MODEL_DIR:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/sglang-cu128/bin/sglang}"

 deploy_llm_small() {
-    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
-        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
+    # Refuse rather than restart-loop: a unit whose runtime or payload is
+    # missing never becomes healthy, and a green deploy would leave the
+    # endpoint dark with nothing in the log to say why.
+    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
+        echo "ERROR: SGLang runtime absent at $LLM_SMALL_RUNTIME" >&2
+        echo "frank-small: build the venv (python3.12 -m venv /home/dank/GitPackages/sglang-cu128" >&2
+        echo "frank-small: and pip install 'sglang[all]' into it) first." >&2
+        return 1
+    fi
+    if [[ ! -f "$LLM_SMALL_MODEL_DIR/model.safetensors.index.json" ]]; then
+        echo "ERROR: payload index absent at $LLM_SMALL_MODEL_DIR" >&2
+        echo "frank-small: run scripts/download-frank-small.sh first;" >&2
        echo "frank-small: refusing to start a unit that cannot load." >&2
        return 1
    fi
+    local count
+    count="$(find "$LLM_SMALL_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
+    if [[ "$count" -ne 8 ]]; then
+        echo "ERROR: payload shard count wrong: $LLM_SMALL_MODEL_DIR (found $count, want 8)" >&2
+        return 1
+    fi
    relink_unit frank-small
    run_privileged systemctl daemon-reload
    run_privileged systemctl enable frank-small.service
@@ -1218,7 +1234,7 @@ deploy_llm_small() {
 # sha256 verification was the fetch script's job at download time, never
 # re-hashed here.
 LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"

 deploy_llm_primary() {
    # Refuse rather than restart-loop: a unit whose image or payload is
@@ -1237,7 +1253,13 @@ deploy_llm_primary() {
        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
        return 1
    fi
-    if ! podman images --format '{{.RepoDigests}}' | grep -qF "${image#*@}"; then
+    # Capture podman's output BEFORE grepping: under set -o pipefail, the
+    # classic `podman images | grep -q` races (grep exits on match, podman
+    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
+    # 2026-09-30 deploy refusal).
+    local digests
+    digests="$(podman images --format '{{.RepoDigests}}')"
+    if ! grep -qF "${image#*@}" <<<"$digests"; then
        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
        echo "frank-primary: fetch it (podman pull $image)" >&2
        return 1
@@ -1264,8 +1286,9 @@ deploy_llm_primary() {
    # A stopped unit can leave its container running: systemd's cgroup kill
    # races conmon's --rm cleanup, and the orphan holds the port and GPUs.
    # Stop the unit, then remove the containers BEFORE waiting on the port.
-    # The legacy identities (the retired R9V container and the ad hoc
-    # experiment container qwen-fp8) hold the same GPUs and die here too.
+    # The legacy identities (the retired R9V container, the ad hoc
+    # experiment container qwen-fp8, the retired glm-probe) hold the same
+    # GPUs and die here too.
    run_privileged systemctl stop frank-primary.service 2>/dev/null || true
    local ctr
    for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the Q4_K_XL frank-large payload on Dankclaw:
+#   unsloth/Qwen3.8-27B-GGUF  ->  Qwen3.8-27B-UD-Q4_K_XL.gguf
+#
+#   bash scripts/download-frank-large-q4.sh
+#   bash scripts/download-frank-large-q4.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
+# the repo is public and ungated (gated: false), and a token on the curl
+# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
+# a token, put it in a mode-600 file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Qwen3.8-27B-UD-Q4_K_XL.gguf   17559178144  (16.35 GiB)
+#
+# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
+# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
+# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
+# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
+# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
+#
+# The speculative drafter is UNCHANGED and still fetched by its own script
+# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
+# appear here at all — the frank-large seam greps THIS file for it to prove
+# the drafter was not folded into the payload fetch.
+#
+# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
+# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
+# spec decode here is the block-diffusion drafter, not MTP.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
+REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
+SIZE=17559178144
+SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
+URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "Q4_K_XL payload complete: $MODEL_DIR"
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+#
+# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# the house rule.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#
+# LFS shards are verified against the Hub LFS sha256; non-LFS text files
+# against their declared byte size. Idempotent: complete files are skipped.
+set -euo pipefail
+
+REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
+  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
+  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
+  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
+  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
+  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "model.safetensors.index.json 106278 -"
+  "config.json 4608 -"
+  "generation_config.json 161 -"
+  "quantization_config.json 1654 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer_config.json 1168 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "README.md 8448 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -8,14 +8,20 @@
 # runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
 # exception because the serve script itself consumes it for --host.

-# The target payload (scripts/download-frank-large-swift.sh): the serve
-# script refuses to start without the main GGUF and the DFlash2 drafter.
-# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
-# are structurally interchangeable for gufo (same qwen35 architecture, the
-# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
-# layout, the same MTP head), so only this path changed.
-MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
-MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
+# The target payload (scripts/download-frank-large-q4.sh): the serve script
+# refuses to start without the main GGUF and the DFlash2 drafter.
+#
+# 2026-10-01 swap (Frank): unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, 16.35 GiB,
+# replacing ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB). Two
+# consequences worth knowing at a glance:
+#   * this is the VANILLA (non-abliterated) Qwen3.8-27B — the endpoint is no
+#     longer uncensored, which was one of the reasons the Swift payload was
+#     chosen on 2026-09-28;
+#   * ~13 GiB of weights became available for KV, which is what makes four
+#     full-sized f16 sessions fit.
+# gufo needs no template for either: its Qwen renderer is compiled in.
+MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL
+MODEL_FILE=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf

 # The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
 # the previous target, so it carries its own directory. It has a sliding
@@ -32,9 +38,21 @@ ALIAS=frank-large
 HOST=0.0.0.0
 PORT=7770

-# One session, full 262144 context (the model's native window).
-CONTEXT_SIZE=262144
-SESSIONS=1
+# Four sessions, 163840 context each (Frank, 2026-10-01: "4x 160k"). The
+# context is per session and the KV is UNQUANTIZED (f16) — gufo has no KV
+# quantization lever at all (execution_policy.hpp pins kv_cache_storage to
+# kFp16), so "unquantized" is inherent rather than configured.
+#
+# The arithmetic that makes this fit: f16 KV costs 64 KiB/token on this model
+# (16 full-attention layers x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes;
+# the other 48 layers are linear-attention with a per-SEQUENCE fp32 state of
+# ~144 MiB, not a per-token cache). 4 x 163840 = 655,360 tokens = 40.0 GiB of
+# KV, and each session's 163,840 tokens is exactly 10 GiB. Budget: weights
+# 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB
+# against the 114 GiB GTT — roughly 50 GiB of headroom, so the session count
+# or the per-session context can both grow from here.
+CONTEXT_SIZE=163840
+SESSIONS=4

 # DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
 # block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
@@ -1,37 +1,44 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
-# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
-# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
-# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
-# digest-pinned image the 2026-09-29 tuning campaign measured — serving
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
-# the MTP4 drafter on the two R9700s.
+# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
+# campaign's verdict, Frank's call): same digest-pinned engine and payload as
+# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
+# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
+# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
+# restorable from the tag.
 #
-# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
-#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
-#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
-#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
-#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
-#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
-#     free; 7.2 GiB boots).
-#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
-#     — the stock flag quantizes them together, which is exactly what was
-#     requested here). Accepted by the attention path; qualified with a
-#     completion probe.
+# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
+#   * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
+#     shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
+#     engine measured "KV cache size: 732,359 tokens; Maximum concurrency
+#     for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
+#     at graph capture (76 MiB short / 0 bytes free); 237568 passed at
+#     718,606 tokens and the +4096 bonus step passed and won.
+#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
+#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
+#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
+#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
+#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
+#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
+#     the gather win) — traded deliberately for clean cache + 262k windows.
+#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
+#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
+#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
+#     era). Do not drop it.
+#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
+#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
+#     and arena. 1024 keeps the workspace floor so the arena stays max.
 #   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
 #     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste; prefetch off is the tuned
-#     incumbent and the +40-51% decode came from deleting the flag. Never
-#     re-add --expert-prefetch without a fresh A/B.
+#     roughly half the gather traffic was waste. Never re-add
+#     --expert-prefetch without a fresh A/B.
 #   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
 #     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics (repetition loops, early halts, invalid measurements).
-# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
-# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
+#     numerics.
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
 # prefixed FRANK_PRIMARY_ so none can leak.
+#

 # --- runtime ----------------------------------------------------------------
 # Digest-pinned: the floating :latest tag moved once already. Local alias
@@ -50,12 +57,61 @@ FRANK_PRIMARY_SERVED_NAME=frank-primary
 FRANK_PRIMARY_PUBLISH=7770:7770
 # Containers removed before every start: the retired R9V identity and the ad
 # hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS=frank-primary-r9v qwen-fp8
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"

-# --- qualified engine shape ---------------------------------------------------
-FRANK_PRIMARY_MAXLEN=200000
+# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
+# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
+# the boot failed (`Engine core initialization failed`, NRestarts climbing).
+# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
+# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
+# while the bf16 path is investigated in a window (see log.md 2026-10-01).
+# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
+# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
+# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
+# the model's FULL native window with UNQUANTIZED KV, plus
+# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
+# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
+# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
+# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
+# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
+# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
+# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
+# the fp8 3-window shape used 28.27 and left 3.59).
+# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
+# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
+# 7.60 GiB default, moving the plan's `room` and therefore the arena
+# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
+# for this shape does not have 2.35 GiB to spare:
+#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
+#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
+# measured as a CrashLoop: every attempt reached graph capture, the engine
+# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
+# With the default overhead the arena is 14.31 GiB and the same shape totals
+# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
+# pool/context reduction that pays for it.
+# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
+# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
+FRANK_PRIMARY_MAXLEN=262144
 FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=7730934784
+FRANK_PRIMARY_KV_BYTES=6100000000
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_SPEC=4
+FRANK_PRIMARY_MNBT=2048
+FRANK_PRIMARY_GATHER_CHUNKS=32
+FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98
+
+# --- client contract: reasoning split, tools, preserve_thinking ----------------
+# Without --reasoning-parser the engine leaves the thinking text INLINE in
+# `content` with a dangling `</think>` and a null `reasoning` field (measured
+# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
+# malformed thinking block clients choke on. The qwen3 parser splits it
+# properly. The GLM rollback (d91379d) had dropped all three of these; the
+# working values are the ones ecaad36 shipped for this model.
+FRANK_PRIMARY_REASONING_PARSER=qwen3
+# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
+# tool_calls work here (without it, tool_choice auto returns 400).
+FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
+# preserve_thinking keeps prior thinking blocks in the rendered prompt across
+# turns; enable_thinking makes thinking the server-side default. Verified the
+# payload's tokenizer_config template consumes both keys.
+FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
@@ -8,8 +8,10 @@
 #
 # Same port, same served alias; the harness models.yml id is unchanged.
 # The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container, so a crashed engine is a
-# nonzero ExecStart and Restart=on-failure is real.
+# `vllm serve`) and then waits on the container. NOTE: that wait can return
+# 0 even when the engine died mid-request (vLLM's API server drains and
+# exits clean on a worker fault), which is why the restart policy is
+# `always` — see the [Service] note.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
 # of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
@@ -23,6 +25,9 @@
 Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0

 [Service]
 User=dank
@@ -39,8 +44,23 @@ ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
 # Boot cost: FP8 payload page-in plus torch compile/graph capture —
 # ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
-Restart=on-failure
+# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
+# torch.OutOfMemoryError kills the engine while vLLM's API server still
+# exits 0, so ExecStart returned clean, systemd logged "Deactivated
+# successfully", and on-failure NEVER fired — the endpoint stayed down until
+# a human noticed. `always` restarts on any container death; an explicit
+# `systemctl stop` still wins (systemd does not restart a unit the operator
+# stopped), so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
+# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
+# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
+# Must exceed the podman grace with margin for the engine's own shutdown.
+TimeoutStopSec=180
+# The engine pins tens of GiB of host RAM (expert host slabs); it is the
+# last process on this host that should be chosen by the kernel OOM killer.
+OOMScoreAdjust=-500

 [Install]
 WantedBy=multi-user.target
@@ -1,46 +1,38 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
+# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
+# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
+# Dankclaw's RTX 3090, since 2026-10-01.
 #
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
-#   cmake --build build-cuda --target llama-server -j
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
-# attention, and without the flag quantized-KV prefill collapses ~12x on
-# CUDA (see the CUDA lesson in AGENTS.md).
+# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
+#   python3.12 -m venv /home/dank/GitPackages/sglang-cu128
+#   /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
+# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
+# self-contained and does not touch the host Python (3.14) or the ROCm
+# stack that serves frank-large.
 #
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
-# only a 1024-position window, which is why the KV pool is a fraction of
-# the full-ctx x all-layers number.
+# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
+# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
+# safetensors shards, 12.0 GiB total. The vision tower
+# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
+# checkpoint, so the image path needs no separate projector file.
+# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
+# + 24 Gated DeltaNet, native 262144 window. text_config declares
+# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
+# tensors (verified against model.safetensors.index.json 2026-10-01 —
+# base BF16 and this INT8 conversion both omit the head), so there is no
+# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
+# model at 0% acceptance and pay the verify tax for nothing.
 #
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
-# context loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
-# is the only legal shape that carries both.
-#
-# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
-# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
-# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
-# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
-# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
-# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
-# which would shrink each slot below the full 262144 window once
-# parallel > 1.
-#
-# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
-# params; it carries an audio encoder too). The fleet's image path lives
-# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
-# WORKS: input_audio content parts transcribe verbatim (verified
-# 2026-09-17 with a 16k mono WAV).
+# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
+# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
+# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
+# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
+# take most of it after boot, and eager-prefill activations then OOM on
+# the first long request. The pool is therefore PINNED at 606,208 tokens
+# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
+# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
+# request and 4 concurrent 54k requests were verified at the 4-slot
+# shape.
+# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
 #
 # Sampling: greedy by default (the fleet contract); a client that sends a
 # temperature gets its own, and the harness applies per-model overrides
@@ -50,69 +42,45 @@
 # reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
 # AGENTS.md). This box is CUDA, but the rule keeps env files portable.

-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
-# FA_ALL_QUANTS=ON.
-LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
+# SGLang runtime: python3.12 venv, self-contained CUDA stack.
+SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang

-# Model package. The MTP sidecar file is required for speculative decoding.
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
-MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
-DRAFT_FILE=mtp-gemma-4-12B-it.gguf
+# Model package (8 shards + configs, sha256-verified by
+# scripts/download-frank-small.sh).
+MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound

 # Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
-# what clients ask for -- the harness entry in models.yml names it.
+# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
+# and what clients ask for -- the harness entry in models.yml names it.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771

-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
-# pool; --parallel divides it). See the SIZING note above for the fit
-# arithmetic; a fifth slot does not fit.
-CTX_SIZE=1048576
-PARALLEL=4
-CACHE_TYPE_K=q8_0
-CACHE_TYPE_V=q8_0
-# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
-# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
-# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
-# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
-# model, so the >12.8 GiB CACHE_RAM floor does not apply.
-CACHE_RAM=2048
+# Two slots at the model's native 262,144 window on fp8 KV. The pool is
+# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
+# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
+# so a compaction request (frank-primary's trigger 183,501 tokens + a
+# 32,768-token summary budget = 216,269) fits ONE request; the grid is
+# the two-way concurrency contract. A single request may still grow to
+# the whole pool (a 170k prompt was verified at the 4-slot shape).
+MEM_FRACTION=0.98
+MAX_TOTAL_TOKENS=606208
+CONTEXT_LENGTH=262144
+MAX_RUNNING=2
+KV_DTYPE=fp8_e4m3
+QUANTIZATION=auto-round
+# DeltaNet (mamba) recurrent states: the ssm state pool is the single
+# biggest hidden consumer on this card. The engine autofills the slot
+# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
+# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
+# bf16 dtype + 32 slots = 795 MiB.
+MAMBA_SSM_DTYPE=bfloat16
+# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
+# cache to keep shared-prefix states, small enough to leave the KV pool
+# room.
+MAX_MAMBA_CACHE_SIZE=32

-# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
-# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
-# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
-# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
-# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
-# tried. Idle VRAM is 23,300 MiB at this geometry.
-BATCH_SIZE=8192
-UBATCH_SIZE=1024
-
-# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
-# cover tokenization, sampling and the HTTP layer only. Deliberately modest
-# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
-NUM_THREADS=8
-THREAD_BATCH=8
-
-# Every layer on the 3090, with flash attention (required for q8 KV).
-NGL=999
-FLASH_ATTN=on
-
-# Speculative decoding: the MTP sidecar (draft context) chained with the
-# draftless ngram-mod matcher. Draft depth and busy-wait level are made
-# explicit here (both were the implicit build defaults: 3 and 50).
-# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
-# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
-# costs more attention re-read at long context than the extra accepted
-# token earns. Keep 3/50.
-SPEC_TYPE=draft-mtp,ngram-mod
-SPEC_DRAFT_N_MAX=3
-POLL_LEVEL=50
-
-# Greedy default (fleet contract); reasoning is extracted into
-# reasoning_content with an unlimited budget (the model decides).
-TEMPERATURE=0.0
-REASONING_FORMAT=deepseek
-REASONING_BUDGET=-1
+# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
+# grammar (both parsers per the SGLang Qwen3.5 cookbook).
+REASONING_PARSER=qwen3
+TOOLCALL_PARSER=qwen3_coder
@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
+Description=SGLang MiMo-V2.6-Distill-Qwen-9B-INT8 (qwen3_5 hybrid, in-checkpoint vision, no MTP) 4x151k fp8 KV (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target

@@ -8,8 +8,7 @@ Type=simple
 User=dank
 Group=dank
 # CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM. Parity with the halogen
-# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
+# though the weights themselves are copied into VRAM.
 LimitMEMLOCK=infinity
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
@@ -0,0 +1,86 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+#   Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+#   provides neither a compiled submodule nor breakable CUDA graphs.
+#   Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+#   16.07 s  34.8 %  708,500 launches  vectorized_elementwise_kernel      (22.7 us each)
+#    4.60 s  10.0 %    42,000           r4d_gemm_moe_w4a16_nt_b16_m64     (the MoE GEMM)
+#    3.11 s   6.7 %   174,708           elementwise_kernel_manual_unroll
+#    2.94 s   6.4 %   138,252           wvSplitK_hf_sml_                  (split-K GEMM)
+#    4.09 s   8.9 %    83,160           topk/router (sbtopk gatherTopK + warpMergeSortTopK
+#                                       + bitonicSortKVInPlace)
+#    3.01 s   6.5 %   133,920           r4d_mhc_* (hyper-connection pre/post/finish)
+#    2.78 s   6.0 %   127,916           inductor triton_poi_* / triton_per_* (the only
+#                                       fusions that DO happen)
+#    0.49 s   1.05%    20,664           r4d_moe_lru_gather_kernel         <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
+@@ -1,12 +1,14 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
+import os
+ from collections.abc import Iterable
+ from typing import ClassVar, Literal
+ 
+ import torch
+ from torch import nn
+ 
+from vllm.compilation.decorators import support_torch_compile
+ from vllm.config import ParallelConfig, VllmConfig
+ from vllm.distributed import (
+     get_ep_group,
+@@ -623,6 +625,18 @@
+         )
+ 
+ 
+_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
+
+
+@support_torch_compile(
+    dynamic_arg_dims={
+        "input_ids": 0,
+        "positions": -1,
+        "intermediate_tensors": 0,
+        "inputs_embeds": 0,
+    },
+    enable_if=lambda _cfg: _GLM_COMPILE,
+)
+ class Glm5NextModel(nn.Module):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
@@ -0,0 +1,119 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+#   Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+#   ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+#   cold lru_cache call       -> RAISE Unsupported
+#   WARM lru_cache call       -> RAISE Unsupported     <-- warming is not enough
+#   plain module-global read  -> compiles
+#   uncached os.path.exists   -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79  available()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81  has_w4a16_moe()
+#   model_executor/kernels/mhc/r4d_dsv4.py:16                    r4d_mhc_enabled()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:694                r4d_indexer_enabled(), mqa_max_heads()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:1124               r4d_indexer_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry()  moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py   (inside the pinned image)
+#   base    sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+#   patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
+++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ -361,3 +361,71 @@
+ @functools.lru_cache
+ def moe_w4a16_mt1():
+     return getattr(lib(), "r4d_gemm_moe_w4a16_nt_b16_m64_mt1", None)
+
+
+# FORK-LOCAL (patch: trace-safe capability probes).  torch.compile traces the
+# BODY of a functools.lru_cache call even when the cache is warm, so a capability
+# probe called from a forward pass is a dynamo graph break.  Measured 2026-09-30
+# on the GLM-5.3-Flash trunk under @support_torch_compile:
+#   Dynamo does not know how to trace the builtin `posix.stat`
+#   r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
+# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
+# (outside any tracing region), so a traced call is a dict read.  The probe set is
+# every no-argument capability probe / symbol lookup reachable from a forward
+# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
+# and the draft/lm-head guards.
+_PROBE_NAMES = (
+    "available",
+    "lib",
+    "has_w4a16_moe",
+    "moe_w4a16",
+    "moe_w4a16_mt1",
+    "has_mhc_dsv4",
+    "r4d_mhc_enabled",
+    "r4d_indexer_enabled",
+    "mqa_max_heads",
+    "r4d_act_quant_enabled",
+)
+
+
+def _freeze_probe(fn):
+    """Return fn bound to its first result.  Result must be a plain value."""
+
+    memo = []
+
+    def probe(*args, **kwargs):
+        if not memo:
+            memo.append(fn(*args, **kwargs))
+        return memo[0]
+
+    return probe
+
+
+def _freeze_probes() -> list[str]:
+    frozen = []
+    for name in _PROBE_NAMES:
+        fn = globals().get(name)
+        if fn is None:
+            continue
+        # Only the fast branch of a bound probe is traced; an empty memo means the
+        # first call still does I/O, so populate it here as well.
+        globals()[name] = _freeze_probe(fn)
+        frozen.append(name)
+    return frozen
+
+
+from vllm.logger import init_logger as _init_logger
+
+_probe_log = _init_logger(__name__)
+
+if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
+    _warmed, _failed = [], []
+    for _n in _freeze_probes():
+        try:
+            globals()[_n]()
+            _warmed.append(_n)
+        except Exception:  # noqa: BLE001 -- probe unavailable here; recompute lazily
+            _failed.append(_n)
+    _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
+                    len(_warmed), _warmed,
+                    f" (deferred: {_failed})" if _failed else "")
@@ -0,0 +1,116 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+#   RuntimeError: Graph breaks are not supported with aot compile.
+#   Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+#   models/glm5next/amd/sparse_indexer.py:263  int(positions[...].max().item()) + 1
+#   models/glm5next/amd/sparse_indexer.py:419  lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+#   fullgraph=True   suppress_errors=False  -> RAISE Unsupported
+#   fullgraph=True   suppress_errors=True   -> RAISE Unsupported   <-- suppress_errors
+#                                                                     does NOT cover breaks
+#   fullgraph=False  suppress_errors=False  -> ok   (traceable segments compiled by
+#                                                    inductor, the rest runs eagerly;
+#                                                    numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py   (inside the pinned image)
+#   base    sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+#   patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+#   (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
+# on the decode forward:
+#
+#   KeyError: 'in_proj_qkvbfg_a'
+#     model.py:748  Glm5NextModel.forward        hidden_states, residual, post, comb = layer(...)
+#     model.py:456  Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
+# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
+# not an arbitrary string.
+#
+# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
+# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
+# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
+# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
+# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
+# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
+# GLM decoder layer. Whether the root cause is the fork's module tree (something
+# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
+#
+# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
+#   * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
+#     KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
+#     (the reported line 456 is that method's `def` line, which is how inlined frames
+#     surface).
+#   * It is NOT raised from dynamo's four nn_modules lookups: the probe in
+#     /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
+#     the probe confirmed installed in the API-server and worker processes.
+#   * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
+#     layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
+#     (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
+#     quantization utils do compute exactly this bare key, but each is guarded
+#     (`if proj_name not in fused_mapping: return`), so none of them can raise it.
+#   * It reproduces on every run of this configuration (5 of 5).
+# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
+# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
+# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
+# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
+# probe. Until that names the caller this is a dynamo-side name-resolution failure.
+# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
+# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
+# dropping outer env vars.
+#
+# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
+# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
+# launcher cell (patch 04 removes the call) are real breaks, but with
+# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
+# fullgraph=True, which this trunk can never satisfy anyway.
+--- a/vllm/compilation/wrapper.py
+++ b/vllm/compilation/wrapper.py
+@@ -19,6 +19,15 @@
+ 
+ logger = init_logger(__name__)
+ 
+
+# FORK-LOCAL (patch: fullgraph opt-out).  torch.compile(fullgraph=True) turns any
+# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
+# (verified in-image, torch 2.11+rocm10.0).  With VLLM_COMPILE_FULLGRAPH=0 the
+# traceable segments are still compiled by inductor and the untraceable ones run
+# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
+# indexer) needs.  Default 1 keeps stock behaviour for every other model.
+_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
+
+ R = TypeVar("R")
+ P = ParamSpec("P")
+ 
+@@ -146,7 +155,7 @@
+         with aot_context:
+             self._compiled_callable = torch.compile(
+                 compiled_ptr,
+-                fullgraph=True,
+                fullgraph=_FULLGRAPH,
+                 dynamic=False,
+                 backend=backend,
+                 options=options,
@@ -0,0 +1,112 @@
+# glm53-compile-04-fused-qkv-rmsnorm-optout.patch — take GLM's fused-norm JIT kernel
+# out of the traced path.
+#
+# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
+# decorator) to matter; not part of the deployed shape.
+#
+# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
+# launcher bug made four consecutive boots misleading.  The scratch launcher
+# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
+# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
+# patch while `_FULLGRAPH` still defaulted to True.  With fullgraph=True *every* break
+# is an error, which is why the reported failure kept moving as breaks were removed:
+#
+#   fullgraph=False, AOT off  -> breaks tolerated; dies later on the dynamo name-map
+#                                `KeyError: 'in_proj_qkvbfg_a'`
+#   fullgraph=True,  AOT off  -> dies on whichever break comes first
+#                                (cell -> workspace -> compiler-disable)
+#
+# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
+# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
+# Launcher fixed; the env var is set again.
+#
+# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
+# break dynamo reported, and after this patch the report moved on to the workspace
+# acquisition — so it removes a real break.  It is not by itself what unblocks a boot:
+# with breaks genuinely tolerated the blocker is the name-map KeyError.
+#
+# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
+# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
+# object this one closes over is bound after use (see below).
+#
+# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30).  With 01+02+03 the
+# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
+# even with VLLM_COMPILE_FULLGRAPH=0, because dynamo cannot build a resume function
+# around it:
+#
+#   torch._dynamo.exc.Unsupported: Read uninitialized cell
+#   Explanation: Attempted to read a cell variable that has not been populated yet.
+#   Developer debug context: CellVariable()
+#
+# User stack:
+#   glm5next/common/model.py:748    Glm5NextModel.forward       layer(...)
+#   glm5next/common/model.py:526    Glm5NextDecoderLayer.forward
+#   glm5next/common/attention.py:590
+#   model_executor/layers/mla.py:208
+#     q_proj_input, kv_c_normed = fused_q_kv_rmsnorm(...)
+#   -> torch/_dynamo/polyfills/__init__.py:392 getattr_and_trace
+#
+# So dynamo is tracing the fork's JIT-kernel launcher framework
+# (VllmTritonJitKernel + @kernel_launcher — 39 call sites) and finds a closure name
+# that is not yet bound.  In models/common/ops/fused_qk_rmsnorm.py the kernel object
+# is bound AFTER the function that closes over it:
+#
+#   def fused_q_kv_rmsnorm(...):          # line 194
+#       ...
+#       _FUSED_Q_KV_RMSNORM_KERNEL(...)   # line 211
+#   _FUSED_Q_KV_RMSNORM_KERNEL = FusedQKVRMSNormKernel()    # line 223
+#
+# Two ways out, and this patch takes the cheap one:
+#   (a) WORKAROUND (here): stop calling the fused kernel from the traced region.  The
+#       flag is already a constructor argument (mla.py:80, default False) that glm5next
+#       turns ON at attention.py:583, so it becomes an env knob.
+#   (b) ROOT FIX (not done): make the launcher trace-safe for all 39 sites.  Any other
+#       @kernel_launcher-backed kernel still in the traced path can produce the same
+#       fatal break, so (b) is the durable fix and (a) is what unblocks iterating.
+#
+# COST of R4D_GLM_FUSE_QKV_RMSNORM=0: one extra small RMSNorm launch per MLA layer per
+# token (q_a_layernorm + kv_a_layernorm instead of one fused kernel) — ~2 launches out
+# of ~3,700 per token on this shape, against the 44 % of device time currently spent in
+# unfused pointwise work.  Correct trade for a compile experiment.
+#
+# TARGET: vllm/models/glm5next/common/attention.py   (inside the pinned image)
+#   base    sha256 362bbf0e6c50cdfa446c6551b9b1ddff425e08f88e3d4140fd8db88f8a619e0c
+#   patched sha256 84969993c250e9f8183a33a6e1ddd607272f48b950b560272eb33b7cdde31189
+#
+# APPLY: `patch -p1` from the vllm package root (verified: applies clean, reproduces
+# the patched sha256), or bind-mount new/.../attention.py over site-packages.
+# R4D_GLM_FUSE_QKV_RMSNORM=1 (default) restores the fused kernel.
+--- a/vllm/models/glm5next/common/attention.py
+++ b/vllm/models/glm5next/common/attention.py
+@@ -1,6 +1,7 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
+import os
+ import torch
+ import torch.nn.functional as F
+ from torch import nn
+@@ -36,6 +37,14 @@
+ from vllm.utils.deep_gemm import PAGED_MQA_PAGE_SIZES
+ from vllm.v1.kv_cache_interface import KpoolTailSpec, MLAAttentionSpec
+ 
+
+# FORK-LOCAL (patch: keep the traced region trace-safe).  R4D_GLM_FUSE_QKV_RMSNORM=0
+# replaces the fused `fused_q_kv_rmsnorm` JIT-kernel launch (one kernel) with two plain
+# RMSNorms, because dynamo cannot trace the launcher's closure ("Read uninitialized cell",
+# fatal even with fullgraph=False).  The fused kernel object in
+# models/common/ops/fused_qk_rmsnorm.py is bound AFTER the function that uses it.
+_FUSE_QKV_RMSNORM = os.environ.get("R4D_GLM_FUSE_QKV_RMSNORM", "1") == "1"
+
+ logger = init_logger(__name__)
+ 
+ # Shared torch.compile config for the indexer's small-kernel leaves. The MLA
+@@ -580,7 +589,7 @@
+             quant_config,
+             prefix,
+             skip_topk=False,
+-            fuse_qkv_rmsnorm=True,
+            fuse_qkv_rmsnorm=_FUSE_QKV_RMSNORM,
+         )
+ 
+     def forward(
@@ -0,0 +1,96 @@
+# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
+# acquisition must not be traced.
+#
+# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is a break worth
+# removing, but NOT part of the confirmed blocker and NOT required for the
+# break-tolerant route: `torch.compiler.disable` is itself a graph break, so this patch
+# helps only a hypothetical fullgraph=True route, where the workspace acquisition would
+# be better RESTRUCTURED out of the forward (allocate at init/profile time) than made
+# opaque. Kept because the discovery is real and the wrapper is a no-op when nothing is
+# compiling.
+#
+# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
+#
+#   torch._dynamo.exc.Unsupported: Unsupported method call
+#   user stack
+#     glm5next/common/model.py:748      Glm5NextModel.forward
+#     glm5next/common/model.py:526      Glm5NextDecoderLayer.forward
+#     glm5next/common/attention.py:599 / :401
+#     glm5next/amd/sparse_indexer.py:711  forward_native
+#     glm5next/amd/sparse_indexer.py:132  sparse_attn_indexer_kpool
+#     v1/worker/workspace.py:135          get_simultaneous
+#     v1/worker/workspace.py:155          _ensure_workspace_size
+#
+# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
+#
+#     if not isinstance(attn_metadata, dict):
+#         # Reserve workspace for indexer during profiling run
+#
+# so the engine dies exactly where it sizes its memory.  `_ensure_workspace_size`
+# reads ContextVar and ubatch state (`_workspace_lane.get()`,
+# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
+#
+# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
+# so dynamo stops analysing the call and executes it eagerly from inside the compiled
+# region.  Chosen over decorating the workspace manager's own methods because that
+# would change workspace behaviour for every model, not just this one.  The wrapper is
+# a no-op when nothing is compiling, so it needs no env gate and is safe in the
+# stock (uncompiled) path — which is why this patch can be reasoned about
+# independently of whether the changelist ever ships.
+#
+# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
+# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
+# can build a resume function around it is exactly what the next boot decides — the
+# difference between this failure and 04's `Read uninitialized cell`.
+#
+# TARGET: vllm/models/glm5next/amd/sparse_indexer.py   (inside the pinned image)
+#   base    sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
+#   patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
+--- a/vllm/models/glm5next/amd/sparse_indexer.py
+++ b/vllm/models/glm5next/amd/sparse_indexer.py
+@@ -35,6 +35,21 @@
+ from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
+ from vllm.v1.worker.workspace import current_workspace_manager
+ 
+@torch.compiler.disable
+def _acquire_workspace(*shapes_and_dtypes):
+    """Acquire shared workspace buffers, untraced.
+
+    Workspace acquisition reads ContextVar/ubatch state and may allocate
+    (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
+    ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
+    ``Unsupported: Unsupported method call`` and kills the engine during the profile
+    run.  ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
+    region inside the compiled graph.  No-op when not compiling.
+    """
+
+    return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
+
+
+ logger = init_logger(__name__)
+ 
+ # kpool write helper: form pools from the current token batch and compress them
+@@ -129,7 +144,7 @@
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        current_workspace_manager().get_simultaneous(
+        _acquire_workspace(
+             values_spec,
+             scales_spec,
+             ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
+@@ -275,11 +290,10 @@
+         # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
+         # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
+         # scales) based on use_fp4_cache.
+-        workspace_manager = current_workspace_manager()
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
+        k_quant_full, k_scale_full = _acquire_workspace(
+             values_spec,
+             scales_spec,
+         )
@@ -0,0 +1,173 @@
+# glm53-dflash-01-aux-hidden-states.patch — give the GLM trunk the EAGLE3 / DFlash2
+# auxiliary-hidden-state interface.
+#
+# STATUS: WIP, and INDEPENDENT of the compile changelist (01-05). It patches the same
+# file as glm53-compile-01-model-decorator.patch, so if both are ever applied, apply 01
+# first. Not part of the deployed shape.
+#
+# WHY.  `--speculative-config '{"method":"dflash","model":"/path/to/drafter", ...}'` with
+# incoai/GLM-5.3-Flash-DFlash2 is accepted -- the config parses, the drafter loads -- and
+# then the worker dies at init with:
+#
+#     RuntimeError: Model does not support EAGLE3 interface
+#
+# raised by v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20 through
+# `supports_eagle3(model)` == `isinstance(model, SupportsEagle3)` (a runtime-checkable
+# Protocol).  Nothing about the drafter, the method name or the payload is wrong:
+# `glm5next` implements neither half of the contract.
+#
+# THE CONTRACT (read out of this image, not assumed):
+#   * DEFINITIVE (v5, after four failed boots): the member set is checked against
+#     the INSTANCE the runner calls -- which for GLM-5.3-Flash is
+#     Glm5NextForConditionalGeneration, NOT Glm5NextForCausalLM.  The multimodal
+#     wrapper wraps the text model as self.language_model
+#     (init_vllm_registered_model, architectures=["Glm5NextForCausalLM"],
+#     glm5next/common/model.py:1113), and model_runner.py:425 calls
+#     set_eagle3_aux_hidden_state_layers(self.model) on the CONDITIONAL.
+#     Presence rule (verified in-image with stock typing, python 3.14):
+#     isinstance walks ALL __protocol_attrs__ -- members of SupportsEagle3 AND
+#     its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
+#     getattr_static()s each one off the instance, breaking on the first miss.
+#     So BOTH wrappers need all three attributes.
+#   * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
+#     "Cannot create a consistent method resolution order (MRO) for bases ..."
+#     (GLM's wrappers already carry four/five Protocol-derived bases).  The
+#     attribute route avoids the MRO entirely.  Also proven by simulation:
+#     the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
+#     position.
+#   * Method placement: without inheriting, the Protocol's default
+#     set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers
+#     do not exist on either wrapper, so both are defined explicitly --
+#     ForCausalLM unwraps to self.model (the EagleModelMixin trunk, v1's
+#     correct bodies), the conditional delegates to self.language_model
+#     (the pixtral.py:498 pattern).
+#   * interfaces.py:1673-1719 -- for reference, the inherited defaults unwrap
+#     `holder = getattr(parent_ref, "model", parent_ref)` and assert
+#     `isinstance(holder, EagleModelMixin)`, defaulting to (2, L // 2, L - 3).
+#   * eagle3_utils.py:26-31 -- layers come from the spec config when it carries them
+#     (this drafter ships target_layer_ids [5, 14, 24, 33, 42]), else from the default.
+#   * deepseek_v2.py:1551-1585 and v1/worker/gpu/cudagraph_utils.py:621 -- the trunk
+#     appends one tensor per aux layer while looping and returns the TUPLE
+#     (hidden_states, aux_hidden_states) only when that list is non-empty.
+#
+# WHAT IT ADDS for GLM: EagleModelMixin on Glm5NextModel; aux capture in the layer
+# loop, where the aux state is the single-stream `hidden_states` ENTERING that layer
+# (GLM's loop carries the already-summed single-stream state -- its non-mHC branch
+# re-splits `residual = hidden_states` inside the layer -- so this is the equivalent
+# of deepseek_v2's `hidden_states + residual`); the conditional tuple return; the
+# three SupportsEagle3 member attributes on BOTH wrappers; and the two interface
+# methods on both (ForCausalLM: unwrap to trunk; conditional: delegate to
+# language_model).
+#
+# HONEST SCOPE: this makes the interface PRESENT, which is exactly what the engine
+# checks -- it does not make drafts GOOD, and per the measured profile it is not a
+# decode-speed fix (a verification pass pays pointwise cost proportional to its token
+# count, which is why MTP4 measured flat).  It is the prerequisite for driving DFlash2
+# on this endpoint at all.
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 bc9913a15c8e7f88e0c36b07bf029b383cc8c2078b58d66dc91aa235f8af77f4
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount the patched file over
+# site-packages (which is what a scratch-launcher DFLASH_PATCH=1 does).
+--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
+@@ -95,6 +95,12 @@
+ logger = init_logger(__name__)
+ 
+ 
+from vllm.model_executor.models.interfaces import (
+    EagleModelMixin,
+    SupportsEagle3,
+)
+
+
+ class Glm5NextMLP(nn.Module):
+     def __init__(
+         self,
+@@ -623,7 +629,7 @@
+         )
+ 
+ 
+-class Glm5NextModel(nn.Module):
+class Glm5NextModel(nn.Module, EagleModelMixin):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+ 
+@@ -730,7 +736,14 @@
+         if self.is_sequence_parallel:
+             hidden_states = sp_shard(hidden_states)
+ 
+-        for layer in self._active_layers:
+        aux_hidden_states: list[torch.Tensor] = []
+        for local_idx, layer in enumerate(self._active_layers):
+            if (getattr(self, "start_layer", 0) + local_idx) in self.aux_hidden_state_layers:
+                # Auxiliary hidden state for EAGLE-3 / DFlash2 drafts: the
+                # single-stream state ENTERING this layer, before its input norm --
+                # the same semantic deepseek_v2 implements by appending
+                # hidden_states + residual at this point in its loop.
+                aux_hidden_states.append(hidden_states)
+             hidden_states, residual, post, comb = layer(
+                 positions, hidden_states, residual, post, comb
+             )
+@@ -749,6 +762,8 @@
+             hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
+ 
+         hidden_states = self.norm(hidden_states)
+        if len(aux_hidden_states) > 0:
+            return hidden_states, aux_hidden_states
+         return hidden_states
+ 
+     def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+@@ -937,6 +952,10 @@
+ class Glm5NextForCausalLM(
+     nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
+ ):
+    supports_eagle3: ClassVar[Literal[True]] = True
+    has_own_lm_head: bool = False
+    has_own_embed_tokens: bool = False
+
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+         self.model_config = vllm_config.model_config
+@@ -963,6 +982,14 @@
+     def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+         return self.model.embed_input_ids(input_ids)
+ 
+    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+        # EAGLE-3 / DFlash2 interface: unwrap to the trunk (the EagleModelMixin).
+        self.model.aux_hidden_state_layers = layers
+
+    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+        num_layers = len(self.model.layers)
+        return (2, num_layers // 2, num_layers - 3)
+
+     def forward(
+         self,
+         input_ids: torch.Tensor | None,
+@@ -1033,6 +1060,10 @@
+ class Glm5NextForConditionalGeneration(
+     Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts
+ ):
+    supports_eagle3: ClassVar[Literal[True]] = True
+    has_own_lm_head: bool = False
+    has_own_embed_tokens: bool = False
+
+     # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
+     # multimodal wrapper must declare the same interfaces so vLLM treats it as
+     # hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
+@@ -1076,6 +1107,13 @@
+ 
+         return Glm5NextForCausalLM.get_mamba_state_copy_func()
+ 
+    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+        # EAGLE-3 / DFlash2: delegate to the language model (pixtral.py:498).
+        self.language_model.set_aux_hidden_state_layers(layers)
+
+    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+        return self.language_model.get_eagle3_default_aux_hidden_state_layers()
+
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super(Glm4vForConditionalGeneration, self).__init__()
+         config = vllm_config.model_config.hf_config
@@ -0,0 +1,317 @@
+--- a/vllm/v1/core/kv_cache_utils.py
+++ b/vllm/v1/core/kv_cache_utils.py
+@@ -1210,17 +1210,38 @@
+         for name, spec in kv_cache_spec.items()
+         if isinstance(spec, KpoolTailSpec)
+     }
+    hidden_specs = {
+        name: spec
+        for name, spec in kv_cache_spec.items()
+        if isinstance(spec, HiddenStateCacheSpec)
+    }
+     attn_specs = {
+         name: spec
+         for name, spec in kv_cache_spec.items()
+-        if not isinstance(spec, (MambaSpec, KpoolTailSpec))
+        if not isinstance(
+            spec, (MambaSpec, KpoolTailSpec, HiddenStateCacheSpec)
+        )
+     }
+-    if not mamba_specs or not all(
+-        type(spec) is MLAAttentionSpec for spec in attn_specs.values()
+-    ):
+    if not mamba_specs or not attn_specs:
+        return None
+    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash
+    # drafter, attn_specs also carries the drafter's attention layers. The
+    # GLM MLA layers keep their own group (no page unification -- per-group
+    # pages are the glm5 design); the drafter layers must form one uniform
+    # group of their own, and the hidden-state layers get per-layer groups
+    # like the generic tail.
+    mla_specs = {
+        name: spec
+        for name, spec in attn_specs.items()
+        if type(spec) is MLAAttentionSpec
+    }
+    draft_specs = {
+        name: spec for name, spec in attn_specs.items() if name not in mla_specs
+    }
+    if not mla_specs:
+        return None
+    if draft_specs and UniformTypeKVCacheSpecs.from_specs(draft_specs) is None:
+         return None
+-
+-    mla_specs = cast(dict[str, MLAAttentionSpec], attn_specs)
+     idx_pages = {
+         spec.page_size_bytes for spec in mla_specs.values() if spec.tokens_per_state > 1
+     }
+@@ -1233,7 +1254,7 @@
+     mla_pages = {mla_specs[name].page_size_bytes for name in mla_names}
+     assert len(mla_pages) == 1
+     mla_page = mla_pages.pop()
+-    uniform_spec = UniformTypeKVCacheSpecs.from_specs(attn_specs)
+    uniform_spec = UniformTypeKVCacheSpecs.from_specs(mla_specs)
+     assert uniform_spec is not None
+ 
+     tail_group: KVCacheGroupSpec | None = None
+@@ -1270,10 +1291,20 @@
+     for index, name in enumerate(mamba_specs):
+         mamba_grouped_names[index % num_groups].append(name)
+ 
+    draft_group: KVCacheGroupSpec | None = None
+    if draft_specs:
+        draft_uniform = UniformTypeKVCacheSpecs.from_specs(draft_specs)
+        assert draft_uniform is not None
+        draft_group = KVCacheGroupSpec(list(draft_specs), draft_uniform)
+    hidden_groups = [
+        KVCacheGroupSpec([name], spec) for name, spec in hidden_specs.items()
+    ]
+     return (
+-        [KVCacheGroupSpec(list(attn_specs), uniform_spec)]
+        [KVCacheGroupSpec(list(mla_specs), uniform_spec)]
+         + ([tail_group] if tail_group is not None else [])
+        + ([draft_group] if draft_group is not None else [])
+         + create_kv_cache_group_specs(padded_specs, mamba_grouped_names)
+        + hidden_groups
+     )
+ 
+ 
+@@ -1289,10 +1320,17 @@
+         int,
+         list[str],
+         int,
+        KVCacheGroupSpec | None,
+        list[KVCacheGroupSpec],
+     ]
+     | None
+ ):
+-    """Recognize the GLM-5.3-Flash grouping after optional PP projection."""
+    """Recognize the GLM-5.3-Flash grouping after optional PP projection.
+
+    FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): also recognizes the
+    drafter's uniform group and the per-layer hidden-state groups, returned
+    as ``draft_group`` / ``hidden_groups`` so the pool builder and the memory
+    estimator can size their tensors beside the aliased GLM pool."""
+     uniform_groups = [
+         group
+         for group in kv_cache_groups
+@@ -1301,17 +1339,28 @@
+     mamba_groups = [
+         group for group in kv_cache_groups if isinstance(group.kv_cache_spec, MambaSpec)
+     ]
+    hidden_groups = [
+        group
+        for group in kv_cache_groups
+        if isinstance(group.kv_cache_spec, HiddenStateCacheSpec)
+    ]
+     attn_group: KVCacheGroupSpec | None = None
+     tail_group: KVCacheGroupSpec | None = None
+    draft_group: KVCacheGroupSpec | None = None
+     for group in uniform_groups:
+         inner = cast(UniformTypeKVCacheSpecs, group.kv_cache_spec).kv_cache_specs
+         if all(type(spec) is MLAAttentionSpec for spec in inner.values()):
+             attn_group = group
+         elif all(isinstance(spec, KpoolTailSpec) for spec in inner.values()):
+             tail_group = group
+        else:
+            draft_group = group
+     if attn_group is None or not mamba_groups:
+         return None
+-    if len(uniform_groups) + len(mamba_groups) != len(kv_cache_groups):
+    if (
+        len(uniform_groups) + len(mamba_groups) + len(hidden_groups)
+        != len(kv_cache_groups)
+    ):
+         return None
+ 
+     attn_uniform = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+@@ -1362,6 +1411,8 @@
+         idx_page,
+         tail_names,
+         tail_page,
+        draft_group,
+        hidden_groups,
+     )
+ 
+ 
+@@ -1395,10 +1446,34 @@
+         return kv_cache_spec
+ 
+     max_page_size = max(page_sizes)
+    # GLM-5.3-Flash + a draft model: the sparse indexer k_cache page is
+    # tokens_per_state x the MLA page (same per-token bytes), and MLA layers
+    # cannot be PADDED to an arbitrary page size (sparse MLA indexes the cache
+    # in whole token rows) -- but they CAN grow their block size when the max
+    # page is an exact multiple of the MLA page. Bump the max page to a
+    # multiple of every MLA-family page so all of them take the divisible
+    # block-growth branch below instead of the padding refusal, while the
+    # draft and mamba layers simply pad to the (slightly larger) page.
+    mla_pages = {
+        spec.page_size_bytes
+        for spec in kv_cache_spec.values()
+        if isinstance(spec, MLAAttentionSpec)
+    }
+    if mla_pages:
+        mla_common = math.lcm(*mla_pages)
+        max_page_size = -(-max_page_size // mla_common) * mla_common
+     new_kv_cache_spec = {}
+     for layer_name, layer_spec in kv_cache_spec.items():
+         if layer_spec.page_size_bytes == max_page_size:
+             new_kv_cache_spec[layer_name] = layer_spec
+        elif isinstance(layer_spec, KpoolTailSpec):
+            # The kpool tail kernel pins the tail block to the kpool size
+            # (tail_kv_cache.view(...) must be [blocks, 2, index_kpool,
+            # head_dim]); growing block_size breaks that contract. Pad the
+            # physical page instead -- the tail is its own group and the pad
+            # wastes bytes only.
+            new_spec = replace(layer_spec, page_size_padded=max_page_size)
+            new_kv_cache_spec[layer_name] = new_spec
+         elif isinstance(layer_spec, MambaSpec):
+             # MambaSpec's page size is determined by its state shapes and does
+             # not scale with block_size, so pad the page instead. This is the
+@@ -1593,8 +1668,31 @@
+ ) -> int:
+     """Return the largest cache group's bytes per block."""
+     if (glm5_layout := _glm5_next_tensor_layout(kv_cache_groups)) is not None:
+-        _, _, mla_names, idx_names, mla_page, idx_page, _, _ = glm5_layout
+-        return len(mla_names) * mla_page + len(idx_names) * idx_page
+        (
+            _,
+            _,
+            mla_names,
+            idx_names,
+            mla_page,
+            idx_page,
+            _,
+            _,
+            draft_group,
+            hidden_groups,
+        ) = glm5_layout
+        bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            bytes_per_block += sum(
+                spec.page_size_bytes for spec in d_inner.values()
+            )
+        for group in hidden_groups:
+            bytes_per_block += (
+                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+            )
+        return bytes_per_block
+ 
+     bytes_per_block = max(
+         sum(
+@@ -1701,12 +1799,30 @@
+             idx_page,
+             tail_names,
+             _,
+            draft_group,
+            hidden_groups,
+         ) = glm5_layout
+         bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+        extra_bytes_per_block = 0
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            extra_bytes_per_block += sum(
+                spec.page_size_bytes for spec in d_inner.values()
+            )
+        for group in hidden_groups:
+            extra_bytes_per_block += (
+                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+            )
+         num_blocks = may_override_num_blocks(
+-            vllm_config, available_memory // bytes_per_block
+            vllm_config,
+            available_memory // (bytes_per_block + extra_bytes_per_block),
+         )
+-        size = bytes_per_block * num_blocks
+        # One backing allocation shared by every tensor (allocate_kv_cache
+        # asserts all tensor sizes equal): the aliased GLM pool plus the
+        # draft/hidden regions appended after it.
+        size = (bytes_per_block + extra_bytes_per_block) * num_blocks
+         attn_specs = cast(
+             UniformTypeKVCacheSpecs, attn_group.kv_cache_spec
+         ).kv_cache_specs
+@@ -1745,6 +1861,37 @@
+                 ).kv_cache_specs
+                 add_tensor(tail_name, tail_specs[tail_name], offset)
+ 
+        # Draft + hidden-state layers get their own tensors appended after
+        # the aliased pool (no aliasing: independent block tables).
+        offset_cursor = bytes_per_block * num_blocks
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            d_page = next(iter(d_inner.values())).page_size_bytes
+            kv_cache_tensors.append(
+                KVCacheTensor(
+                    size=size,
+                    layers=list(draft_group.layer_names),
+                    layer_stride=d_page * num_blocks,
+                    block_stride=d_page,
+                    offset=offset_cursor,
+                )
+            )
+            offset_cursor += d_page * num_blocks * len(draft_group.layer_names)
+        for group in hidden_groups:
+            h_page = group.kv_cache_spec.page_size_bytes
+            kv_cache_tensors.append(
+                KVCacheTensor(
+                    size=size,
+                    layers=list(group.layer_names),
+                    layer_stride=h_page * num_blocks,
+                    block_stride=h_page,
+                    offset=offset_cursor,
+                )
+            )
+            offset_cursor += h_page * num_blocks * len(group.layer_names)
+
+         return KVCacheConfig(
+             num_blocks=num_blocks,
+             kv_cache_tensors=kv_cache_tensors,
+@@ -2440,6 +2587,8 @@
+             idx_page,
+             tail_names,
+             _,
+            draft_group,
+            hidden_groups,
+         ) = glm5_layout
+         uniform_spec = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+         total_blocks = uniform_spec.max_memory_usage_pages(vllm_config)
+@@ -2452,7 +2601,30 @@
+         )
+         if tail_names:
+             total_blocks += 1
+-        return total_blocks * (len(mla_names) * mla_page + len(idx_names) * idx_page)
+        pool_bytes = total_blocks * (
+            len(mla_names) * mla_page + len(idx_names) * idx_page
+        )
+        extra_bytes = 0
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            d_spec = next(iter(d_inner.values()))
+            d_blocks = cdiv(
+                d_spec.max_memory_usage_bytes(vllm_config), d_spec.page_size_bytes
+            )
+            extra_bytes += d_blocks * d_spec.page_size_bytes * len(
+                draft_group.layer_names
+            )
+        for group in hidden_groups:
+            h_spec = group.kv_cache_spec
+            h_blocks = cdiv(
+                h_spec.max_memory_usage_bytes(vllm_config), h_spec.page_size_bytes
+            )
+            extra_bytes += (
+                h_blocks * h_spec.page_size_bytes * len(group.layer_names)
+            )
+        return pool_bytes + extra_bytes
+ 
+     bytes_per_block = _pool_bytes_per_block(kv_cache_groups)
+     total_blocks = 0
@@ -0,0 +1,31 @@
+--- a/vllm/model_executor/offloader/expert_activation_registry.py
+++ b/vllm/model_executor/offloader/expert_activation_registry.py
+@@ -316,7 +316,27 @@
+ 
+ 
+ def _glm53_kv_bytes_per_token(cfg: "VllmConfig") -> int:
+-    return 13600
+    per_tok = 13600
+    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash/dspark
+    # drafter, the KV spec set gains the drafter's attention layers and the aux
+    # hidden-state cache (one bf16 hidden vector per target aux layer per
+    # token). Both are full-length per-token costs the aliased glm5 pool
+    # cannot absorb, so the reserve must include them. The drafter is a 5-layer
+    # qwen3 (8 kv heads x 128 x K+V x bf16 = 4096 B/token); the aux count is
+    # the dflash target_layer_ids count (5).
+    # NOTE: use_aux_hidden_state_outputs lives on the gpu_model_runner
+    # instance (gpu_model_runner.py:623/678), not on the spec config -- it is
+    # not set yet when the expert plan runs. The spec-config METHOD is the
+    # plan-time signal for the same fact.
+    sc = getattr(cfg, "speculative_config", None)
+    if sc is not None and getattr(sc, "method", "") in (
+        "dflash",
+        "dspark",
+        "extract_hidden_states",
+    ):
+        hidden = int(cfg.model_config.hf_config.hidden_size)
+        per_tok += 5 * hidden * 2 + 4096
+    return per_tok
+ 
+ 
+ for _arch in ("Glm5NextForConditionalGeneration", "Glm5NextForCausalLM"):
@@ -0,0 +1,65 @@
+# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
+#
+# STATUS: NOT part of the deployed frank-primary shape. It exists because the
+# fork's plan under-models this model's KV, and it is the enabler for a future
+# MTP attempt (the production GLM shape runs without MTP and boots fine on the
+# plan's own reserve).
+#
+# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
+# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
+# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
+# cache at that reserve, so the plan's own arithmetic can make a legal
+# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
+#   - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB  -> boots
+#   - MTP4,  max_model_len 131072: reserve 1.74 GiB <  need 2.07 GiB  -> refuses
+#   - MTP4,  max_model_len 262144: reserve 3.43 GiB <  need 3.62 GiB  -> refuses
+#     ("To serve at least one request with the model's max seq len ...")
+# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
+# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
+# where the KV reserve would eat a third of the expert arena.
+#
+# THE FIX: let an explicit operator KV request raise the cap. The compile and
+# graph-capture headroom still comes off the top, so the blast radius is one
+# flag the operator chose.
+#
+# APPLY (into a host dir that is then bind-mounted read-only over the image):
+#   IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+#   D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
+#   podman run --rm --entrypoint cat $IMAGE \
+#     /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
+#     > $D/vllm/v1/worker/gpu_worker.py
+#   ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
+#   # then add to the podman run:
+#   #   -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
+#   #   -e CLAV_KV_RESERVE_FLOOR_GIB=<needed>   (or pass --kv-cache-memory-bytes)
+#
+# Base file sha256 (the pristine file out of the pinned image):
+#   1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
+# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
+# these bytes exactly):
+#   80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
+#
+--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
+@@ -824,6 +824,21 @@
+         expert_cfg = self.vllm_config.offload_config.expert
+         if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
+             cap = int(expert_cfg.planned_kv_reserve_bytes)
+            # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
+            # 13600 B/token + a small mamba term, but this model also carries a
+            # per-sequence KDA state and (under MTP) the draft's KV, so the
+            # reserve can sit BELOW what max_model_len legitimately needs -- and
+            # the cap then makes a legal config unbootable (measured: need
+            # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
+            # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
+            # is the operator's word: let it raise the cap. The compile/capture
+            # headroom still comes off the top.
+            _explicit = self.cache_config.kv_cache_memory_bytes
+            _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
+            if _explicit:
+                cap = max(cap, int(_explicit))
+            if _floor:
+                cap = max(cap, int(float(_floor) * (1 << 30)))
+             if self.available_kv_cache_memory_bytes > cap:
+                 logger.info_once(
+                     "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
@@ -18,7 +18,7 @@
 # is not an OSI licence.
 #
 # Shape: one session, full 262144 context (the model's native window). The
-# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
+# 27B dense weights (16.35 GiB at UD-Q4_K_XL) + the DFlash2 drafter (1.07) + KV +
 # compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
 # context measured 66.5 GiB of the 116.7 GiB device.
 #
@@ -78,7 +78,7 @@ GUFO_BIN="$RUNTIME_DIR/bin/gufo"
 # and keeps its own long filename, while the DFlash2 drafter is unchanged and
 # still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
 # the deployed payload, so the env file only has to state them when it moves.
-: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
+: "${MODEL_FILE:=$MODEL_DIR/Qwen3.8-27B-UD-Q4_K_XL.gguf}"
 : "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
 : "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"

@@ -11,12 +11,14 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
+# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
+# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
+# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
+# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
+# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
+# expert prefetch OFF (36% precision measured), TunableOp off.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
@@ -45,12 +47,16 @@ set +a
 : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
 : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
 : "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
-: "${FRANK_PRIMARY_SPEC:?FRANK_PRIMARY_SPEC must be set (0 disables)}"
-: "${FRANK_PRIMARY_UTIL:?FRANK_PRIMARY_UTIL must be set}"
-
-# --- gates: never serve a half-finished payload or a missing image -----------
-if ! podman images --format '{{.RepoDigests}}' | grep -qF "${FRANK_PRIMARY_IMAGE#*@}"; then
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
+: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
+: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
+: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
+# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
+# classic `podman images | grep -q` pattern races — grep exits on its first
+# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
+# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
    exit 1
@@ -99,17 +105,24 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
+  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
  "$FRANK_PRIMARY_IMAGE" /models \
  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
  --tensor-parallel-size 2 \
+  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
+  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
  --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
+  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
+  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
  --trust-remote-code \
  > "$LOG" 2>&1

@@ -1,37 +1,28 @@
 #!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
+# Serve letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound as
+# frank-small on Dankclaw's RTX 3090 via SGLang.
 #
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
-# belongs to the halogen engine that serves frank-large.
+# Topology: the whole checkpoint (12.0 GiB, INT8 W8A16 AutoRound) lives in
+# VRAM; the 3090's 24 GiB holds weights + a ~11.8 GiB fp8 KV pool at
+# --mem-fraction-static 0.98. The host CPU only tokenizes.
 #
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
-# passed via --model-draft with --spec-type draft-mtp: the draft context
-# loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so this chain is the
-# only shape that carries both.
+# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
+# DeltaNet, native 262144 window, and a vision tower (qwen3_5_vision)
+# baked into the safetensors: no separate projector file, the image path
+# is in the checkpoint. The config declares an MTP layer, but the shipped
+# weights carry none (no mtp.* tensors in the index), so this service runs
+# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
+# 4.24 GiB ghost model drafting at 0% acceptance.
 #
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
-# maximises the TOTAL pool instead, which would divide a smaller window
-# across slots. The 4-slot layout is measured to fit the card (see the
-# SIZING note in frank-small.env); more slots would fail the KV allocation.
+# Context: CONTEXT_LENGTH x MAX_RUNNING slots must fit the fp8 KV pool,
+# which the engine autofits around the weights AND the DeltaNet ssm state
+# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
+# pinned (bf16, 32 slots). With the grid at its measured fit the card has
+# no headroom for prefill CUDA-graph capture, so prefill runs eager
+# (--disable-prefill-cuda-graph); decode graphs are untouched.
 #
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
-# an audio encoder — input_audio content parts transcribe verbatim
-# (verified 2026-09-17). The fleet's image path lives here, which is why
-# halogen no longer sets HALOGEN_VISION_TOWER.
-#
-# Sampling: greedy by default (the fleet contract). Clients that send their own
-# temperature get their own values, and the harness overrides per model from
-# models.yml. --jinja is pinned even though this build defaults it on: the
-# model's own template supplies reasoning_effort, preserve_thinking and the
-# tool-call grammar, and a future build that flips that default must not
-# silently take tools away.
+# Sampling: greedy by default (the fleet contract); clients that send their
+# own temperature get their own values.
 #
 # Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
 set -euo pipefail
@@ -46,101 +37,61 @@ if [[ -f "$ENV_FILE" ]]; then
 fi

 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
-: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
-: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
-# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
-# payload, which frank-small served on 2026-09-16/17 and which no longer
-# exists on disk. frank-small.env has always overridden it, so the drift was
-# invisible — but a missing or unreadable env file would have left this
-# script pointing at a payload that is not there, turning a config problem
-# into a "missing asset" that names the wrong model.
-: "${DRAFT_FILE:=}"
+: "${SGLANG_BIN:=/home/dank/GitPackages/sglang-cu128/bin/sglang}"
+: "${MODEL_DIR:=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-: "${CTX_SIZE:=}"
-: "${PARALLEL:=1}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${CACHE_RAM:=8192}"
-: "${BATCH_SIZE:=2048}"
-: "${UBATCH_SIZE:=512}"
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
-# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
-# busy-wait level (build default 50).
-: "${SPEC_DRAFT_N_MAX:=3}"
-: "${POLL_LEVEL:=50}"
-: "${NUM_THREADS:=8}"
-: "${THREAD_BATCH:=8}"
-: "${NGL:=999}"
-: "${FLASH_ATTN:=on}"
-: "${SPEC_TYPE:=draft-mtp}"
-: "${TEMPERATURE:=0.0}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${REASONING_BUDGET:=-1}"
+: "${MEM_FRACTION:=0.98}"
+: "${CONTEXT_LENGTH:=151552}"
+: "${MAX_TOTAL_TOKENS:=606208}"
+: "${MAX_RUNNING:=4}"
+: "${KV_DTYPE:=fp8_e4m3}"
+: "${QUANTIZATION:=auto-round}"
+: "${REASONING_PARSER:=qwen3}"
+: "${TOOLCALL_PARSER:=qwen3_coder}"
+: "${MAMBA_SSM_DTYPE:=bfloat16}"
+: "${MAX_MAMBA_CACHE_SIZE:=32}"

 # Fail loudly rather than half-loading a model into VRAM: a missing file here
 # is a download that has not finished, and the unit's restart policy would
 # otherwise turn that into a crash loop that reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
+for path in "$SGLANG_BIN" "$MODEL_DIR/model.safetensors.index.json" \
+    "$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin/nvcc"; do
    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
+shard_count=$(find "$MODEL_DIR" -maxdepth 1 -name 'model-*.safetensors' | wc -l)
+if [[ "$shard_count" -ne 8 ]]; then
+    echo "frank-small: expected 8 safetensors shards in $MODEL_DIR, found $shard_count" >&2
+    exit 1
 fi

 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true

-MMPROJ_ARGS=()
-if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
-    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
-fi
+# FlashInfer JITs its attention kernels with nvcc at boot. The host's
+# /usr/bin/nvcc is CUDA 12.4 and dies on `--compress-mode` (a 12.8+
+# option), so the venv carries a matching CUDA 13.0 nvcc (extracted from
+# the cuda-nvcc-13-0 deb); it must shadow the system one on PATH.
+export PATH="$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin:$PATH"

-SPEC_ARGS=()
-if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
-    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
-fi
-
-DRAFT_ARGS=()
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
-fi
-
-# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
-# whatever the weights, projector, draft context and compute buffers leave
-# on the card; the engine prints the chosen value at startup.
-if [[ -n "${CTX_SIZE:-}" ]]; then
-    CTX_ARGS=(--ctx-size "$CTX_SIZE")
-else
-    CTX_ARGS=(--fit on)
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL_DIR/$MODEL_FILE" \
-    "${MMPROJ_ARGS[@]}" \
-    "${DRAFT_ARGS[@]}" \
-    --alias "$ALIAS" \
+# Sampling: SGLang has no server-wide sampling defaults; the harness sends
+# its per-model recipe from models.yml (compat.extraBody), and other clients
+# get the engine's own defaults.
+exec "$SGLANG_BIN" serve \
+    --model-path "$MODEL_DIR" \
+    --served-model-name "$ALIAS" \
    --host "$BIND_ADDR" \
    --port "$LISTEN_PORT" \
-    --gpu-layers "$NGL" \
-    --flash-attn "$FLASH_ATTN" \
-    "${CTX_ARGS[@]}" \
-    --parallel "$PARALLEL" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --poll "$POLL_LEVEL" \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --cache-ram "$CACHE_RAM" \
-    "${SPEC_ARGS[@]}" \
-    --jinja \
-    --metrics \
-    --temp "$TEMPERATURE" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget "$REASONING_BUDGET"
+    --mem-fraction-static "$MEM_FRACTION" \
+    --context-length "$CONTEXT_LENGTH" \
+    --max-running-requests "$MAX_RUNNING" \
+    --kv-cache-dtype "$KV_DTYPE" \
+    --quantization "$QUANTIZATION" \
+    --reasoning-parser "$REASONING_PARSER" \
+    --tool-call-parser "$TOOLCALL_PARSER" \
+    --mamba-ssm-dtype "$MAMBA_SSM_DTYPE" \
+    --max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE" \
+    --disable-prefill-cuda-graph \
+    --max-total-tokens "$MAX_TOTAL_TOKENS"
@@ -244,13 +244,148 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
 are `podman cp`'d into the container after every recreate (the serve script
 re-copies them; a recreate wipes container-local files).

+## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
+
+Two same-day cutovers moved this page's findings into production:
+
+1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
+   window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
+   `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
+   production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
+   measured 37.4-40.6 tok/s at ~54% acceptance.
+2. **The overnight redo campaign re-measured EVERY lever at matched
+   geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
+   and concluded the day: the incumbent cutover to **bf16 KV, no
+   speculative decode** (Frank's call).
+
+### What the redo campaign measured (all at shallow depth, 166/1087/4064)
+
+| lever | verdict |
+|---|---|
+| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
+| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
+| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
+| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
+| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
+| UVA | 2048 POISON (-39%); 8192 ~parity |
+| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
+| six slots | every SEQS-6 KV sizing refused at engine init |
+
+Depth caveat: every number above is at a near-empty KV. Deep-context
+behavior (the gather win growing with depth, bf16's decode cost at depth)
+is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
+
+### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
+
+```
+FRANK_PRIMARY_MAXLEN=262144    x SEQS 2 (524,288 tokens of contract)
+FRANK_PRIMARY_KV_BYTES=7300000000   PER RANK (bf16 = auto; 13,839 B/token)
+FRANK_PRIMARY_KV_DTYPE=auto   --kv-cache-dtype auto (bf16, unquantized)
+FRANK_PRIMARY_PRECISE_MEMORY=1  --expert-precise-memory (fitted overhead
+                                5.25 GiB instead of the 7.60 GiB default,
+                                so ~1,025 more arena slots)
+FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
+R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
+NO --speculative-config, NO --expert-prefetch
+```
+
+Boot-probed 2026-10-01 (transient systemd units at the production uid +
+memlock, full-log capture; the rootless bench probe had masked the production
+memory environment). The night searched two concurrency contracts:
+
+| rung | result |
+|---|---|
+| 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
+| 3 × 245760, bf16 | **OOM identically** |
+| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
+| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
+| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
+| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
+| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
+| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
+| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
+
+Boot is NOT proof of runtime survivability: the five-slot shape passed every
+boot gate and still died on the first long-generation request. After any
+knife-edge shape change, send a request with a large `max_tokens` before
+calling it done. The deployed shape is **four contexts of the model's full
+native 262,144 window**. fp8 KV is what buys the capacity —
+7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
+decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
+graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
+needs 12.47 GB even at fp8.
+
+The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
+reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
+reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
+which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
+read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
+only available while the container lives).
+
+Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
+`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
+
+## Parallelism options on this box (2026-10-01 survey)
+
+Surveyed from the running image's own CLI surface. Verdicts are specific to
+dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
+with only 2 KV heads.
+
+| option | flag | verdict on this box |
+|---|---|---|
+| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
+| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
+| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
+| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
+| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
+| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
+
+### DCP is structurally unavailable here — do not re-test on 2 GPUs
+`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
+
+```
+Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
+greater than the model's total number of KV heads (2). Increase
+`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
+```
+
+DCP shards the KV along the *sequence* dimension after KV heads have been spread
+across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
+heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
+fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
+`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
+(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
+each rank would hold half the sequence's KV — but it is unreachable without two
+more cards.
+
+### Measuring decode: the harness lives INSIDE the container
+`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
+run in the engine's namespace and the files must be copied in first:
+
+```
+podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
+podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
+podman exec frank-primary-qwen python3 /mtp_measure.py
+```
+
+Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
+No such file or directory` — measured 2026-10-01, which silently cost a whole
+arm's numbers. A window script must do the copy, not assume the path exists.
+
+### The deployed shape, measured again (2026-10-01 A/B window)
+A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
+2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
+concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
+deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
+reproducible across boots.
+
 ## See Also

 - [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
  (which frontier MoE fits this box; the same PCIe-expert-stream wall at
  V4.1 scale)
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  (production is the R9V fork; this page's container is the ad hoc detour)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
+  fork era; production has been the tcclaviger fork since)
 - [[x4-pair-experiment]] — the same residency law measured on a narrower
  link: streaming over PCIe loses, computing in RAM does not care
 - [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
@@ -0,0 +1,706 @@
+---
+type: runbook
+title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - glm5.3
+  - w4a16
+  - int4
+  - expert-offload
+  - expert-cache
+  - autoround
+  - vllm
+timestamp: 2026-09-30
+---
+
+# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+
+> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
+> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
+> **failure**: after five patches the drafter served and drafted (12.8 tok/s
+> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
+> **acceptance pinned at 1.00-1.38 on every text class including verbatim
+> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
+> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
+> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
+> untested alternative capture semantics: post-layer output state
+> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
+> (170 GiB) + drafter (2.2 GiB) staged to
+> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
+> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
+> generic unifier/tailpin/grouping fixes, apply-verified against the image.
+
+
+> **Question this page answers.** Can the fleet serve the 320B-class
+> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
+> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
+> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
+
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
+[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
+[[VLLM Index]], [[LLM/index|LLM fleet]].
+
+## The deployed shape
+
+| term | value |
+| --- | --- |
+| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
+| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
+| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
+| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
+| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
+| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
+| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
+
+```mermaid
+graph LR
+    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
+    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
+    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
+    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
+    D --> F[73.64 GiB pinned host rows / rank]
+    E --> G[decode step]
+    F -->|host misses<br/>1.05% of device time| G
+```
+
+## Why this payload and not another
+
+The W4A16 fast path on this fork is reachable **only** through the INC
+(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
+for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
+their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
+offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
+cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
+
+The chain that makes Intel's checkpoint work, with the exact evidence:
+
+```
+[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
+[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
+[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
+[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
+[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
+                           host 73.64 GiB pinned, host rows prepacked on the GPU
+```
+
+`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
+format-agnostic but requires the experts class to expose the hot/cold split.
+
+## The numbers, and the wall
+
+Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
+
+| depth | prefill | decode |
+| --- | --- | --- |
+| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
+| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
+| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
+
+Decode is flat across context and across every arena change — the signature of a
+per-step cost that is not KV- or residency-driven at the margin.
+
+### The measured profile, and the real wall
+
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
+
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
+| --- | --- | --- | --- |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
+
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
+
+Two consequences that kill three whole families of planned work:
+
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+  accident.** Pointwise cost scales with the token count, so a draft batch that
+  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+  the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+   `Dynamo does not know how to trace the builtin 'posix.stat'` —
+   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+   wrapped body to build its guard even when the cache is warm (it warns about
+   exactly this), verified in-image — a *warm* lru_cache-calling function still
+   fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+   four processes) and die on
+   `RuntimeError: Graph breaks are not supported with aot compile`, because this
+   build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
+   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
+   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
+   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
+   submodule attribute name, also present in the `load_weights` shard table
+   (`model.py:780-785`). Reading the full traceback (the reported line is inside
+   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
+   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
+   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
+   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
+   from those names at :1225. So the compiler is failing to **resolve a submodule by
+   name** while tracing the GLM decoder layer; whether the root cause is the fork's
+   module tree or dynamo's registration is not yet settled. The next step is the one
+   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
+   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
+   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
+   environment).
+
+4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
+   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
+   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
+   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
+   break dynamo reported; after this patch the report moves on to the workspace
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
+   below.
+5. **A launcher bug that made four boots misleading — worth knowing because the
+   failure kept moving.** The scratch launcher's compile branches were restructured
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
+   whichever break came first:
+
+   | config | outcome |
+   | --- | --- |
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
+
+**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
+with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
+check which configuration you are actually in (`podman exec <ctr> printenv
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
+survives) and it prints, at the miss, the node target, the near-miss keys, and
+whether another live `OutputGraph` holds the key.
+
+**State of the changelist (2026-09-30, latest).** The trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
+`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
+
+- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
+  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
+  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
+  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
+  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
+  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
+  is how inlined frames surface).
+- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
+  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
+  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
+  with the probe confirmed installed in both the API-server and worker processes.
+- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
+  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
+  the `load_weights` shard table (`model.py:780-785`). The four
+  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
+  exactly this bare key, but every one of them is guarded (`if proj_name not in
+  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
+  keyed by class name, not module path.
+- It reproduces on **every** run of this configuration (5 of 5), and it is
+  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
+  that were never fatal in the first place — the KeyError is the only thing standing
+  between this changelist and a boot.
+
+**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
+`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
+frames with their source), or wrap the inlining candidates in that forward — the mHC
+CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
+indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
+is a dynamo-side name-resolution failure and not a fixable fork bug.
+
+Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
+fallbacks on the decode path that the fork author already replaced, so they are not
+the blockers they appear to be at first glance.
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+  sources. The obvious second lever — the 9 % router — is gated
+  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+  (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+**The load was tried, and the gap is exactly one interface.** Payload fetched to
+`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
+2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
+`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+accepts the config and the drafter and dies at worker init with:
+
+```
+RuntimeError: Model does not support EAGLE3 interface
+```
+
+So nothing about the drafter, the method name, or the payload is wrong — the GLM
+trunk simply does not implement the aux-hidden-state contract the dflash runner
+requires. The reference is in this same image, twice: `deepseek_v2.py`
+(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
+capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
+small:
+
+- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
+  by the wrapper's `set_aux_hidden_state_layers()`;
+- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
+  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
+  rank holds fewer tokens than `positions`);
+- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
+  list is non-empty;
+- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
+  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
+  the runner derives trunk indices `[6, 15, 25, 34, 43]`).
+
+**The GLM-specific unknown** is what "the single-stream state entering layer k"
+means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
+`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
+returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
+`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
+(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
+a semantic choice to validate by acceptance rate, not by inspection.
+
+**Budget no speedup.** Per the profile above, a verification pass pays pointwise
+cost proportional to its token count — the same reason MTP4 measured flat — and
+MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
+*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
+it is small), not as a performance fix, and the honest expectation on this shape is
+flat decode with a prefill regression like MTP's until the pointwise mass is fused.
+
+**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
+independent of the compile changelist — it patches the same file as compile-01, so
+compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
+implements exactly the contract read out of this image: `EagleModelMixin` on
+`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
+layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
+the three `SupportsEagle3` member attributes plus both interface methods on the
+wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
+`v1/worker/gpu/cudagraph_utils.py:621`).
+
+**The patch went through five versions, and the decisive discovery was where the
+instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
+is called on the model the runner LOADED — and for GLM-5.3-Flash that is
+`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
+which holds the text model as `self.language_model = Glm5NextForCausalLM`
+(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
+with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
+both wrappers** and is the first version to clear the gate (both TP workers log
+`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
+
+The full evolution, each an instructive trap:
+
+1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
+   are not the whole contract.
+2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
+   `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
+   resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
+   Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
+   cannot take `SupportsEagle3` in *any* position.
+3. **v3** declared the members as class attributes (MRO-safe) — but only on
+   `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
+4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
+   `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
+   `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
+   experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
+5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
+   (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
+   `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
+   `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
+
+**The lesson generalizes twice over**: when a runtime error names an interface, read the
+interface's declaration first — and establish WHICH INSTANCE the check runs against
+before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
+models, not pass-throughs, and the text model behind `self.language_model` is invisible
+to a check run on the wrapper.
+
+**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
+`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
+torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
+hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
+instead — it needs no startup hook.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
+
+### The KV-cache grouping journey: clearing the unifier, then the kpool tail
+
+With patch 01 v5 the gate passes and the boot moves to KV-cache layout
+construction — where the next two blockers live, both in
+`v1/core/kv_cache_utils.py`:
+
+1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
+   spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
+   entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
+   specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
+   `unify_kv_cache_spec_page_size` raises
+   `NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
+   MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
+   they CAN grow `block_size` when the max page is an exact multiple. The
+   indexer's page is just `tokens_per_state x` the MLA page, so
+   **`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
+   to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
+   layers then take the legal divisible block-growth branch, drafter + mamba
+   layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
+   passes, torch.compile passes, and the boot reaches cudagraph capture — the
+   NotImplementedError is gone.
+2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
+   `kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
+   `assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
+   `[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
+   `Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
+   page (8192 B) *divided* the raised max page, so the unifier's divisible branch
+   grew its `block_size` by the ratio and the view became
+   `[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
+   promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
+   so `disable_hybrid_kv_cache_manager` is unset and
+   `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
+   (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
+   growing its block) **verified 2026-09-30**: the assert is gone and the boot
+   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
+   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
+
+3. **The capacity wall: the generic path cannot alias the mamba states.**
+   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
+   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
+   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
+   per-token KV). Two coupled causes, read out of the image:
+   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
+     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
+     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
+     expert plan's KV registry is calibrated at 13,600 B/token). On the
+     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
+     unaliased state pages appears.
+   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
+     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
+     KV at the plan's reserve, so even a correct generic-path estimate would
+     be starved.
+
+   Learned en route: the engine AUTO-SETS the attention block size to
+   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
+   tokens to ensure that attention page size is >= mamba page size`), so every
+   attention group's block counts scale from that floor — a 16-token drafter
+   block would need 4x the shared `num_blocks`; with the global 2,432 all
+   groups align.
+
+   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
+   for the registry) keeps the aliased glm5 pool and extends it: the grouping
+   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
+   `MLAAttentionSpec` subclass — excluded before the exact-type
+   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
+   specs; the drafter layers form their own uniform group (per-group pages,
+   NO unification — the MLA block size stays at the engine-set 2,432 and the
+   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
+   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
+   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
+   equal) as aliased pool + appended draft/hidden tensors with their own
+   strides; the memory estimator adds the extras. Patch 04b extends the
+   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
+   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
+   `speculative_config.use_aux_hidden_state_outputs` is set.
+
+   The pure logic is smoke-tested locally (CPU torch + the image's real spec
+   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
+   layout recognition, builder block math and memory accounting all pass.
+   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
+   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
+   2,432 tokens ≈ 70,500 ≥ 65,536.
+
+All of these mount over site-packages via the scratch launcher's
+`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
+`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
+`expert_activation_registry.py`; apply-verified against the image's pristine
+files). The window script's decision pipeline was also fixed for a SIGPIPE
+death (`head -80` on a `podman logs` grep pipeline killed one run before
+log-save and restore; the log is now captured to a file FIRST, then grepped).
+
+
+## Measured arms (all 2026-09-30, one boot each)
+
+| arm | decode | prefill | verdict |
+| --- | --- | --- | --- |
+| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
+| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
+| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
+| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
+
+### Why MTP is rejected here (it is not that it fails)
+
+MTP **works** on GLM in this fork: the head loads, the architecture resolves to
+`Glm5NextMTPModel`, and acceptance is excellent —
+
+```
+SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
+Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
+Avg Draft acceptance rate: 69-97%
+```
+
+— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
+prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
+verification forward gathers the **union** of the batch's experts, so spec
+decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
+of arena. On a compute-bound (fully-resident) shape this would invert; here it
+does not.
+
+## Served surface (verified 2026-09-30 against the live engine)
+
+| capability | state | evidence |
+| --- | --- | --- |
+| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
+| completion | yes | `bench_glm.py` runs above |
+| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
+| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
+| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
+
+Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
+requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
+breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
+newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
+
+## Fork traps found (for the engine authors)
+
+1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
+   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
+   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
+   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
+   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
+   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
+   the MTP draft step demands an op that cannot exist here:
+   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
+   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
+   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
+   CDNA check, not the env var.)
+   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
+     more tokens in the same reserve, and this fork's reserve model is
+     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
+     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
+     dominated by the per-sequence KDA state (34 linear-attention layers) and the
+     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
+     cover — so halving the KV dtype cannot move it.
+   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
+     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
+     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
+     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
+     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
+     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
+     **do not spend effort here** — the payoff is capacity, and the reserve is
+     dtype-blind.
+2. **The plan's KV reserve under-models this model, and the cap enforces the
+   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
+   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
+   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
+   `--max-model-len` becomes unbootable:
+   `ValueError: To serve at least one request with the model's max seq len
+   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
+   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
+   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
+   fixed) — where the reserve would eat a third of the arena.
+   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
+   request raise the cap; **it is not part of the deployed shape**, which boots
+   on the plan's own reserve.
+3. **A window is forced, not chosen.** The production `max_model_len 131072` is
+   what the reserve model requires for this payload (below ~92k the boot
+   refuses); it is not a capacity decision.
+4. **Expert prefetch remains poison** — unchanged from the qwen campaign
+   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
+
+## Operations
+
+```bash
+# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
+# dirs are untouched on disk):
+git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
+./scripts/deploy.sh dankserver --service frank-primary
+
+# payload inventory (34 shards + 2 extras, weight-map index):
+bash scripts/download-frank-primary.sh --verify-only
+
+# live telemetry (needs the profiler window; with_stack=false is mandatory —
+# with stack the stop path hangs and the trace buffer grows to tens of GB):
+podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+```
+
+**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
+**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
+`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
+liveness probe a lying oracle for a restore — twice a restore script printed
+`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
+still in place. The restore oracle must be a **conjunction**: `is-active` ==
+`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
+mask symlink **and** the container is owned by the unit and not `Stopping`. The
+repair recipe, all four steps needed:
+
+```bash
+sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
+sudo systemctl daemon-reload
+sudo systemctl reset-failed frank-primary.service                            # a failed unit refuses to start
+podman rm -f frank-primary-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
+sudo systemctl start frank-primary.service                                   # then ~7 min to health 200
+```
+
+Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
+glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
+container, so one whole round produced no evidence at all (`no container with name
+glm-probe found`) — check `proc://` for running jobs before opening a window.
+
+**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
+bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
+name-map lookups and vLLM module forwards) produced a self-replicating chain of
+`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
+imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
+that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
+the hook) and spawned again. It was still alive 18 minutes later with the container
+still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
+identical interpreters, each child's PPID being the previous one. Guard (now in the
+hook): arm once per environment via a marker in `os.environ`, which children inherit —
+
+```python
+if os.environ.get("VLLM_PROBE_ARMED") == "1":
+    pass                       # an ancestor already armed it; stay inert
+else:
+    os.environ["VLLM_PROBE_ARMED"] = "1"
+    <install probes>
+```
+
+Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
+generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
+variable — **never** pattern-kill an interpreter name, because the media stack on this
+host (romm, jellyfin, plex, glances) runs python in production.
+
+```bash
+# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
+
+# experiment windows (production stopped + `systemctl mask --runtime`, which
+# self-clears at reboot) ran from the host-side scratch launcher
+# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
+# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
+# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
+# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
+# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
+# word-split by the launcher's unquoted $EXTRA).
+bash /home/dank/serve_glm_autoround.sh
+```
+
+Boot gate: the engine refuses a legal-looking shape in two measured ways — the
+KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
+64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
+what killed the first two boots at util 0.95.
+
+`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
+shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
+the default allocator's fragmentation fails the profile/graph-capture phase
+(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
+MiB is free").
+
+The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
+walks every guarded vhost — including `http://192.168.0.112:8989/` on
+FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
+timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
+healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
+`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
+and the WoL packet did not wake it (macOS "Wake for network access" needs
+toggling on the mini itself).
+
+## Rollback targets
+
+| state | where |
+| --- | --- |
+| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
+| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
@@ -0,0 +1,166 @@
+---
+type: reference
+title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
+description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - sglang
+  - rtx3090
+  - qwen
+  - mtp
+  - vision
+  - int8
+timestamp: 2026-10-01
+---
+
+# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
+
+> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
+> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
+> (7771), same alias (`frank-small`), same client contract surface.
+
+## Identity
+
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
+  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
+  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
+  systemd unit `scripts/frank-small.service` (`User=dank`,
+  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
+  tunables from `scripts/frank-small.env`.
+- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
+  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
+
+## Shape
+
+- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
+  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
+  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
+  tower (27 layers) in the safetensors — no separate mmproj file.
+- **Context:** two 262,144-token slots at the model's native window
+  (`--context-length 262144`, `--max-running-requests 2`) against a
+  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
+  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
+  ~82k tokens of pool slack, and a compaction request (frank-primary's
+  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
+  one request. A single request may grow to the whole pool — a
+  170,019-token prompt was verified at the 4-slot shape — the grid is
+  the two-way concurrency contract.
+- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
+  dim x 1 B), pool 9.31 GiB.
+- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
+  ssm state pool is a first-class consumer. The engine autofills the
+  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
+  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
+  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
+  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
+  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
+  dies in capture, or later in the first long request once Triton's
+  deferred kernel loads eat the remainder). Hence
+  `--disable-prefill-cuda-graph`; decode graphs are untouched.
+- **Quantization:** `--quantization auto-round` — the checkpoint's
+  `quantization_config.json` declares `quant_method: "auto-round"` and
+  SGLang loads it natively (vision tower, `embed_tokens` and
+  `linear_attn.in_proj` stay fp16).
+
+## Spec method
+
+**None — and this is a checkpoint fact, not a tuning choice.** The
+`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
+safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
+`model.safetensors.index.json` for both this INT8 conversion and the base
+`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
+`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
+boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
+decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
+paying the 4-token verify tax for no token. The serve script passes no
+`--speculative-*` flag, and the seam gates that.
+
+## CUDA 13 toolchain (the sharp edge)
+
+The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
+flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
+venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
+`--compress-mode`) and its headers clash with the 13.0 device passes.
+
+Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
+
+| Package | Supplies |
+|---|---|
+| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
+| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
+| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
+| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
+| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
+| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
+
+Three local modifications the tree carries:
+
+1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
+   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
+2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
+   `../../../include` / `../../../lib64` (the deb ships them as real
+   near-empty dirs; the stub pass must see the 13.0 headers, not
+   `/usr/include` 12.4).
+3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
+   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
+   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
+   the unpatched 13.0 declarations are incompatible in the C++ host stub).
+
+JIT caches: `~/.cache/sglang/jit/sm86/` and
+`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
+change lands.
+
+## Payload
+
+Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
+files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
+all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
+re-checks). The serve script refuses to start with a missing shard.
+
+## Client contract
+
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to **`reasoning_content`** (live-verified
+  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
+  as llama.cpp; models.yml sets `reasoningContentField:
+  "reasoning_content"`). `--reasoning-parser qwen3`.
+- **Tools:** `--tool-call-parser qwen3_coder`.
+- **Vision:** supported (in-checkpoint tower).
+- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
+  `--top-k` — sampling rides on the request; the harness sends the Qwen
+  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
+- **Reasoning effort:** the full OpenAI set is accepted (the chat template
+  has no effort gate).
+
+## Measured
+
+- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
+  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
+  above is what makes that compile succeed).
+- Decode **83.9 tok/s** (no spec decode — nothing to accept).
+- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
+  50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
+  incl. 8 decode tokens, ~2,056 t/s).
+- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
+  (216k tokens of live KV at once). 2-slot shape: two concurrent
+  50,000-token requests both answered (28.8 s wall).
+- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
+  well past frank-primary's 183,501-token compaction trigger (70% of its
+  262,144 window) — prefills and completes in ONE request.
+- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
+
+## Rollback
+
+[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
+remain intact on disk; its unit/env/serve trio is in `old/` after the
+cutover.
+
+# Citations
+
+- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
+- SGLang: https://github.com/sgl-project/sglang
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "halogen-flash-server on Dankclaw (frank-large)"
-description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
+title: halogen-flash-server on Dankclaw (frank-large)
+description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
 tags:
  - llm
  - halogen
@@ -13,3 +13,466 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
+
+# halogen-flash-server on Dankclaw (frank-large)
+
+> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
+> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
+> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
+> [[frank-large-ik-uncensored]], the current rollback), and came back as the
+> shape the fleet asked for: two 200k-token conversations rather than the
+> native-context pair. Deployed as a rootless Quadlet container,
+> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
+> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
+
+The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
+architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
+Upstream's own comparison table measures it at **roughly 4x the prefill
+throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
+ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
+prompt with real context. That, not decode, is the reason for the swap.
+
+## Shape
+
+One container runs both halves of the stack:
+
+| half | port | reachable |
+|---|---|---|
+| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
+| OpenAI front-end | 8731 | published as host **7770** |
+
+The `all` mode is upstream's default and the shape a single-container host
+wants: the front-end talks to the engine over loopback inside the container, so
+there is one process tree, one log, and one published port.
+
+`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
+`/health`, and in every response body. Requests are **never rejected for naming
+a different model** — the server serves whatever asks. Keeping the fleet alias
+is therefore free, and it means Hermes, the harness config, the OpenWebUI
+discovery, the `services.json` probe and the docs needed no rename.
+
+## Weights
+
+`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
+
+| file | size | role |
+|---|---|---|
+| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
+| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
+| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
+| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
+
+`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
+only** — the entrypoint acts only when the checkpoint is absent, so a restart
+never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
+mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
+`/home/dank/models/halogen`.
+
+**Refreshing the weights.** Delete the mount's contents (or the whole
+directory, then recreate it **empty**) and start the unit: the entrypoint sees
+the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
+traps. First, the bind mount's source directory must *exist* — podman refuses
+the container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
+directory and the fetch starts on the next start. Second, "refresh" only means
+"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
+(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
+2,383,306,048 → **2,478,095,488**) while the checkpoint returned
+byte-identical in size (124,068,083,904) — so an update can live entirely in
+the sidecars, and equal file sizes are not proof of an unchanged repo.
+
+`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
+refresh) comes along with the repo but is unused: it trades the `o_proj`
+calibration for about 2 % of decode, and the quality sidecar is the default for
+a reason.
+
+## Deployed flags
+
+Everything else is configured by environment variable — there is no config
+file, and the flags are read once at startup. These are the departures from
+upstream's defaults:
+
+| flag | value | why |
+|---|---|---|
+| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
+| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
+| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
+| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
+| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
+| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
+| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
+| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
+| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
+
+Everything else is upstream's default, deliberately: prompt cache mode `2`
+(two entries per conversation — end-of-system-prompt and end-of-history — which
+already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
+stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
+temperature 0, and it stands down whenever a second conversation is
+generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
+gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
+
+## Memory model
+
+Three numbers explain the whole box:
+
+- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
+  GPU in place, never copied. The kernel counts the locked pages as reclaimable
+  file cache, so `free`, `MemAvailable` and every dashboard built on them
+  **overstate free memory by the size of the model**. The server prints the
+  corrected figure at startup; believe that line, not Grafana.
+- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
+  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
+  reads the two together in the engine's own words: `400384 positions, ~27.2
+  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
+  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
+  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
+  128 GB machine *with the weights pinned* — and lower on a machine carrying
+  more besides this server, which Dankclaw is.
+- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
+  table is read through. Too little and long prompts crawl with the disk busy
+  and the process in uninterruptible sleep — that is a cache problem, not a
+  memory problem, and the fix is a *smaller* pool, not a bigger one.
+
+`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
+the pool itself if the configured one will not fit, printing what it chose.
+Read those lines on every restart; they are the ground truth for sizing.
+
+**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
+startup account, including three deliberate experiments:
+
+| pool | max_tok | reserve | KV | arena | need | budget | verdict |
+|---|---|---|---|---|---|---|---|
+| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
+| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
+| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
+| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
+| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
+| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
+| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
+| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
+
+### The sizing rule
+
+One budget covers the pool, the arena **and** the vision tower together:
+
+```
+device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
+              = 121.2    - 67.7             - reserve
+                                    reserve 24 -> 29.5 GiB
+                                    reserve 15 -> 38.5 GiB
+
+need = KV (28 KiB/position) + arena + vision (1.97 measured)
+arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
+```
+
+The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
+121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
+Use the unrounded value. Two of the three terms are fixed hardware and
+checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
+buys the *fit margin*, not the host's memory. It is a term in this inequality,
+not an allocation, and the section at the end of this page shows the live test
+that proves it.
+
+Trust the engine's printed requirement over this model wherever they disagree:
+a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
+of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
+enough to plan with, and the authoritative number to cut the other way.
+
+The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
+point to 0.1 GiB — including upstream's three published "on the device" figures
+(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
+**pool + arena totals at the default `max_tok`**, not pool alone. That is why
+their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
+49.4. Reading those figures as pool-only is the trap that makes a doubling look
+affordable when it is not.
+
+At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
+13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
+ceiling, not a conservative setting.** The corner check agrees: at 32768 only
+7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
+262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
+= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
+fits unmodified.
+
+### What the reserve actually costs
+
+The device budget is not the whole story. Memory position by configuration
+(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
+
+| config | held | host left | contiguous 2 MiB blocks |
+|---|---|---|---|
+| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
+| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
+| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
+| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
+| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
+
+At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
+engine volunteered the stall note. It did not print it at the 401,408 trim, and
+it prints it again after the 2026-09-16 retune. Three data points, and they do
+not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
+not, 475 did. So treat the note as "the machine is tight" and read the
+`host memory left` line rather than waiting to be told:
+
+```
+halogen: NOTE this server now holds most of the machine, and 310 contiguous
+2 MiB blocks remain. It runs best on a host of its own. If other large
+processes share this machine they will compete for what is left, and when it
+runs out BOTH they and this server can stall for minutes at a time at full
+CPU with no disk activity and no output, which is not a crash and needs no
+restart.
+```
+
+That is the failure mode to watch, and it is the least diagnosable of the
+three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
+looks like a hang, for the neighbours as much as for frank-large. `free` is
+useless for spotting it — `MemAvailable` still reads ~74 GiB because the
+page cache counts as reclaimable. Watch the `host memory left for everything
+else` line and the contiguous-block count at boot instead.
+
+28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
+so it is survivable.
+
+### The reserve cannot buy air — ceiling 16
+
+The last row above is the experiment that proves it. Raising the reserve from
+15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
+GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
+allocation; the actual footprint is `weights + KV + arena + vision`, none of
+which the reserve touches. So the knob has exactly two regimes:
+
+| reserve | budget | vs required 36.5 | effect |
+|---|---|---|---|
+| 15 | 38.5 | +2.0 | deployed |
+| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
+| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
+| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
+| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
+
+The 36.5 requirement is the engine's own, read from a clamped boot:
+`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
+thing a bigger reserve can do is take the doubling away.
+
+**To actually free host memory, change the pool or the arena**, and the reserve
+then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
+7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
+are one-line edits that the fit check will confirm on the next boot. The
+2026-09-13 trim to 401,408 was this remedy applied at half strength —
+`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
+with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
+modelled 3.4 because the engine holds the freed KV as working slack.
+
+### The `WARNING` line only appears when the fit check had to move
+
+This one cost me a wrong conclusion, so it is worth stating plainly. When the
+configured pool already fits, the engine reserves it **and prints nothing** —
+the 09-11 baseline boot goes straight from the pre-flight line to
+`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
+its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
+after a clamp**.
+
+So the absence of a warning in old logs is not evidence of headroom. Both
+`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
+baseline has lived in that state since it was deployed without incident,
+because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
+at that point). The engine's budget is a conservative floor, not a hard wall.
+
+### Two failure modes, and only one is dangerous
+
+Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
+outgrows the budget, and the server cannot allocate at all:
+
+```
+kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
+kv pool: WARNING even 262144 positions (the context, the smallest pool that can
+         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
+dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
+```
+
+This is the quiet one: the unit stays `active`, the container reports
+`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
+`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
+large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
+slower".
+
+Under-running the ceiling is far more benign and far easier to miss. At 524288
+with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
+`deploy.sh` still reported healthy in 12 seconds:
+
+```
+kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
+         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
+```
+
+A green health check does not mean the pool you asked for is the pool you got —
+read the `kv pool:` lines after every restart.
+
+The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
+the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
+`/proc/meminfo` figure at all.
+
+**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
+decode numbers were measured with, and it means this container holds most of
+the host. If a co-tenant then takes the remainder, both it and the engine can
+stall for minutes at a time at full CPU with no disk activity and no output.
+That is documented behaviour, not a crash, and it needs no restart.
+`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
+decode speed — a last resort, not a tuning option.
+
+## Client contract
+
+The differences from llama.cpp that bite are all in the request, not the
+deployment:
+
+- **No server-side sampler defaults.** `temperature` absent or `0` is
+  **greedy** decode; above 0 the request samples on the same drafter. Supported
+  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
+  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
+  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
+  value outside its defined range (values are refused, never clamped).
+  The fleet runs greedy for clients that stay silent — deterministic, and
+  byte-identical with the drafter.
+- **The token budget covers thinking.** The default is **8192**, and a budget
+  that runs out mid-thought returns `finish_reason: "length"` with an **empty
+  `content`** and the partial reasoning in `reasoning_content` — which most
+  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
+  request is a 400 rather than a silent truncation. `max_tokens`,
+  `max_completion_tokens` and `max_output_tokens` all work and must agree if
+  more than one is sent.
+- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
+  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
+  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
+  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
+  the window while sitting below the compaction trigger. That collided live on
+  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
+  harness-side: keep the declared `maxTokens` at or under
+  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
+  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
+  breaks the two-resident-conversations property.
+- **`reasoning_effort`** is accepted at the top level and inside
+  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
+  model's default is `xhigh`. `minimal` aliases the template's `low` and still
+  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
+  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
+- `/v1/models`, `/health` and completions all report the served id, and
+  `reasoning_content` carries the reasoning trace.
+
+**Pool admission is the one arithmetic trap.** A request reserves its prompt
+**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
+the request waits in arrival order, and after the queue timeout (3600 s) it
+gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
+the pool can therefore never be admitted at all. The 2026-09-16 shape is a
+400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
+conversations at the ceiling fit the pool exactly, so a pair that is both
+near-full and both generating is pool-bound — the second waits for the first to
+release its region. The pool binds **how many** conversations stay resident,
+not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
+tables above.
+
+## Vision
+
+**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
+the tower is a 1.97 GiB charge against the same device budget as the pool and
+the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
+which loads the BF16 projector alongside its own weights. This server refuses
+image content parts and says why, rather than ignoring them:
+
+```json
+{"error": {"message": "this server does not accept images: the engine was
+started without a vision tower (HALOGEN_VISION_TOWER)", "type":
+"invalid_request_error"}}
+```
+
+The llama.cpp side handles images the same way this engine used to: a `data:`
+URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
+refused deliberately (fetching one would make the server issue outbound
+requests to wherever a client pointed it). Audio and video have no path
+anywhere in the fleet — that left with [[frank-npu]].
+
+## Deploy and operate
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+```
+
+The handler retires the ik-era **root** unit first (stop, disable, move
+`/etc/systemd/system/frank-large.service` into
+`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
+`disable` may delete the file outright, which the handler now reports), stops
+the Quadlet's own unit if it is running, then pulls
+`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
+`localhost/` alias the unit references, and **waits for port 7770 to be free**
+before installing and starting anything.
+
+That last gate is not paranoia. On the first cutover attempt the ik server
+logged "Deactivated successfully", and pasta — rootless podman's userspace
+network stack, which binds the published port at container CREATE — still
+reported `Address already in use` a second later, exiting the unit with code
+126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
+starts a doomed container has not deployed anything, so `wait_port_free` polls
+every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
+the socket table if it never clears.
+
+Boot persistence comes from `[Install] WantedBy=default.target` in the unit
+file plus linger on the deploy user — generated units **cannot** be
+`systemctl --user enable`d on this host.
+
+```bash
+systemctl --user status frank-large.service     # the generated unit
+podman logs frank-large | head -40              # version banner + memory ledger
+curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
+curl -s http://127.0.0.1:7770/v1/models
+```
+
+`/health` asks the **engine** for a PONG before answering, so a 200 means
+"ready to serve", not merely "process alive". During the first-start download
+it stays unhealthy by design.
+
+## Troubleshooting
+
+| symptom | cause |
+|---|---|
+| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
+| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
+| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
+| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
+| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
+| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
+
+`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
+already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
+deliberately **not** set: it logs every allocation over 64 MB, which is
+steady-state noise on a shared host — raise it only for a live OOM report.
+The lines worth grepping first:
+
+```bash
+podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
+```
+
+Retrieval: `podman logs frank-large` covers the **current** instance only —
+the Quadlet runs `--replace --rm`, so everything before the last recreate is
+gone from the container log. `journalctl --user -u frank-large.service` is
+the durable record. Both are rootless: `sudo podman logs frank-large` looks
+in root's storage and answers "no such container".
+
+## Rollback
+
+The predecessor is intact: the ik payload stays at
+`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
+ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
+serve script and the Qwen-Sharp template are archived in the repo under
+`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
+is a `git revert` of this change set plus a deploy — **but the Quadlet must be
+removed from `~/.config/containers/systemd/` first**, or the restarted root
+unit and the still-installed container both claim port 7770. The port gate in
+the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
+
+The `amd_iommu=off` kernel argument that came with this change is separate and
+reverts on its own — see [[dankclaw-iommu-translation-off]].
+
+## Citations
+
+- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
+- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
+- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -1,14 +1,21 @@
 # LLM Fleet — Knowledge Base

-Documentation for the LLM inference fleet on the two hosts (2026-09-27):
-**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
-serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
-PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
-shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
-stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
-Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
-iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
-QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
+Documentation for the LLM inference fleet on the two hosts (2026-10-01):
+**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
+digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
+orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
+on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
+(the model's full native window)**, fp8 KV (5 boot but leave no runtime
+headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
+reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
+measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
+the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
+runs two endpoints —
+gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
+unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
+on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
+HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
+port 7771).

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -22,11 +29,11 @@ graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[R9V radiance-vLLM fork, docker-pinned
-via docker-podman shim
-frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next Q8_0
-TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
+        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+podman run, frank-primary, port 7770]
+        C[Qwen3.8-Flash-Next FP8 bf16 no-spec
+TP=2, expert offload automatic, 4 x 262144 ctx,
+fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -58,20 +65,26 @@ frank-small, port 7771]

 ## Active Deployment

- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  runbook (2026-09-27, root systemd unit, port 7770): the R9V
-  radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
-  image bundle `2dac17a2`, driven through a docker→podman shim) serving
-  the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
-  blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
-  (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
-  131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
-  `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
-  inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
-  `reasoning`, parser `qwen3_coder`. The tuned shape lives in the
-  on-host q8xl setup state. Rollback:
-  [[frank-primary-r9v-radiance-qwen38-uncensored]]
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
+  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
+  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
+  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
+  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
+  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
+  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
+  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
+  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
+  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
+  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
+  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
+  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
+  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; payload and R9V state dirs intact on disk, git tag
+  frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
+ [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
  (same-day swap to the Q8_0 payload; rollback target, state + payload
  intact on disk): the R9V radiance-vLLM fork serving
  Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -169,6 +169,18 @@ The LLM inference fleet on the two hosts. Full index:
  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
  slab size)
+- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
+  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
+  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
+  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
+  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
+  checkpoint of the same model reaches TritonWNA16Experts, which has no
+  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
+  35 tok/s target is NOT met** — the step is bounded by the fork's gather
+  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
+  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
+  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
+  `glm47` tool/reasoning parser wiring
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
@@ -257,10 +269,16 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
+  2026-10-01: SGLang in a self-contained python3.12 venv serving
+  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
+  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
+  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
+  4 x 151,552 slots, bf16 mamba states, eager prefill
+- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
+  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
  Dankclaw's RTX 3090, port 7771
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
@@ -1,5 +1,151 @@
 # Bundle Change Log

+## 2026-10-01
+- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
+  GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
+  served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
+  captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
+  interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
+  draft+hidden groups, registry method-signal), but **acceptance pinned at
+  1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
+  aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
+  8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
+  (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
+  not originate from this engine (APIServer access log: every completion
+  POST was host-local; his workstation appears once, a /v1/models GET).
+  Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
+  (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
+  (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
+  KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
+  unifier/tailpin/grouping/registry fixes, apply-verified).
+
+## 2026-09-30
+- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
+  the 35 tok/s target, and the shortfall is measured rather than inferred.**
+  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
+    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
+    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
+    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
+    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
+    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
+    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
+    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
+    refused to serve at all. That checkpoint was purged with the other retired
+    payloads.
+  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
+    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
+    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
+    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
+  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
+    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
+    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
+  - **The wall — and a corrected one.** The first account of the shortfall
+    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+    fusions that DO happen 6 %. The real cause is that the fork compiles its own
+    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+    engine says so at boot ("...provides neither a compiled submodule nor
+    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+  - **Fork work staged (compile changelist, WIP — does not boot yet).**
+    `scripts/patches/glm53-compile-01-model-decorator.patch`,
+    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+    evidence header and base/patched sha256. Boots established, in order: (1)
+    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+    available -> os.path.exists` breaks the graph (a plain module-global read
+    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+    `fullgraph` because its indexer metadata syncs to host
+    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+    outright. With all three the trunk **traces** (`Dynamo bytecode transform
+    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+    mHC/KDA path — a latent bug the compile exposes. That is the remaining
+    blocker.
+  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+    sources), which binary-blocks the second-largest lever: the fused router is
+    gated `num_experts in {128, 256, 512}` and only
+    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+    reserve is dtype-blind), only removing gfx12's only MLA decode path.
+  - **DFlash2 for GLM is a target-side port, not a drafter port.**
+    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+    proportional to its token count.
+  - **DFlash2 tried, and the gap was exactly that one interface — now closed.**
+    Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
+    2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
+    `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+    accepted the config and the drafter and died at worker init with
+    `RuntimeError: Model does not support EAGLE3 interface`. The port
+    (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
+    hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
+    text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
+    wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
+    `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
+    `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
+    workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
+    DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
+    by experiment-window RAM contention (see the window discipline below), so the
+    drafter-serving end state was still unproven when this entry was written.
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
+    indexer's in-forward workspace acquisition
+    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
+    — and the cause was **my own scratch launcher**: its compile branches were
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
+    (the flag only logs). The moving reports are still useful — they are the break
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
+  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
+    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
+    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
+    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
+    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
+    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
+    chaining the image's apport hook), and the scratch launcher used to reset
+    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
+    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
+    the pinned image and to reproduce their recorded sha256.
+  - Measured and rejected: **MTP4** (the head loads, arch resolves to
+    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+    The profiler window above explains it: the per-token cost is dominated by
+    pointwise glue that scales with the token count, so verifying a draft batch
+    pays ~N x that glue for the same N tokens of output);
+    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
+    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
+    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
+  - Fork bug found: the plan's KV reserve under-models this model's
+    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
+    that reserve, making a legal `--max-model-len` unbootable. Shipped
+    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
+    request raise the cap) — NOT needed by the deployed shape.
+  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
+    state dirs are intact on disk.
+

 ## 2026-09-29
 - **frank-large tuning investigation: a measured NEGATIVE result, nothing
@@ -516,3 +662,216 @@
 - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
 - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
 - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
+
+## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
+
+bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
+--expert-precise-memory:
+
+- with precise: NRestarts 1->3, each attempt ~6.7 min then
+  `Deactivated successfully` (API server drains after an EngineCore fatal
+  error) — cause not captured, the serve script rm'd the container.
+- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
+  climbing, container `Exited (1)`, and the journal finally showed the real
+  error:
+  `RuntimeError: Worker failed with error 'Constraints violated
+   (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
+   as dynamic but your code specialized it to be a constant (3)'`
+  — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
+  `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
+  fresh container filesystem, so a stale compile cache is ruled out: the
+  conflict is produced by this configuration's own compile path.
+
+Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
+KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
+(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
+`--enforce-eager` is the bypass that would sidestep the constraint path.
+
+Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
+Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
+
+## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
+
+**Reverted.** The two changes Frank asked for were applied together and the
+result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
+~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
+and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
+new `Restart=always` turned a silent death into a visible loop).
+
+The budget arithmetic explains it without needing the error text, because the
+arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
+that is exactly what `rocm-smi` measured):
+
+| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
+|---|---|---|
+| non-expert weights | 6.28 | 6.28 |
+| pinned pool | 6.80 | 5.68 |
+| CUDA graphs | 0.55 | 0.55 |
+| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
+| non-torch | ~1.45 | ~1.45 |
+| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
+| free at boot | **~0.13 GiB** | 3.59 |
+
+`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
+the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
+put graph capture ~0.13 GiB from the ceiling, where every transient allocation
+dies. So the flag is dropped and bf16 2 x 262144 ships with the default
+overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
+
+**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
+default — it is the budget for the phase that allocates last. Reclaim it only
+by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
+
+## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
+
+Frank's final shape for the night. Two changes at once, both measured in
+principle before the boot:
+
+- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
+  bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
+  11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
+  1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
+  524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
+  3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
+- **`--expert-precise-memory`.** The registry's default overhead for this
+  architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
+  nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
+  The flag selects the fitted figure, which moves only the plan's `room` term
+  — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
+  cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
+  1024 nbt above 4096, so MNBT 2048 pays nothing.
+
+Budget note worth keeping: the arena is `(budget - non_expert_weights -
+kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
+planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
+with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
+planned arena; it comes out of free VRAM instead.
+
+## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
+
+Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
+UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
+sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
+uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
+drafter and its fetcher are unchanged.
+
+Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
+KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
+to `kFp16`), so "unquantized" is inherent.
+
+KV arithmetic (why this fits where one 262144 session used to sit):
+f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
+`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
+2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
+per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
+655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
+weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
+~62 GiB against the 114 GiB GTT.
+
+## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
+
+`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
+deploy gate twice and then passed cleanly on a manual re-run:
+- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
+  — Plex redirects its web UI to the hosted app; the same failure appeared on
+  the previous deploy too.
+- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
+  — the homepage answered `http=200` in 40 ms from the dev box at the same
+  moment, so this is a browser-level flake.
+- run 3 (manual): **33 passed**, exit 0.
+
+So a red gate on this spec alone is NOT proof the fleet is broken, and the
+deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
+`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
+stable underlying issue and it lives in Plex's web-client/image config, not in
+any repo service.
+
+## 2026-10-01 — frank-primary bf16 no-spec cutover
+
+- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
+  night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
+  KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
+  FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
+  "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
+  request: 3.03x".
+- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
+  (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
+  allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
+  (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
+  PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
+  model's native 262,144 does not fit with a 3-seq contract. Every rung was
+  booted under a transient systemd unit at uid dank with
+  `LimitMEMLOCK=infinity` (the rootless bench probe masked the production
+  memory environment) with full engine-log capture.
+- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
+  `--gpu-memory-utilization` shrinks the engine's own budget rather than
+  buying slack; Frank forbade it and the winning ladder ran at 0.98.
+- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
+  `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
+  container-env passthrough added; shape header rewritten.
+- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
+  spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
+  qualified-shape and flags tests flipped to the new env keys.
+- Overnight redo campaign (prefetch OFF everywhere) tabled in
+  `/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
+  with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
+  1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
+  six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
+  KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
+- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
+- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
+  updated out of the GLM era.
+- **Client contract fixed (thinking + tools).** The live endpoint returned a
+  null `reasoning` field and dumped the thinking text INLINE in `content`
+  with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
+  — the malformed thinking block clients choke on — because the deployed argv
+  had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
+  three contract flags and the qwen rollback inherited the gap. Restored from
+  `ecaad36`'s values, confirmed against this fork's registries:
+  `--reasoning-parser qwen3`, `--enable-auto-tool-choice
+  --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
+  '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
+  "NO tool dialect" test is flipped to a PRESENT test.
+- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
+  fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
+  free) — six full-length contexts do not fit at any KV dtype — so the
+  contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
+  rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
+  5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
+  three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
+  same night) is the documented max at 3 contexts.
+- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
+  with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
+  <= 262,144) and
+  `preserve_thinking` moved from a bare top-level body field to
+  `chat_template_kwargs` (the form vLLM actually reads).
+- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
+  passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
+  at runtime; a real client request (995-token prompt, `max_tokens=261149`,
+  sampling params straight from models.yml) killed Worker_TP1 with
+  `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
+  the engine shut down. Boot is not proof of runtime survivability — after a
+  knife-edge shape change, send a large-`max_tokens` request before calling it
+  done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
+  GiB per rank for activations/graph capture. Frank's call 2026-10-01.
+- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
+  `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
+  makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
+  "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
+  down until a human noticed. An explicit `systemctl stop` still wins, so
+  bench windows and deploys are unaffected.
+- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
+  MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
+  contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
+  and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
+  the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
+  decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
+  arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
+- **DCP investigated and closed** (see the parallelism survey in the runbook):
+  `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
+  TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
+  knob on this box.
+- Production bring-up: the shape boots and serves on :7770 — engine log
+  verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
+  contract probes (reasoning split, tool_calls). Rollback target for the
+  2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
@@ -303,8 +303,8 @@
    "section": "AI",
    "details": {
      "role": "Fast utility model",
-      "spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.",
-      "inputs": ["text", "image", "audio"],
+      "spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.",
+      "inputs": ["text", "image"],
      "use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence."
    },
    "url": "",
@@ -19,13 +19,13 @@ load helpers
 ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
 UNIT="$REPO_ROOT/scripts/frank-large.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
+FETCH="$REPO_ROOT/scripts/download-frank-large-q4.sh"
 DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
 BUILD="$REPO_ROOT/scripts/build-gufo.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 # The deployed target's filename, kept here so a rename in the env file and a
 # rename in this stub cannot drift apart unnoticed.
-TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
+TARGET_STUB="Qwen3.8-27B-UD-Q4_K_XL.gguf"
 setup() {
    STAGE="$(mktemp -d)"
    RUNTIME_DIR="$STAGE/gufo"
@@ -144,8 +144,10 @@ argv_value() {
    [ "$(argv_value --served-model-name)" = "frank-large" ]
    [ "$(argv_value --host)" = "0.0.0.0" ]
    [ "$(argv_value --port)" = "7770" ]
-    [ "$(argv_value --context)" -eq 262144 ]
-    [ "$(argv_value --sessions)" -eq 1 ]
+    # 2026-10-01: four sessions x 163840 context (Frank's "4x 160k"), f16 KV.
+    # 163840 tokens x 64 KiB/token = 10 GiB of KV per session exactly.
+    [ "$(argv_value --context)" -eq 163840 ]
+    [ "$(argv_value --sessions)" -eq 4 ]
    [ "$(argv_value --speculative)" = "dflash2" ]
    [ "$(argv_value --draft-tokens)" -gt 0 ]
    [ "$(argv_value --draft-policy)" = "adaptive" ]
@@ -167,11 +169,12 @@ argv_value() {

@test "frank-large stays text-only" {
    # The payload has no mmproj, so an --mmproj would point at nothing. The
-    # fleet's image path is frank-small; models.yml declares frank-large
-    # text-only.
+    # fleet's image path is frank-small: its SGLang MiMo payload carries the
+    # vision tower inside the checkpoint (no projector file at all), and
+    # models.yml declares frank-large text-only.
    serve_argv
    ! argv_has --mmproj
-    grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
+    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REPO_ROOT/scripts/frank-small.env"
 }

@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
@@ -221,7 +224,7 @@ argv_value() {
    # so a plain grep stayed green after the URL was repointed at another
    # repository -- the script would then download something the serve script
    # never asked for.
-    grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
+    grep -qE '^URL="https://huggingface\.co/unsloth/Qwen3\.8-27B-GGUF/resolve/main/' "$FETCH"
    grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
    for f in "$FETCH" "$DRAFTER_FETCH"; do
        # -f so an HTTP error is a failure rather than a small "artifact";
@@ -251,9 +254,10 @@ argv_value() {
 }

@test "the drafter is still fetched by the script that carries its Hub repo" {
-    # Guards a future "simplification" that folds the drafter into the Swift
-    # script and silently orphans the artifact the serve script still points
-    # at via DRAFTER_DIR.
+    # Guards a future "simplification" that folds the drafter into the
+    # payload-fetch script and silently orphans the artifact the serve script
+    # still points at via DRAFTER_DIR. The literal drafter name must not
+    # appear in that script AT ALL, comments included.
    grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
    ! grep -q 'DFlash2' "$FETCH"
 }
@@ -1,19 +1,25 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-29
-# cutover that replaced the R9V radiance fork): the serve script drives
-# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
-# engine flag qualified by boot probe on 2026-09-29 and carried in the repo
-# env file. The seam pins what the flags alone do not say: the qualification
-# law (600k ctx / 3 sessions / q8-q8 KV were MEASURED to fit — 729,545 tokens
-# of capacity at 7.2 GiB per rank — and KV_BYTES is PER RANK), the self-healing
-# law (the run returns before the engine is up, so the serve script must wait
-# on the container for Restart=on-failure to mean anything), the memlock law
-# that makes the tiered-expert pin legal, the deployer's image/payload gates
-# with the false-green MainPID check, and the prefetch law (expert prefetch
-# measured at 36% precision — OFF is the incumbent; a regression that
-# reintroduces it silently costs ~40-50% decode).
+# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
+# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
+# script drives `podman run` DIRECTLY (the image entrypoint execs
+# `vllm serve`), with every engine flag qualified by boot probe and carried
+# in the repo env file. The seam pins what the flags alone do not say: the
+# KV qualification law (the night ladder: 262144/245760 OOM at graph
+# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
+# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
+# clean cache + 262k windows), the automatic-offload law (the fork plans
+# the tiered-expert offload itself; no flag exists to set), the self-healing
+# law (the run returns before the engine is up, so the serve script must
+# wait on the container for Restart=on-failure to mean anything), the
+# memlock law that makes the pinned host rows legal, the deployer's
+# image/payload gates with the false-green MainPID check, and the measured
+# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
+# no-spec family measured fastest once the prefetch tax was removed; MTP
+# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
+# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
+# tool parser qwen3_xml, preserve_thinking on).

 load helpers

@@ -36,7 +42,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # dank session's own hard limit is 8 MiB and cannot be raised without
    # restarting user@1000.service).
    grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    grep -q '^Restart=on-failure$' "$UNIT"
+    # Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
+    # server exit CLEAN, so ExecStart returns 0 and on-failure never fires
+    # (measured 2026-10-01 — the endpoint stayed down until a human noticed).
+    grep -q '^Restart=always$' "$UNIT"
+    ! grep -q '^Restart=on-failure$' "$UNIT"
+    # Exiting the engine takes as long as vLLM's drain: ExecStop runs
+    # `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
+    # kills the stop and records 'Failed with result timeout' (measured
+    # 2026-10-01). Guard the relationship, not just the literal.
+    stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
+    [ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
+    grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
+    # StartLimitIntervalSec belongs to [Unit]; a copy in [Service] is an
+    # 'Unknown key' warning systemd logs on every load.
+    [ "$(grep -c '^StartLimitIntervalSec=' "$UNIT")" = "1" ]
+    awk '/^\[Service\]/{in_svc=1} /^StartLimitIntervalSec=/{if(in_svc) exit 1}' "$UNIT"
    grep -q '^RestartSec=' "$UNIT"
    grep -q '^User=dank$' "$UNIT"
    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
@@ -69,30 +90,132 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }

@test "the qualified engine shape is in the env file" {
-    # 2026-09-29 boot probe: MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8
-    # fits — the engine measured "GPU KV cache size: 729,545 tokens,
-    # Maximum concurrency for 200,000 tokens per request: 3.65x" at
-    # KV_BYTES 7.2 GiB per rank. KV_BYTES is PER RANK: sizing it to the
-    # global requirement OOMs the worker (measured twice). fp8 KV is the
-    # requested q8/q8 (e4m3 for both K and V together). MTP4 spec decode,
-    # util 0.98 (0.99 fails the free-memory wall).
-    grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
+    # 2026-10-01 night ladder (Frank's method), then a runtime-survivability
+    # correction. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
+    # SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
+    # free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
+    # bytes free at runtime and a real client request (995-token prompt with
+    # max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+    # ... 0 bytes free`. FOUR slots at the full native 262144 window is the
+    # deployed shape: ~1.93 GiB more headroom per rank for
+    # activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
+    # own budget instead of buying slack).
+    # FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
+    # --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
+    # 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
+    # tokens, 3,206 margin). The precise flag drops the plan's overhead
+    # 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
+    grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
+    # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
+    # crash-looped at graph capture. The serve script keeps the opt-in flag.
+    ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }

@test "the serve script wires the engine flags from the env" {
-    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
+    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
+    ! grep -q -- '--speculative-config' "$SERVE"
    grep -q -- '--tensor-parallel-size 2' "$SERVE"
-    grep -q 'num_speculative_tokens' "$SERVE"
    grep -q -- '--enable-prefix-caching' "$SERVE"
    grep -q -- '--trust-remote-code' "$SERVE"
+    # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
+    # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
+    grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
+    ! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
+    # --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
+    # default (7.60 GiB = fitted + compile/graph-capture headroom), which only
+    # moves the plan's `room` term and therefore the expert arena: ~1,025
+    # more slots at 2.344 MiB each.
+    grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
+}
+
+@test "tools and reasoning are wired — the client contract is served" {
+    # 2026-10-01: the rollback-era trio served NO tool dialect and left
+    # thinking inline in `content` with a dangling `</think>` and a null
+    # `reasoning` field (measured: content 'We need answer...Brief.\n</think>
+    # \n\n391') — the malformed thinking block clients choke on. The working
+    # values are the ones ecaad36 shipped for this model and this fork's
+    # registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
+    # preserve_thinking keeps prior thinking blocks across turns.
+    grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
+    grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
+    grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
+    grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
+    grep -q 'preserve_thinking' "$ENV_FILE"
+    grep -q 'enable_thinking' "$ENV_FILE"
+}
+
+@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
+    # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
+    # expert planner moves the overflow to pinned host rows by itself (the
+    # arena 6,574 slots x 2.344 MiB plus host rows). There is no
+    # --enable-expert-offload flag in this arm; the legality of the pin is
+    # the unit's unbounded LimitMEMLOCK.
+    ! grep -q -- '--enable-expert-offload' "$SERVE"
+    ! grep -q -- '--expert-precise-memory' "$SERVE"
+    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
+    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
+}
+
+@test "speculative decode is ABSENT BY MEASUREMENT" {
+    # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
+    # spec family against no-spec at matched geometry: no-spec measured
+    # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
+    # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
+    # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
+    # never completed; until one shows a real win, no spec is the incumbent.
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
+    ! grep -q -- '--speculative-config' "$SERVE"
+}
+
+@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
+    # --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
+    # bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
+    # attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
+    # 1,563. That halves the pool capacity per byte, which is why the contract
+    # is two full-size windows instead of three; in exchange the KV is exact
+    # (no quantization error) and measured +4% decode at shallow depth, where
+    # fp8's dequant cost dominates over the extra bytes read.
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+}
+
+@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
+    # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
+    # sweep's biggest lever, +18.7% decode vs default geometry, and the win
+    # GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
+    # g64x128 is past the peak; g8x16 mild. They are container ENV (the
+    # fork's knobs), not engine flags — passed through with -e.
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+}
+
+@test "MNBT is at Frank's 2048 with the arena intact" {
+    # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
+    # interchangeable on decode AND identical on arena (the sweep measured
+    # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
+    # 2048 is Frank's setting as of 2026-10-01 — it keeps the workspace floor
+    # so the expert arena stays max.
+    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
 }

@test "expert prefetch is absent BY LAW, not by omission" {
@@ -174,6 +297,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
    grep -q 'config.json' "$REMOTE_DEPLOY"
    grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
+    grep -q 'glm-probe' "$REMOTE_DEPLOY"
    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
@@ -1,25 +1,30 @@
 #!/usr/bin/env bats
-# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
-# went back to the halogen engine on the iGPU. This seam pins what the flags
-# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
-# is involved), the repo-relative paths the unit executes, the separate client
-# port, the MTP contract (the head is a separate sidecar file loaded as the
-# draft model, chained behind ngram-mod), the vision handoff from frank-large,
-# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
+# Seam: the frank-small SGLang root unit on dankclaw.
+# frank-small is the RTX 3090 endpoint; since 2026-10-01 it serves
+# letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid,
+# INT8 W8A16 AutoRound, in-checkpoint vision) via SGLang in a
+# python3.12 venv. This seam pins what the flags alone do not say: boot
+# persistence for a NATIVE runtime (no generated quadlet is involved), the
+# repo-relative paths the unit executes, the separate client port, the
+# no-spec-decode gate (the checkpoint ships NO mtp.* tensors, so a NEXTN
+# draft would be an uninitialised ghost model), the 8-shard payload gate,
+# and the KV budget: fp8 KV at 0.98 utilization whose 4-slot grid must fit
+# the card's 24 GiB.

 load helpers

 UNIT="$REPO_ROOT/scripts/frank-small.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
+DOWNLOAD="$REPO_ROOT/scripts/download-frank-small.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"

-@test "frank-small unit, serve script and env exist in scripts/" {
+@test "frank-small unit, serve script, env and download script exist in scripts/" {
    [ -f "$UNIT" ]
    [ -f "$SERVE" ]
    [ -f "$ENV_FILE" ]
+    [ -f "$DOWNLOAD" ]
 }

@test "frank-small is boot-persistent, self-healing and unprivileged" {
@@ -35,64 +40,67 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
    grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }

-@test "frank-small offloads every layer to the 3090 with flash attention" {
-    grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
-    grep -q '^NGL=999$' "$ENV_FILE"
-    grep -q -- '--gpu-layers "$NGL"' "$SERVE"
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
-    grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
-    # Prometheus scrapes this endpoint; without --metrics there is no /metrics.
-    grep -q -- '--metrics' "$SERVE"
+@test "frank-small serves the MiMo-V2.6 checkpoint from the self-contained SGLang venv" {
+    grep -q '^SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang$' "$ENV_FILE"
+    grep -q '^MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound$' "$ENV_FILE"
+    grep -q -- '--model-path "$MODEL_DIR"' "$SERVE"
+    # The AutoRound checkpoint declares its quant method; SGLang must load it
+    # as auto-round, not re-quantize online.
+    grep -q '^QUANTIZATION=auto-round$' "$ENV_FILE"
+    grep -q -- '--quantization "$QUANTIZATION"' "$SERVE"
 }

-@test "frank-small caps its CPU RAM cache below the build default" {
-    # Without the flag the 8192 MiB default returns and the env cap is a
-    # silent no-op; this memory-tight host needs the cap to hold.
-    grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
-}
-
-@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
-    grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
-    grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
-    # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
-    # nextn layers); it loads as --model-draft and shares the target's last
-    # layers' KV. ngram-mod is draftless, so the chain is legal.
-    grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
-    grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
+@test "frank-small runs WITHOUT speculative decoding (no MTP weights shipped)" {
+    # The config declares mtp_num_hidden_layers=1 but the safetensors index
+    # carries zero mtp.* tensors (base and INT8 conversion alike, verified
+    # 2026-10-01). SGLang would draft with an uninitialised 4.24 GiB model
+    # at 0% acceptance; the serve script must not pass any speculative flag.
+    ! grep -q -- '--speculative-' "$SERVE"
+    ! grep -q '^SPEC_' "$ENV_FILE"
 }

@test "frank-small carries the fleet's image path, and only it" {
-    grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
-    grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
+    # The vision tower is baked into the safetensors (qwen3_5_vision): no
+    # separate projector file, and the serve script must not look for one.
+    ! grep -q '^MMPROJ_PATH=' "$ENV_FILE"
+    ! grep -q -- '--mmproj' "$SERVE"
    ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
 }

-@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
-    local parallel cache_k cache_v ctx fixed per_slot
-    parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
-    cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
-    cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
-    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
-    # Every slot must be the model's FULL trained window (262144), not a
-    # divided one -- CTX_SIZE is the total pool and --parallel divides it.
-    [ $((ctx / parallel)) -eq 262144 ]
-    [ $((ctx % parallel)) -eq 0 ]
-    [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
-    # Fit budget from the measured boot arithmetic on the 3090 (24576
-    # MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
-    # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
-    # (weights, projector, sidecar draft, compute buffers, CUDA context)
-    # is 11996-3450 = 8546 MiB. If the config asks for more slots than
-    # the card holds, this inequality fails before deploy instead of the
-    # KV allocation failing at boot.
-    fixed=8546
-    per_slot=3450
-    [ $((fixed + parallel * per_slot)) -le 24576 ]
-    # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
-    # without which prefill collapses about 12x (see the CUDA lesson in
-    # AGENTS.md). The build-cuda tree must carry it.
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
+@test "frank-small's slot grid is fp8 KV and fits the measured card budget" {
+    local ctx running kv ssmdtype mambasize required
+    ctx="$(sed -nE 's/^CONTEXT_LENGTH=([0-9]+)$/\1/p' "$ENV_FILE")"
+    running="$(sed -nE 's/^MAX_RUNNING=([0-9]+)$/\1/p' "$ENV_FILE")"
+    kv="$(sed -nE 's/^KV_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
+    ssmdtype="$(sed -nE 's/^MAMBA_SSM_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
+    mambasize="$(sed -nE 's/^MAX_MAMBA_CACHE_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    [ -n "$ctx" ] && [ -n "$running" ] && [ -n "$kv" ]
+    # fp8 KV is the only dtype whose 16 KiB/token makes four slots fit;
+    # 64 tokens per MiB of pool.
+    case "$kv" in fp8*) ;; *) return 1 ;; esac
+    # The DeltaNet ssm state pool is the hidden fit cost: the engine
+    # autofills mamba slots to spend a fixed ~5.1 GB budget, so dtype
+    # alone does not shrink it — the SLOT COUNT must be pinned (bf16,
+    # bounded). These two gates are what make the measured pool below
+    # exist; change either and re-measure the pool at boot.
+    [ "$ssmdtype" = "bfloat16" ]
+    [ -n "$mambasize" ] && [ "$mambasize" -ge "$running" ] && [ "$mambasize" -le 64 ]
+    grep -q -- '--mamba-ssm-dtype "$MAMBA_SSM_DTYPE"' "$SERVE"
+    grep -q -- '--max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE"' "$SERVE"
+    # The slot window is inside the model's native 262144 window.
+    [ "$ctx" -le 262144 ]
+    # The grid must equal the pinned pool. The engine's autofit picks a
+    # bigger pool (687,388 tokens) but leaves ~0.53 GB — Triton's
+    # deferred kernel loads take most of it and eager-prefill activations
+    # then OOM on the first long request. 606,208 (4 x 151,552) is the
+    # verified ceiling: 170k single request and 4 concurrent 54k requests
+    # run against it (2026-10-01). A weights or mamba-flag change moves
+    # this number; re-measure before trusting the grid.
+    pinned_pool=606208
+    required=$((running * ctx))
+    [ "$required" -le "$pinned_pool" ]
+    grep -q -- '--max-total-tokens "$MAX_TOTAL_TOKENS"' "$SERVE"
+    grep -q -- '--disable-prefill-cuda-graph' "$SERVE"
 }

@test "frank-small runs on its own port beside frank-large's" {
@@ -106,16 +114,21 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
    ! grep -q '^PORT=7771$' "$ENV_LARGE"
 }

-@test "the deploy handler installs the root unit and refuses a missing model" {
+@test "the deploy handler installs the root unit and refuses a missing payload" {
    grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
+    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REMOTE_DEPLOY"
    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
    grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
 }

@test "frank-small's serve script refuses to start on a half-finished download" {
+    # The payload is 8 safetensors shards; the serve script counts them and
+    # the download script pins each one's LFS sha256.
    grep -q 'frank-small: missing' "$SERVE"
+    grep -q 'expected 8 safetensors shards' "$SERVE"
+    grep -q 'model-00001-of-00008.safetensors' "$DOWNLOAD"
+    grep -q 'model-00008-of-00008.safetensors' "$DOWNLOAD"
 }