---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
tags:
  - dankserver
  - r9700
  - gfx1201
  - glm5.3
  - w4a16
  - int4
  - expert-offload
  - expert-cache
  - autoround
  - vllm
timestamp: 2026-09-30
---

# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

> **Question this page answers.** Can the fleet serve the 320B-class
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
> reachable** on this payload as the fork ships it. The bound is **not** the
> model, the quant, the link, or the expert gather (measured: 1.05 % of device
> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
> because the fork compiles its own Qwen trunk and not its GLM one. See
> [The measured profile](#the-measured-profile-and-the-real-wall).

Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
hardware — its conclusion was drawn from throughput arithmetic and is superseded
by this page's kernel-time measurement),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].

## The deployed shape

| term | value |
| --- | --- |
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |

```mermaid
graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|host misses<br/>1.05% of device time| G
```

## Why this payload and not another

The W4A16 fast path on this fork is reachable **only** through the INC
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

```
[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU
```

`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
format-agnostic but requires the experts class to expose the hot/cold split.

## The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):

| depth | prefill | decode |
| --- | --- | --- |
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |

Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.

### The measured profile, and the real wall

**This section replaces an earlier explanation, which the measurement falsified.**
The first reading of this page bounded the step with throughput arithmetic: 42 MoE
layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
the gather is the wall". That arithmetic was circular — it *assumed* the gather
consumed the whole step — and it is wrong. The engine, asked directly, disagrees:

A torch-profiler window (`--profiler-config`, `POST /start_profile` /
`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
device-activity time) gives the true split. Total device kernel time **46.18 s /
496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
the profiler attached):

| ms | % | launches | kernel |
| --- | --- | --- | --- |
| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |

So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
wall clock, so this is real device work, not launch-gap overhead.

Two consequences that kill three whole families of planned work:

- **Spec decode cannot help this shape, and the MTP measurement was not an
  accident.** Pointwise cost scales with the token count, so a draft batch that
  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
  the gather was never the bound.
- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.

## Fork work: the compile changelist (WIP, does not boot yet)

The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
every boot:

```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.
```

Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
its own evidence header, base/patched sha256 and the next blocker:

| patch | target | fixes |
| --- | --- | --- |
| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |

What the boots established, in order (each one ~7 min):

1. **01 alone dies** on the first cold probe call inside the traced region:
   `Dynamo does not know how to trace the builtin 'posix.stat'` —
   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
   wrapped body to build its guard even when the cache is warm (it warns about
   exactly this), verified in-image — a *warm* lru_cache-calling function still
   fails `fullgraph=True`, while a plain module-global read compiles.
2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
   four processes) and die on
   `RuntimeError: Graph breaks are not supported with aot compile`, because this
   build runs AOT compile by default and AOT refuses breaks.
3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
   submodule attribute name, also present in the `load_weights` shard table
   (`model.py:780-785`). Reading the full traceback (the reported line is inside
   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
   from those names at :1225. So the compiler is failing to **resolve a submodule by
   name** while tracing the GLM decoder layer; whether the root cause is the fork's
   module tree or dynamo's registration is not yet settled. The next step is the one
   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
   environment).

That last row is the state of the work: the changelist is one unresolved
submodule-name lookup away from booting, and it must be cleared (plus whatever
follows it) before the compile payoff can be measured. Two verified facts make the
rest of the shape concrete: `suppress_errors` does **not** cover graph breaks (only
`fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because
its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).

### Two structural constraints on any further fork work

- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
  sources. The obvious second lever — the 9 % router — is gated
  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
  (and runtime-JIT Triton) are available.
- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
  window; it only removes the only MLA decode path gfx12 has (trap 1 above).

### DFlash2 for GLM — what the drafter actually is

`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
head_dim 128, sliding attention (window 2048), `is_causal: false`,
`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
`num_target_layers: 45`**.

The fork already maps that architecture:
`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus
capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
implements neither.

**The load was tried, and the gap is exactly one interface.** Payload fetched to
`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
accepts the config and the drafter and dies at worker init with:

```
RuntimeError: Model does not support EAGLE3 interface
```

So nothing about the drafter, the method name, or the payload is wrong — the GLM
trunk simply does not implement the aux-hidden-state contract the dflash runner
requires. The reference is in this same image, twice: `deepseek_v2.py`
(`set_aux_hidden_state_layers` / `get_eagle3_aux_hidden_state_layers` at :1948-1951,
capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
small:

- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
  by the wrapper's `set_aux_hidden_state_layers()`;
- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
  rank holds fewer tokens than `positions`);
- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
  list is non-empty;
- `get_eagle3_aux_hidden_state_layers()` supplies a default triple when the drafter
  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
  the runner derives trunk indices `[6, 15, 25, 34, 43]`).

**The GLM-specific unknown** is what "the single-stream state entering layer k"
means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
a semantic choice to validate by acceptance rate, not by inspection.

**Budget no speedup.** Per the profile above, a verification pass pays pointwise
cost proportional to its token count — the same reason MTP4 measured flat — and
MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
it is small), not as a performance fix, and the honest expectation on this shape is
flat decode with a prefill regression like MTP's until the pointwise mass is fused.

Note also that the KV-cap patch's predecessor arithmetic in
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
"gather-bound" conclusion came from the same throughput arithmetic that this page's
profiler window has now falsified.

## Measured arms (all 2026-09-30, one boot each)

| arm | decode | prefill | verdict |
| --- | --- | --- | --- |
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |

### Why MTP is rejected here (it is not that it fails)

MTP **works** on GLM in this fork: the head loads, the architecture resolves to
`Glm5NextMTPModel`, and acceptance is excellent —

```
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%
```

— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
verification forward gathers the **union** of the batch's experts, so spec
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
of arena. On a compute-bound (fully-resident) shape this would invert; here it
does not.

## Served surface (verified 2026-09-30 against the live engine)

| capability | state | evidence |
| --- | --- | --- |
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
| completion | yes | `bench_glm.py` runs above |
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |

Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).

## Fork traps found (for the engine authors)

1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
   the MTP draft step demands an op that cannot exist here:
   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
   CDNA check, not the env var.)
   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
     more tokens in the same reserve, and this fork's reserve model is
     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
     dominated by the per-sequence KDA state (34 linear-attention layers) and the
     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
     cover — so halving the KV dtype cannot move it.
   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
     **do not spend effort here** — the payoff is capacity, and the reserve is
     dtype-blind.
2. **The plan's KV reserve under-models this model, and the cap enforces the
   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
   `--max-model-len` becomes unbootable:
   `ValueError: To serve at least one request with the model's max seq len
   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
   fixed) — where the reserve would eat a third of the arena.
   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
   request raise the cap; **it is not part of the deployed shape**, which boots
   on the plan's own reserve.
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
   what the reserve model requires for this payload (below ~92k the boot
   refuses); it is not a capacity decision.
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

## Operations

```bash
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health

# experiment windows (production stopped + `systemctl mask --runtime`, which
# self-clears at reboot) ran from the host-side scratch launcher
# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
# word-split by the launcher's unquoted $EXTRA).
bash /home/dank/serve_glm_autoround.sh
```

Boot gate: the engine refuses a legal-looking shape in two measured ways — the
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
what killed the first two boots at util 0.95.

`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
the default allocator's fragmentation fails the profile/graph-capture phase
(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
MiB is free").

The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
walks every guarded vhost — including `http://192.168.0.112:8989/` on
FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
and the WoL packet did not wake it (macOS "Wake for network access" needs
toggling on the mini itself).

## Rollback targets

| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |