serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

---

type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left). tags:

• dankserver
• r9700
• gfx1201
• glm5.3
• w4a16
• int4
• expert-offload
• expert-cache
• autoround
• vllm timestamp: 2026-09-30

---

frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

Question this page answers. Can the fleet serve the 320B-class GLM-5.3-Flash at int4 on two R9700s with expert offload, and how fast is it? Answer. Yes — it serves, with vision, tool calls and 131,072 ctx. It runs at 9.9-10.0 tok/s decode / 918 tok/s prefill, and 35 tok/s is not reachable on this payload as the fork ships it. The bound is not the model, the quant, the link, or the expert gather (measured: 1.05 % of device time). It is the unfused pointwise glue of an uncompiled trunk — 44 % of a 93 ms per-token budget spread over ~1,428 elementwise launches per token — because the fork compiles its own Qwen trunk and not its GLM one. See The measured profile.

Related: dankserver-qwen38-fp8-expert-offload-tuning (the same fork, the same hardware — its conclusion was drawn from throughput arithmetic and is superseded by this page's kernel-time measurement), dankserver-frontier-moe-feasibility (the earlier fit analysis), VLLM Index, LLM/index.

The deployed shape

term	value
payload	Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4, group 128, packing auto_round:auto_gptq, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB
runtime	docker.io/tcclaviger/vllm@sha256:da4029666… (same digest-pinned image as the qwen cutover)
engine	TP=2, --enable-expert-offload --expert-precise-memory, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching
MoE kernel	R4dW4a16MoEMethod → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot
expert cache	14.51 GiB = 2,383 slots per rank; 12,096 host-backed experts, 73.64 GiB pinned per rank
unit	scripts/frank-primary.service (root unit, LimitMEMLOCK=infinity for the pin)
measured	decode 9.9-10.0 tok/s, prefill 918 tok/s (warm, 2.6k-token prompt), KV cache 138,561 tokens

graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|host misses<br/>1.05% of device time| G

Why this payload and not another

The W4A16 fast path on this fork is reachable only through the INC (AutoRound) quant method. compressed_tensors checkpoints of the same model — for example dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16, 182 GiB — resolve their MoE through the WNA16 oracle to TritonWNA16Experts, which has no offload recipe, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU

R4dW4a16MoEExperts is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split.

The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (bench_glm.py, real prose, greedy):

depth	prefill	decode
2,621 tok prompt, run 1 (cold cache)	259 tok/s	9.88
2,621 tok prompt, runs 2-3 (warm)	918 / 885 tok/s	9.91 / 9.41
493 tok prompt (telemetry window)	124 tok/s	8.90

Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin.

The measured profile, and the real wall

This section replaces an earlier explanation, which the measurement falsified. The first reading of this page bounded the step with throughput arithmetic: 42 MoE layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s = the gather is the wall". That arithmetic was circular — it assumed the gather consumed the whole step — and it is wrong. The engine, asked directly, disagrees:

A torch-profiler window (--profiler-config, POST /start_profile / stop_profile, 496 decode tokens at the deployed shape, rank0 trace parsed for device-activity time) gives the true split. Total device kernel time 46.18 s / 496 tokens = ~93 ms per token against a measured ~104 ms step (9.6 tok/s with the profiler attached):

ms	%	launches	kernel
16,074	34.8	708,500	vectorized_elementwise_kernel (22.7 us each)
4,602	10.0	42,000	r4d_gemm_moe_w4a16_nt_b16_m64 — the MoE GEMM
3,108	6.7	174,708	elementwise_kernel_manual_unroll
2,943	6.4	138,252	wvSplitK_hf_sml_ — split-K GEMM (projections / head)
2,829	6.1	83,160	router: sbtopk::gatherTopK + warpMergeSortTopK + bitonicSortKVInPlace (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below)
3,015	6.5	133,920	r4d_mhc_pre_finish/pre_partial/post — hyper-connections
2,782	6.0	127,916	inductor triton_poi_* / triton_per_* (the fusions that DO happen)
487	1.05	20,664	r4d_moe_lru_gather_kernel — the accused
250	0.54	20,832	_moe_lru_tele_kernel — the telemetry that measured it (TELEM was on)

So the gather is 1.05 % of device time, and r4d_moe_lru_fused_kernel (the LRU manager, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the PCIe link are not the story: 44 % of the budget is pointwise glue — bf16<->fp32 conversions, adds, pow/rsqrt from fp32 RMSNorm, launch_clamp_scale quant passes, reduce_kernel means — spread over ~1,428 elementwise launches per token (3,734 kernel launches per token in total). The GPU is busy 86 % of the wall clock, so this is real device work, not launch-gap overhead.

Two consequences that kill three whole families of planned work:

• Spec decode cannot help this shape, and the MTP measurement was not an accident. Pointwise cost scales with the token count, so a draft batch that verifies 4 tokens pays ~4x the glue for the same per-token output — which is exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason the gather was never the bound.
• Arena size and hit rate are not levers. Confirmed twice: --expert-precise-memory (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.

Fork work: the compile changelist (WIP, does not boot yet)

The fork compiles its own flagship trunk — vllm/models/qwen4_exp/amd/model.py carries @support_torch_compile — and not its GLM trunk. The engine says so on every boot:

Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.

Inductor fusion is therefore the only lever that touches the 44 % pointwise mass. Three patches are staged in scripts/patches/glm53-compile-0*.patch, each carrying its own evidence header, base/patched sha256 and the next blocker:

patch	target	fixes
-01-model-decorator	models/glm5next/common/model.py	decorates Glm5NextModel (env R4D_GLM_COMPILE, default on)
-02-trace-safe-probes	model_executor/kernels/r4d_lib.py	freezes 10 capability probes that do os.path.exists/.so lookups inside the traced forward
-03-fullgraph-optout	compilation/wrapper.py	makes fullgraph=True opt-out (env VLLM_COMPILE_FULLGRAPH=0)

What the boots established, in order (each one ~7 min):

1. 01 alone dies on the first cold probe call inside the traced region: Dynamo does not know how to trace the builtin 'posix.stat' — r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists. functools.lru_cache does not protect these probes: dynamo re-traces the wrapped body to build its guard even when the cache is warm (it warns about exactly this), verified in-image — a warm lru_cache-calling function still fails fullgraph=True, while a plain module-global read compiles.

2. 01+02 get past it (libr4d capability probes frozen: 10 warm [...] in all four processes) and die on RuntimeError: Graph breaks are not supported with aot compile, because this build runs AOT compile by default and AOT refuses breaks.

3. 01+02+03 with VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 traces the trunk successfully (Dynamo bytecode transform time: 1.08 s) and then dies on the decode forward with KeyError: 'in_proj_qkvbfg_a' (Glm5NextModel.forward line 748 calling the decoder layer at line 456). in_proj_qkvbfg_a is the KDA layer's merged column-parallel projection (kda.py:217, called at :407) — a real submodule attribute name, also present in the load_weights shard table (model.py:780-785). Reading the full traceback (the reported line is inside dynamo-rewritten code) traces the throw to dynamo's own name→module map: torch/_dynamo/output_graph.py registers into nn_modules at :1259/:1384, looks up with self.nn_modules[node.target] at :3000 (a bare-name KeyError when absent), and documents stripping _buffers[..]/_parameters[..]/_modules[..] from those names at :1225. So the compiler is failing to resolve a submodule by name while tracing the GLM decoder layer; whether the root cause is the fork's module tree or dynamo's registration is not yet settled. The next step is the one torch itself suggests in that error — TORCHDYNAMO_VERBOSE=1 for the internal stack trace (and note the launcher resets ENVARGS, so it must be added inside serve_glm_autoround.sh's COMPILE_PATCH branch rather than in the outer environment).

4. 04 removes the fused-norm call from the traced region (R4D_GLM_FUSE_QKV_RMSNORM=0: the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel object in models/common/ops/fused_qk_rmsnorm.py is bound after the function that uses it — _FUSED_Q_KV_RMSNORM_KERNEL at line 223 vs its use at line 211). Its necessity is not yet established — see the confound below.

5. A confound worth knowing about, because it cost two boots. Two break-tolerance runs of the same config disagree, and the difference is one logging flag: 01+02+03 with TORCH_LOGS=graph_breaks dies on a fatal Unsupported: Read uninitialized cell at mla.py:208 (the fused-norm call), while the same config without the flag gets past break handling entirely and dies later on the dynamo name-map KeyError. The likely mechanism: graph_breaks logging makes dynamo build an explanation for each break (walk frames, name the reason), and that analysis is what trips on the uninitialized cell — without the flag, the break is taken silently. So that fatal-looking cell error is probably an artifact of the diagnostic. Related: with graph_breaks on, the next reported failure is Unsupported: Unsupported method call at sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135 get_simultaneous / :155 _ensure_workspace_size, i.e. the indexer allocates its shared workspace inside the forward (its own comment: "will allocate on first use") — also flagged under explanation, also not yet shown to block a plain break-tolerant compile.

How to see the breaks (and its trap). TORCH_LOGS=graph_breaks names each break with a user stack, and VLLM_USE_AOT_COMPILE=0 is required because AOT compile hard-refuses breaks — but per the confound above, treat the flagged fatal errors as candidates, and confirm each one against a flag-free run before believing it blocks the boot. For the name-map KeyError there is a ready-made probe: put /home/dank/dynamo-probe on PYTHONPATH with VLLM_DYNAMO_DUMP_ATTRS=1 (a sitecustomize.py — a .pth on PYTHONPATH is NOT executed, and the image's own sitecustomize is chained so its apport hook survives) and it prints, at the miss, the node target, the near-miss keys, and whether another live OutputGraph holds the key.

State of the changelist: the trunk traces (Dynamo bytecode transform time: 1.08 s), and the one failure confirmed by a flag-free run is the dynamo name-map KeyError: 'in_proj_qkvbfg_a'. Two further candidates are queued behind it (the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two verified facts bound the rest of the shape: suppress_errors does not cover graph breaks (only fullgraph=False does), and the GLM trunk can never satisfy fullgraph because its indexer metadata syncs to host (glm5next/amd/sparse_indexer.py:263,419: int(...max().item())) — both are dead fallbacks on the decode path that the fork author already replaced, so they are not the blockers they appear to be at first glance.

Two structural constraints on any further fork work

• No new HIP kernels are possible. The image ships r4d.so as a binary with no sources. The obvious second lever — the 9 % router — is gated num_experts in {128, 256, 512} in router/r4d_route_router.py and the binary contains only r4d_moe_route_kernel<128|256|512> instantiations. GLM has 288 experts, so it falls back to the generic sort-based top-k and cannot be moved to the fused router without rebuilding libr4d. Only Python/config/inductor changes (and runtime-JIT Triton) are available.
• --kv-cache-dtype fp8 is a dead end for capacity too. The plan's KV reserve is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no window; it only removes the only MLA decode path gfx12 has (trap 1 above).

DFlash2 for GLM — what the drafter actually is

incoai/GLM-5.3-Flash-DFlash2 is not a GLM-shaped drafter and needs no MLA/DSA port: its config.json declares architectures: ["DFlash2DraftModel"] and model_type: "qwen3" — 5 dense layers, hidden 4096, 32 heads / 8 kv heads, head_dim 128, sliding attention (window 2048), is_causal: false, use_cache: false, dflash_config = block 8, conv group 16, selector rank 256 / top-k 16, and target_layer_ids: [5, 14, 24, 33, 42] against num_target_layers: 45.

The fork already maps that architecture: registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM"), and the spec method is "dflash" (config/speculative.py:67). So the port is target-side only: the GLM trunk must expose auxiliary hidden states at those five layer indices, the same interface deepseek_v2.py (also MLA+MoE) implements — set_aux_hidden_state_layers() / get_eagle3_aux_hidden_state_layers() plus capture at idx in self.aux_hidden_state_layers in the layer loop. glm5next implements neither.

The load was tried, and the gap is exactly one interface. Payload fetched to /home/dank/models/GLM-5.3-Flash-DFlash2 and hash-verified (model.safetensors 2,342,169,800 B, sha256 b038e1d9… = the Hub's LFS oid). Booting the engine with --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}' accepts the config and the drafter and dies at worker init with:

RuntimeError: Model does not support EAGLE3 interface

So nothing about the drafter, the method name, or the payload is wrong — the GLM trunk simply does not implement the aux-hidden-state contract the dflash runner requires. The reference is in this same image, twice: deepseek_v2.py (set_aux_hidden_state_layers / get_eagle3_aux_hidden_state_layers at :1948-1951, capture at :1554 and :1579) and the fork's own qwen4_exp trunk. The contract is small:

• self.aux_hidden_state_layers: tuple[int, ...] on the trunk, default (), set by the wrapper's set_aux_hidden_state_layers();
• in the layer loop, if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>) (DeepSeek appends hidden_states + residual, with a TP all-gather fixup when the rank holds fewer tokens than positions);
• Glm5NextModel.forward returns (hidden_states, aux_hidden_states) when the list is non-empty;
• get_eagle3_aux_hidden_state_layers() supplies a default triple when the drafter config omits target_layer_ids (this drafter provides [5, 14, 24, 33, 42], so the runner derives trunk indices [6, 15, 25, 34, 43]).

The GLM-specific unknown is what "the single-stream state entering layer k" means for a trunk whose layers carry mHC multi-stream state: DeepSeek's (hidden_states, residual) pair has no counterpart here — GLM's decoder layer returns (hidden_states, residual, post, comb) with hc_expand/hc_pre mixing hc_mult streams, so the plausible answer is the pre-mHC single-stream input x (4096-wide, matching the drafter's hidden_size), not a post-layer value. That is a semantic choice to validate by acceptance rate, not by inspection.

Budget no speedup. Per the profile above, a verification pass pays pointwise cost proportional to its token count — the same reason MTP4 measured flat — and MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as enabling work (it is the prerequisite for any spec decode on this endpoint, and it is small), not as a performance fix, and the honest expectation on this shape is flat decode with a prefill regression like MTP's until the pointwise mass is fused.

Note also that the KV-cap patch's predecessor arithmetic in dankserver-qwen38-fp8-expert-offload-tuning should be read the same way: its "gather-bound" conclusion came from the same throughput arithmetic that this page's profiler window has now falsified.

Measured arms (all 2026-09-30, one boot each)

arm	decode	prefill	verdict
baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048	10.03 / 10.08 / 9.41	877 / 878 / 885	qualified incumbent
+ --expert-precise-memory (arena 13.08 → 14.51 GiB)	9.88 / 9.91	918	neutral — proves residency is no longer the binding term
+ MTP4 spec decode (draft loads, acceptance excellent)	9.64-11.16	255	rejected — see below
+ MTP4 + explicit --kv-cache-memory-bytes (patched cap)	boots, 151,767-token KV	—	enabler for the MTP arm only

Why MTP is rejected here (it is not that it fails)

MTP works on GLM in this fork: the head loads, the architecture resolves to Glm5NextMTPModel, and acceptance is excellent —

SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%

— and it buys nothing: decode 9.6-11.2 tok/s against 10.0 without it, while prefill collapses 918 → 255 tok/s. The reason is structural to offload: a verification forward gathers the union of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not.

Served surface (verified 2026-09-30 against the live engine)

capability	state	evidence
/v1/models	glm53-flash, max_model_len 131,072	model list probe
completion	yes	bench_glm.py runs above
tool calls	yes — needs --enable-auto-tool-choice --tool-call-parser glm47	a get_weather prompt returns tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}], finish_reason: tool_calls
vision	yes	a 64x64 solid-red PNG is answered "The image appears to be a solid red color…"
reasoning channel	yes — needs --reasoning-parser glm47	without it the thinking leaks into content (measured: content began with the answer, then "Let me verify: … = 391", then a bare </think) with an empty reasoning field; with it, a train-time prompt answers in content ("Step 1: Add the hours. 14:20 + 3 hours = 17:20 …") and the chain of thought lands in reasoning — the vLLM field name the fleet documents (llama.cpp endpoints use reasoning_content)

Without the tool flags the engine answers HTTP 400 — "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set — which breaks every tool-using agent on the endpoint. Both parsers are glm47, the newest GLM dialect the fork ships (glm45/glm47 exist for both).

Fork traps found (for the engine authors)

1. --kv-cache-dtype fp8 is illegal on gfx12 with this model — and it buys nothing even if fixed. The ROCM_AITER_MLA_SPARSE backend's _use_rocm_sparse_triton() requires a non-fp8 KV cache, and that rope-free Triton path is the only MLA decode available on gfx12: aiter's MLA decode op is registered under @if_aiter_supported, which is gated on CDNA > 2 (is_aiter_found_and_supported() → get_cdna_version() > 2). With fp8 KV the MTP draft step demands an op that cannot exist here: RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''. KV dtype therefore stays auto/bf16. (VLLM_ROCM_USE_AITER_MLA=1 does not help — the umbrella gate is the CDNA check, not the env var.)
   • The capacity payoff is measured at zero. The point of fp8 KV is fitting more tokens in the same reserve, and this fork's reserve model is dtype-blind: the fp8 boot printed the identical plan line as bf16 — KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages). The reserve is dominated by the per-sequence KDA state (34 linear-attention layers) and the DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely cover — so halving the KV dtype cannot move it.
   • What a fix would take, if it were ever wanted: a new fp8 path in the rope-free Triton kernels (rocm_sparse_attn_prefill / rocm_sparse_attn_decode in v1/attention/ops/rocm_aiter_mla_sparse.py) that dequantizes pages in-kernel, plus loosening _use_rocm_sparse_triton(). The fork's _r4d_fp8_paged_mqa_logits shows the author already has an fp8 MQA-logits path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30: do not spend effort here — the payoff is capacity, and the reserve is dtype-blind.
2. The plan's KV reserve under-models this model, and the cap enforces the mistake. The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. gpu_worker then caps KV at that reserve, so a legal --max-model-len becomes unbootable: ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB)). Raising max_model_len only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. scripts/patches/gpu_worker-kv-cap.patch lets an explicit operator KV request raise the cap; it is not part of the deployed shape, which boots on the plan's own reserve.
3. A window is forced, not chosen. The production max_model_len 131072 is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision.
4. Expert prefetch remains poison — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

Operations

# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health

# experiment windows (production stopped + `systemctl mask --runtime`, which
# self-clears at reboot) ran from the host-side scratch launcher
# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
# word-split by the launcher's unquoted $EXTRA).
bash /home/dank/serve_glm_autoround.sh

Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (TritonMLA indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is part of the qualified shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation fails the profile/graph-capture phase (measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00 MiB is free").

The deploy.sh e2e gate also drives the admin panel's open-links spec, which walks every guarded vhost — including http://192.168.0.112:8989/ on FrankBook-Pro. When that Mac mini is asleep the spec fails on a page.goto timeout and the deploy reports DEPLOY INCOMPLETE even though the engine is healthy; wake the host (magic packet to de:6b:6d:c1:52:23) or re-run tests/e2e/run.sh once it is up. Seen 2026-09-30: 32 passed, this one failed, and the WoL packet did not wake it (macOS "Wake for network access" needs toggling on the mini itself).

Rollback targets

state	where
qwen FP8 + MTP4 (2026-09-29 → 09-30)	commit 2339fae, payload /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
R9V radiance Qwen3.8-Flash-Next Q8_0	frank-primary-r9v-radiance-qwen38-q8, state dirs /home/dank/r9v-state/{q8xl,uncensored}