serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
tags:
  - dankserver
  - r9700
  - gfx1201
  - glm5.3
  - w4a16
  - int4
  - expert-offload
  - expert-cache
  - autoround
  - vllm
timestamp: 2026-09-30
---

# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

> **Question this page answers.** Can the fleet serve the 320B-class
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
> reachable** with this payload on this hardware. The bound is not the model,
> the quant, or the link: it is the fork's expert-gather read path, and it is
> measured, not inferred.

Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
same hardware, the same gather-path wall, found first on qwen),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].

## The deployed shape

| term | value |
| --- | --- |
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |

```mermaid
graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|8.2 GB/s effective| G
```

## Why this payload and not another

The W4A16 fast path on this fork is reachable **only** through the INC
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

```
[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU
```

`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
format-agnostic but requires the experts class to expose the hot/cold split.

## The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):

| depth | prefill | decode |
| --- | --- | --- |
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |

Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.

### The arithmetic that bounds it

Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.

| hit rate | miss traffic | step @ 8.2 GB/s | decode |
| --- | --- | --- | --- |
| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |

Two independent measurements pin the missed term:

- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
  gives the hit rate directly — the LRU is *working*:
  `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
  read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
  pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
  36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.

So the step is bounded by the fork's gather read path, and the only other lever
is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.

## Measured arms (all 2026-09-30, one boot each)

| arm | decode | prefill | verdict |
| --- | --- | --- | --- |
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |

### Why MTP is rejected here (it is not that it fails)

MTP **works** on GLM in this fork: the head loads, the architecture resolves to
`Glm5NextMTPModel`, and acceptance is excellent —

```
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%
```

— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
verification forward gathers the **union** of the batch's experts, so spec
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
of arena. On a compute-bound (fully-resident) shape this would invert; here it
does not.

## Served surface (verified 2026-09-30 against the live engine)

| capability | state | evidence |
| --- | --- | --- |
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
| completion | yes | `bench_glm.py` runs above |
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |

Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).

## Fork traps found (for the engine authors)

1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model.** The
   `ROCM_AITER_MLA_SPARSE` backend's `_use_rocm_sparse_triton()` requires a
   non-fp8 KV cache, and that rope-free Triton path is the **only** MLA decode
   available on gfx12: aiter's MLA decode op is registered under
   `@if_aiter_supported`, which is gated on **CDNA > 2**
   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
   the MTP draft step demands an op that cannot exist here:
   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
   CDNA check, not the env var.)
2. **The plan's KV reserve under-models this model, and the cap enforces the
   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
   `--max-model-len` becomes unbootable:
   `ValueError: To serve at least one request with the model's max seq len
   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
   fixed) — where the reserve would eat a third of the arena.
   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
   request raise the cap; **it is not part of the deployed shape**, which boots
   on the plan's own reserve.
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
   what the reserve model requires for this payload (below ~92k the boot
   refuses); it is not a capacity decision.
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

## Operations

```bash
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
```

Boot gate: the engine refuses a legal-looking shape in two measured ways — the
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
what killed the first two boots at util 0.95.

`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
the default allocator's fragmentation fails the profile/graph-capture phase
(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
MiB is free").

The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
walks every guarded vhost — including `http://192.168.0.112:8989/` on
FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
and the WoL packet did not wake it (macOS "Wake for network access" needs
toggling on the mini itself).

## Rollback targets

| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |