dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)

Browse Source 
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).

Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.

Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.

Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).

Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.

Rollback: 2339fae (qwen FP8 + MTP4).
This commit is contained in:
Frank
2026-09-30 04:22:11 -07:00
parent 2339faeaa4
commit f2bcf84e88
12 changed files with 668 additions and 224 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
+224
View File
@@ -0,0 +1,224 @@
---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
tags:
- dankserver
- r9700
- gfx1201
- glm5.3
- w4a16
- int4
- expert-offload
- expert-cache
- autoround
- vllm
timestamp: 2026-09-30
---
# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
> **Question this page answers.** Can the fleet serve the 320B-class
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
> reachable** with this payload on this hardware. The bound is not the model,
> the quant, or the link: it is the fork's expert-gather read path, and it is
> measured, not inferred.
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
same hardware, the same gather-path wall, found first on qwen),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].
## The deployed shape
| term | value |
| --- | --- |
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
```mermaid
graph LR
A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
D --> F[73.64 GiB pinned host rows / rank]
E --> G[decode step]
F -->|8.2 GB/s effective| G
```
## Why this payload and not another
The W4A16 fast path on this fork is reachable **only** through the INC
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
The chain that makes Intel's checkpoint work, with the exact evidence:
```
[expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
host 73.64 GiB pinned, host rows prepacked on the GPU
```
`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
format-agnostic but requires the experts class to expose the hot/cold split.
## The numbers, and the wall
Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
| depth | prefill | decode |
| --- | --- | --- |
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.
### The arithmetic that bounds it
Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.
| hit rate | miss traffic | step @ 8.2 GB/s | decode |
| --- | --- | --- | --- |
| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |
Two independent measurements pin the missed term:
- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
gives the hit rate directly — the LRU is *working*:
`layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.
So the step is bounded by the fork's gather read path, and the only other lever
is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.
## Measured arms (all 2026-09-30, one boot each)
| arm | decode | prefill | verdict |
| --- | --- | --- | --- |
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
### Why MTP is rejected here (it is not that it fails)
MTP **works** on GLM in this fork: the head loads, the architecture resolves to
`Glm5NextMTPModel`, and acceptance is excellent —
```
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%
```
— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
verification forward gathers the **union** of the batch's experts, so spec
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
of arena. On a compute-bound (fully-resident) shape this would invert; here it
does not.
## Served surface (verified 2026-09-30 against the live engine)
| capability | state | evidence |
| --- | --- | --- |
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
| completion | yes | `bench_glm.py` runs above |
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
## Fork traps found (for the engine authors)
1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model.** The
`ROCM_AITER_MLA_SPARSE` backend's `_use_rocm_sparse_triton()` requires a
non-fp8 KV cache, and that rope-free Triton path is the **only** MLA decode
available on gfx12: aiter's MLA decode op is registered under
`@if_aiter_supported`, which is gated on **CDNA > 2**
(`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
the MTP draft step demands an op that cannot exist here:
`RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
(`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
CDNA check, not the env var.)
2. **The plan's KV reserve under-models this model, and the cap enforces the
mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
`--max-model-len` becomes unbootable:
`ValueError: To serve at least one request with the model's max seq len
(131072), (2.07 GiB KV cache is needed, which is larger than the available KV
cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
fixed) — where the reserve would eat a third of the arena.
`scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
request raise the cap; **it is not part of the deployed shape**, which boots
on the plan's own reserve.
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
what the reserve model requires for this payload (below ~92k the boot
refuses); it is not a capacity decision.
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
(36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
## Operations
```bash
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary
# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only
# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
```
Boot gate: the engine refuses a legal-looking shape in two measured ways — the
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
what killed the first two boots at util 0.95.
## Rollback targets
| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
serverAppConfigs-docs/LLM/index.md
+33 -24
View File
@@ -1,10 +1,13 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-09-27):
**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
Dankclaw runs two llama.cpp endpoints —
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
@@ -22,11 +25,11 @@ graph LR
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[R9V radiance-vLLM fork, docker-pinned
via docker-podman shim
frank-primary, port 7770]
C[Qwen3.8-Flash-Next Q8_0
TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
podman run, frank-primary, port 7770]
C[GLM-5.3-Flash W4A16 int4
TP=2, expert offload 73.64 GiB/rank pinned,
14.51 GiB LRU arena, 131072 ctx, vision, tools]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -58,20 +61,26 @@ frank-small, port 7771]
## Active Deployment
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
runbook (2026-09-27, root systemd unit, port 7770): the R9V
radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
image bundle `2dac17a2`, driven through a docker→podman shim) serving
the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
(80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
`reasoning`, parser `qwen3_coder`. The tuned shape lives in the
on-host q8xl setup state. Rollback:
[[frank-primary-r9v-radiance-qwen38-uncensored]]
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; payload and R9V state dirs intact on disk, git tag
frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(same-day swap to the Q8_0 payload; rollback target, state + payload
intact on disk): the R9V radiance-vLLM fork serving
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +