Frank
f2bcf84e88
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
112 lines
7.1 KiB
Bash
112 lines
7.1 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-09-30 cutover from orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (MTP4) to
|
|
# Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet has served
|
|
# through the fork's W4A16 expert-offload path. ROLLBACK: the pre-cutover qwen
|
|
# state is commit 2339fae ("frank-primary: fix the SIGPIPE race in both
|
|
# image-digest gates") — restore scripts/ from it and deploy; the qwen payload
|
|
# (173 GiB) and the R9V state dirs are untouched on disk.
|
|
#
|
|
# QUALIFIED 2026-09-30 by boot probe + bench against the live engine:
|
|
# * THE 35 tok/s TARGET IS NOT MET: decode 9.9-10.0 tok/s, prefill 918 tok/s
|
|
# (warm, 2.6k-token prompt) at 131,072 ctx x 2 seqs. See the runbook's
|
|
# "Why 35 tok/s is out of reach on this hardware" — the bound is the fork's
|
|
# expert-gather read path (8.2 GB/s effective vs the link's 22.74 GB/s),
|
|
# the same structural gap the 2026-09-29 qwen campaign measured.
|
|
# * Arena 14.51 GiB = 2,383 slots x 6.234 MiB per rank; 12,096 host-backed
|
|
# experts per rank, 73.64 GiB pinned. Live per-layer telemetry
|
|
# (R4D_LRU_TELEMETRY=1, 2026-09-30): hit rate 0.53-0.66, ~500-730 installs
|
|
# vs ~2,000 routed per layer over the window.
|
|
# * --expert-precise-memory is ON: it replaces the 6.25 GiB default
|
|
# compile-headroom figure with the registry's fitted 5.46 GiB, buying
|
|
# 1.43 GiB of cache (13.08 -> 14.51 GiB). Decode measured flat across that
|
|
# change (10.03 -> 9.88 tok/s), which is the evidence that the hit rate
|
|
# saturates near 0.6 and residency is no longer the binding term.
|
|
# * KV dtype stays bf16/AUTO — fp8 KV is NOT legal on this model. It disables
|
|
# _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend (the
|
|
# predicate requires a non-fp8 cache), and that rope-free Triton path is the
|
|
# only MLA decode available on gfx12: aiter's MLA decode op is registered
|
|
# under @if_aiter_supported, i.e. gated on CDNA>2. With fp8 KV the MTP draft
|
|
# step dies with "'_OpNamespace' 'vllm' object has no attribute
|
|
# 'rocm_aiter_mla_decode_fwd'".
|
|
# * MTP4 spec decode is ABSENT DELIBERATELY, unlike the qwen shape. It WORKS
|
|
# here (the MTP head loads, architecture resolves to Glm5NextMTPModel, and
|
|
# acceptance is excellent: mean accepted length 3.76-4.90, per-position
|
|
# 0.92/0.85/0.69/0.58, 69-97%) — but it buys nothing on an offload-bound
|
|
# deployment: measured decode 9.6-11.2 tok/s (vs 10.0 without) while prefill
|
|
# COLLAPSED 918 -> 255 tok/s. The verification forward gathers the union of
|
|
# the batch's experts, so spec decode does not amortize a residency-bound
|
|
# gather the way it amortizes compute. It also costs 2.6 GiB of arena.
|
|
# Reinstate only with a fresh A/B on a non-offload-bound shape.
|
|
# * PREFETCH IS ABSENT BY LAW, unchanged from the qwen campaign: expert-cache
|
|
# telemetry measured the prefetcher at 36% precision and deleting the flag
|
|
# bought +40-51% decode. Never re-add --expert-prefetch without a fresh A/B.
|
|
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
|
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
|
# numerics (repetition loops, early halts, invalid measurements).
|
|
# * KV is NOT capped explicitly: the fork's plan reserve (1.76 GiB at 131,072)
|
|
# already covers what max_model_len needs for this shape. It does NOT for an
|
|
# MTP shape — see the runbook's fork-bug section and
|
|
# scripts/patches/gpu_worker-kv-cap.patch.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak.
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage. Same image as the qwen cutover — the W4A16 path
|
|
# (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM) ships in it.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
# Intel AutoRound W4A16: symmetric int4, group 128, packing "auto_round:auto_gptq".
|
|
# quantization_config.quant_method is "auto-round", which the fork overrides to
|
|
# "inc" (inc.py:364) — that is what routes the MoE to R4dW4a16MoEMethod instead
|
|
# of the compressed-tensors path's Triton experts, which has no offload recipe.
|
|
# 34 numbered shards + model_extra_{conv,tensors}.safetensors, 113,074 tensors.
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound
|
|
FRANK_PRIMARY_MODEL_SHARDS=34
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-glm
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the pre-cutover qwen identity (it holds
|
|
# the same port and GPUs), the retired R9V identity, and the ad hoc experiment
|
|
# identity.
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
|
|
# --- qualified engine shape ---------------------------------------------------
|
|
# 131,072 x 2 seqs. The window is NOT chosen for capacity: the engine's own
|
|
# plan reserve is one max_model_len request, and below ~92k tokens the fork's
|
|
# reserve model (13,600 B/token + a small mamba term) falls short of what the
|
|
# engine actually needs for this model's per-sequence KDA state, which makes the
|
|
# boot refuse ("2.07 GiB needed vs 1.74 GiB available"). 131,072 clears it with
|
|
# the plan's own cap intact. KV cache as built: 138,561 tokens, 1.06x
|
|
# concurrency for a 131,072-token request.
|
|
FRANK_PRIMARY_MAXLEN=131072
|
|
FRANK_PRIMARY_SEQS=2
|
|
# 0.99 fails boot ("Free memory on device cuda:1 ..."); 0.98 leaves the
|
|
# compile/capture headroom the plan models.
|
|
FRANK_PRIMARY_UTIL=0.98
|
|
# Prefill chunk. 2,048 is the qualified value; the expert union a chunk routes
|
|
# to saturates quickly, so the chunk size is an install-amortization knob, not a
|
|
# residency one. Measured 2026-09-30: NBATCH 8,192 boots (arena 12.51 GiB) and
|
|
# leads to the SAME warm prefill (922 vs 918 tok/s) — no reason to pay its
|
|
# 2 GiB of profile buffer.
|
|
FRANK_PRIMARY_NBATCH=2048
|
|
# GLM's own tool-call dialect. Without --enable-auto-tool-choice the endpoint
|
|
# rejects tool_choice "auto" outright ("\"auto\" tool choice requires
|
|
# --enable-auto-tool-choice and --tool-call-parser to be set"), which breaks
|
|
# every tool-using agent on the endpoint. Verified 2026-09-30: a get_weather
|
|
# prompt returns tool_calls with finish_reason "tool_calls".
|
|
FRANK_PRIMARY_TOOLPARSER=glm47
|
|
# GLM's reasoning dialect. Without it the model's thinking leaks into `content`
|
|
# (measured: content began with the answer, then "Let me verify: ... = 391"
|
|
# followed by a bare "</think") and the reasoning field comes back empty. With
|
|
# it, the chain of thought lands in `reasoning` — the vLLM field name the fleet
|
|
# documents (llama.cpp endpoints use reasoning_content).
|
|
FRANK_PRIMARY_REASONINGPARSER=glm47
|