Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
799 lines
49 KiB
Markdown
799 lines
49 KiB
Markdown
---
|
||
type: runbook
|
||
title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
|
||
tags:
|
||
- dankserver
|
||
- r9700
|
||
- gfx1201
|
||
- glm5.3
|
||
- w4a16
|
||
- int4
|
||
- expert-offload
|
||
- expert-cache
|
||
- autoround
|
||
- vllm
|
||
timestamp: 2026-09-30
|
||
---
|
||
|
||
# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||
|
||
> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
|
||
> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
|
||
> **failure**: after five patches the drafter served and drafted (12.8 tok/s
|
||
> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
|
||
> **acceptance pinned at 1.00-1.38 on every text class including verbatim
|
||
> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
|
||
> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
|
||
> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
|
||
> untested alternative capture semantics: post-layer output state
|
||
> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
|
||
> (170 GiB) + drafter (2.2 GiB) staged to
|
||
> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
|
||
> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
|
||
> generic unifier/tailpin/grouping fixes, apply-verified against the image.
|
||
|
||
|
||
> **Question this page answers.** Can the fleet serve the 320B-class
|
||
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
|
||
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
|
||
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
|
||
> reachable** on this payload as the fork ships it. The bound is **not** the
|
||
> model, the quant, the link, or the expert gather (measured: 1.05 % of device
|
||
> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
|
||
> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
|
||
> because the fork compiles its own Qwen trunk and not its GLM one. See
|
||
> [The measured profile](#the-measured-profile-and-the-real-wall).
|
||
|
||
## DFlash2 target-side port — the four laws and the KV ladder
|
||
|
||
Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
|
||
/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
|
||
|
||
**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
|
||
|
||
1. **Patch the instance the runner calls.** `model_runner.py:425` calls
|
||
`set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
|
||
GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
|
||
`self.language_model = Glm5NextForCausalLM`). Members placed only on the
|
||
CausalLM fail the check on the conditional — same error text, wrong file.
|
||
Patch BOTH wrappers.
|
||
2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
|
||
`@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
|
||
(including base-protocol members) with `getattr_static`, breaking on the
|
||
first miss. Both wrappers need, as class attributes:
|
||
`supports_eagle3: ClassVar[Literal[True]] = True`,
|
||
`has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
|
||
Verify in-image with stock typing — no boot needed.
|
||
3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
|
||
Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
|
||
"Cannot create a consistent method resolution order" (sim-proven: the
|
||
ordering rejects it in every base position). The attribute route avoids
|
||
the MRO, so the two methods must be written by hand: the CausalLM
|
||
unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
|
||
delegates to `self.language_model` (pixtral.py:498 pattern);
|
||
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
|
||
with the drafter config's `target_layer_ids` winning when present.
|
||
4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
|
||
— GLM's loop carries the single-stream state entering the layer.
|
||
|
||
**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
|
||
|
||
1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
|
||
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
|
||
whole token rows) but CAN grow block_size when the max page is an exact
|
||
multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
|
||
the `lcm` of all MLA-family pages first (patch 02).
|
||
3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
|
||
divisible branch grew KpoolTailSpec's block; its kernel pins block ==
|
||
index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
|
||
instead of growing (patch 03).
|
||
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
|
||
GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
|
||
exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
|
||
`_get_kv_cache_groups_glm5_next` to partition
|
||
HiddenStateCacheSpec + drafter attention specs; the drafter forms its
|
||
own uniform group (per-group pages, NO unification → the MLA block
|
||
stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
|
||
`block_size % 512 == 0` unstressed); layout returns a 10-tuple
|
||
(+draft_group, +hidden_groups); the builder sizes ONE shared backing
|
||
allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
|
||
aliased pool + appended draft/hidden tensors (patch 04). After 04 the
|
||
capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
|
||
the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
|
||
graphs on both TP ranks).
|
||
5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
|
||
KV) + 5×hidden×2 (aux hidden cache) keyed on
|
||
`speculative_config.method in {dflash, dspark, extract_hidden_states}`.
|
||
NEVER key on `use_aux_hidden_state_outputs` — it lives on the
|
||
gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
|
||
time, so the check silently fails and the reserve stays 13,600
|
||
(patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
|
||
5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
|
||
|
||
**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
|
||
torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
|
||
register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
|
||
`from __future__ import annotations`, and run synthetic GLM+draft+hidden
|
||
spec sets through the patched grouping/layout/builder functions. A 7-minute
|
||
boot per hypothesis is the alternative.
|
||
|
||
**Window discipline:** stop + mask `frank-archon` FIRST and gate on
|
||
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
|
||
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
|
||
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
|
||
to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
|
||
under pipefail via SIGPIPE before log-save, losing the capture. The abort
|
||
path must restore prod (unmask + daemon-reload + reset-failed + start),
|
||
never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
|
||
the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
|
||
"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
|
||
echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
|
||
Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
|
||
torch.compile is the frontier (~380 s).
|
||
|
||
**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
|
||
with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
|
||
prefill. Reconcile with a clean single-request measurement before trusting
|
||
either.
|
||
|
||
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
|
||
hardware — its conclusion was drawn from throughput arithmetic and is superseded
|
||
by this page's kernel-time measurement),
|
||
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
|
||
[[VLLM Index]], [[LLM/index|LLM fleet]].
|
||
|
||
## The deployed shape
|
||
|
||
| term | value |
|
||
| --- | --- |
|
||
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
|
||
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
|
||
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
|
||
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
|
||
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
|
||
| unit | `scripts/frank-archon.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
|
||
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
|
||
|
||
```mermaid
|
||
graph LR
|
||
A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
|
||
B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
|
||
C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
|
||
D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
|
||
D --> F[73.64 GiB pinned host rows / rank]
|
||
E --> G[decode step]
|
||
F -->|host misses<br/>1.05% of device time| G
|
||
```
|
||
|
||
## Why this payload and not another
|
||
|
||
The W4A16 fast path on this fork is reachable **only** through the INC
|
||
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
|
||
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
|
||
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
|
||
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
|
||
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
|
||
|
||
The chain that makes Intel's checkpoint work, with the exact evidence:
|
||
|
||
```
|
||
[expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round"
|
||
[inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
|
||
[r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
|
||
[expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
|
||
[expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
|
||
host 73.64 GiB pinned, host rows prepacked on the GPU
|
||
```
|
||
|
||
`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
|
||
format-agnostic but requires the experts class to expose the hot/cold split.
|
||
|
||
## The numbers, and the wall
|
||
|
||
Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
|
||
|
||
| depth | prefill | decode |
|
||
| --- | --- | --- |
|
||
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
|
||
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
|
||
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
|
||
|
||
Decode is flat across context and across every arena change — the signature of a
|
||
per-step cost that is not KV- or residency-driven at the margin.
|
||
|
||
### The measured profile, and the real wall
|
||
|
||
**This section replaces an earlier explanation, which the measurement falsified.**
|
||
The first reading of this page bounded the step with throughput arithmetic: 42 MoE
|
||
layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
|
||
missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
|
||
the gather is the wall". That arithmetic was circular — it *assumed* the gather
|
||
consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
|
||
|
||
A torch-profiler window (`--profiler-config`, `POST /start_profile` /
|
||
`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
|
||
device-activity time) gives the true split. Total device kernel time **46.18 s /
|
||
496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
|
||
the profiler attached):
|
||
|
||
| ms | % | launches | kernel |
|
||
| --- | --- | --- | --- |
|
||
| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
|
||
| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
|
||
| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
|
||
| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
|
||
| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
|
||
| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
|
||
| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
|
||
| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
|
||
| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
|
||
|
||
So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
|
||
LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
|
||
PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
|
||
conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
|
||
passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
|
||
token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
|
||
wall clock, so this is real device work, not launch-gap overhead.
|
||
|
||
Two consequences that kill three whole families of planned work:
|
||
|
||
- **Spec decode cannot help this shape, and the MTP measurement was not an
|
||
accident.** Pointwise cost scales with the token count, so a draft batch that
|
||
verifies 4 tokens pays ~4x the glue for the same per-token output — which is
|
||
exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
|
||
the gather was never the bound.
|
||
- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
|
||
(arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
|
||
|
||
## Fork work: the compile changelist (WIP, does not boot yet)
|
||
|
||
The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
|
||
carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
|
||
every boot:
|
||
|
||
```
|
||
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
|
||
provides neither a compiled submodule nor breakable CUDA graphs.
|
||
Overriding to FULL_DECODE_ONLY.
|
||
```
|
||
|
||
Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
|
||
Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
|
||
its own evidence header, base/patched sha256 and the next blocker:
|
||
|
||
| patch | target | fixes |
|
||
| --- | --- | --- |
|
||
| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
|
||
| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
|
||
| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
|
||
|
||
What the boots established, in order (each one ~7 min):
|
||
|
||
1. **01 alone dies** on the first cold probe call inside the traced region:
|
||
`Dynamo does not know how to trace the builtin 'posix.stat'` —
|
||
`r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
|
||
**`functools.lru_cache` does not protect these probes**: dynamo re-traces the
|
||
wrapped body to build its guard even when the cache is warm (it warns about
|
||
exactly this), verified in-image — a *warm* lru_cache-calling function still
|
||
fails `fullgraph=True`, while a plain module-global read compiles.
|
||
2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
|
||
four processes) and die on
|
||
`RuntimeError: Graph breaks are not supported with aot compile`, because this
|
||
build runs AOT compile by default and AOT refuses breaks.
|
||
3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
|
||
trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
|
||
the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
|
||
line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
|
||
layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
|
||
submodule attribute name, also present in the `load_weights` shard table
|
||
(`model.py:780-785`). Reading the full traceback (the reported line is inside
|
||
dynamo-rewritten code) traces the throw to dynamo's own name→module map:
|
||
`torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
|
||
looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
|
||
when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
|
||
from those names at :1225. So the compiler is failing to **resolve a submodule by
|
||
name** while tracing the GLM decoder layer; whether the root cause is the fork's
|
||
module tree or dynamo's registration is not yet settled. The next step is the one
|
||
torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
|
||
stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
|
||
`serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
|
||
environment).
|
||
|
||
4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
|
||
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
|
||
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
|
||
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
|
||
**It removes a real break** (under `fullgraph=True` the fused call was the first
|
||
break dynamo reported; after this patch the report moves on to the workspace
|
||
acquisition) — but it is not by itself what blocks a boot, because with breaks
|
||
genuinely tolerated the blocker is the name-map `KeyError`. See the run history
|
||
below.
|
||
5. **A launcher bug that made four boots misleading — worth knowing because the
|
||
failure kept moving.** The scratch launcher's compile branches were restructured
|
||
on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
|
||
consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
|
||
True. With `fullgraph=True` **every** break is an error, so those boots reported
|
||
whichever break came first:
|
||
|
||
| config | outcome |
|
||
| --- | --- |
|
||
| `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
|
||
| `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
|
||
|
||
So the reported breaks are real *as breaks*, but they are not what blocks a
|
||
break-tolerant compile — and an earlier version of this page blamed
|
||
`TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
|
||
logs. The launcher sets the env var again.
|
||
|
||
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
|
||
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
|
||
hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
|
||
check which configuration you are actually in (`podman exec <ctr> printenv
|
||
VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
|
||
KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
|
||
with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
|
||
NOT executed, and the image's own `sitecustomize` is chained so its apport hook
|
||
survives) and it prints, at the miss, the node target, the near-miss keys, and
|
||
whether another live `OutputGraph` holds the key.
|
||
|
||
**State of the changelist (2026-09-30, latest).** The trunk **traces**
|
||
(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
|
||
`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
|
||
|
||
- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
|
||
is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
|
||
→ eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
|
||
_call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
|
||
adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
|
||
frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
|
||
is how inlined frames surface).
|
||
- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
|
||
`/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
|
||
`VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
|
||
with the probe confirmed installed in both the API-server and worker processes.
|
||
- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
|
||
whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
|
||
the `load_weights` shard table (`model.py:780-785`). The four
|
||
`proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
|
||
exactly this bare key, but every one of them is guarded (`if proj_name not in
|
||
fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
|
||
keyed by class name, not module path.
|
||
- It reproduces on **every** run of this configuration (5 of 5), and it is
|
||
independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
|
||
that were never fatal in the first place — the KeyError is the only thing standing
|
||
between this changelist and a boot.
|
||
|
||
**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
|
||
`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
|
||
frames with their source), or wrap the inlining candidates in that forward — the mHC
|
||
CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
|
||
indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
|
||
is a dynamo-side name-resolution failure and not a fixable fork bug.
|
||
|
||
Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
|
||
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
|
||
`fullgraph` because its indexer metadata syncs to host
|
||
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
|
||
fallbacks on the decode path that the fork author already replaced, so they are not
|
||
the blockers they appear to be at first glance.
|
||
|
||
### Two structural constraints on any further fork work
|
||
|
||
- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
|
||
sources. The obvious second lever — the 9 % router — is gated
|
||
`num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
|
||
contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
|
||
experts, so it falls back to the generic sort-based top-k and **cannot be moved to
|
||
the fused router without rebuilding libr4d**. Only Python/config/inductor changes
|
||
(and runtime-JIT Triton) are available.
|
||
- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
|
||
is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
|
||
window; it only removes the only MLA decode path gfx12 has (trap 1 above).
|
||
|
||
### DFlash2 for GLM — what the drafter actually is
|
||
|
||
`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
|
||
port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
|
||
`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
|
||
head_dim 128, sliding attention (window 2048), `is_causal: false`,
|
||
`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
|
||
top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
|
||
`num_target_layers: 45`**.
|
||
|
||
The fork already maps that architecture:
|
||
`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
|
||
and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
|
||
**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
|
||
layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
|
||
`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
|
||
capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
|
||
implements neither.
|
||
|
||
**The load was tried, and the gap is exactly one interface.** Payload fetched to
|
||
`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
|
||
2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
|
||
`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
|
||
accepts the config and the drafter and dies at worker init with:
|
||
|
||
```
|
||
RuntimeError: Model does not support EAGLE3 interface
|
||
```
|
||
|
||
So nothing about the drafter, the method name, or the payload is wrong — the GLM
|
||
trunk simply does not implement the aux-hidden-state contract the dflash runner
|
||
requires. The reference is in this same image, twice: `deepseek_v2.py`
|
||
(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
|
||
capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
|
||
small:
|
||
|
||
- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
|
||
by the wrapper's `set_aux_hidden_state_layers()`;
|
||
- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
|
||
(DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
|
||
rank holds fewer tokens than `positions`);
|
||
- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
|
||
list is non-empty;
|
||
- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
|
||
config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
|
||
the runner derives trunk indices `[6, 15, 25, 34, 43]`).
|
||
|
||
**The GLM-specific unknown** is what "the single-stream state entering layer k"
|
||
means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
|
||
`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
|
||
returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
|
||
`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
|
||
(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
|
||
a semantic choice to validate by acceptance rate, not by inspection.
|
||
|
||
**Budget no speedup.** Per the profile above, a verification pass pays pointwise
|
||
cost proportional to its token count — the same reason MTP4 measured flat — and
|
||
MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
|
||
*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
|
||
it is small), not as a performance fix, and the honest expectation on this shape is
|
||
flat decode with a prefill regression like MTP's until the pointwise mass is fused.
|
||
|
||
**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
|
||
independent of the compile changelist — it patches the same file as compile-01, so
|
||
compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
|
||
implements exactly the contract read out of this image: `EagleModelMixin` on
|
||
`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
|
||
layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
|
||
the three `SupportsEagle3` member attributes plus both interface methods on the
|
||
wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
|
||
`v1/worker/gpu/cudagraph_utils.py:621`).
|
||
|
||
**The patch went through five versions, and the decisive discovery was where the
|
||
instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
|
||
is called on the model the runner LOADED — and for GLM-5.3-Flash that is
|
||
`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
|
||
which holds the text model as `self.language_model = Glm5NextForCausalLM`
|
||
(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
|
||
with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
|
||
both wrappers** and is the first version to clear the gate (both TP workers log
|
||
`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
|
||
|
||
The full evolution, each an instructive trap:
|
||
|
||
1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
|
||
are not the whole contract.
|
||
2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
|
||
`mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
|
||
resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
|
||
Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
|
||
cannot take `SupportsEagle3` in *any* position.
|
||
3. **v3** declared the members as class attributes (MRO-safe) — but only on
|
||
`Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
|
||
4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
|
||
`has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
|
||
`__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
|
||
experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
|
||
5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
|
||
(`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
|
||
`Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
|
||
`pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
|
||
|
||
**The lesson generalizes twice over**: when a runtime error names an interface, read the
|
||
interface's declaration first — and establish WHICH INSTANCE the check runs against
|
||
before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
|
||
models, not pass-throughs, and the text model behind `self.language_model` is invisible
|
||
to a check run on the wrapper.
|
||
|
||
**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
|
||
`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
|
||
torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
|
||
hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
|
||
instead — it needs no startup hook.
|
||
|
||
Note also that the KV-cap patch's predecessor arithmetic in
|
||
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
|
||
"gather-bound" conclusion came from the same throughput arithmetic that this page's
|
||
profiler window has now falsified.
|
||
|
||
### The KV-cache grouping journey: clearing the unifier, then the kpool tail
|
||
|
||
With patch 01 v5 the gate passes and the boot moves to KV-cache layout
|
||
construction — where the next two blockers live, both in
|
||
`v1/core/kv_cache_utils.py`:
|
||
|
||
1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
|
||
spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
|
||
entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
|
||
specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
|
||
`unify_kv_cache_spec_page_size` raises
|
||
`NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
|
||
MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
|
||
they CAN grow `block_size` when the max page is an exact multiple. The
|
||
indexer's page is just `tokens_per_state x` the MLA page, so
|
||
**`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
|
||
to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
|
||
layers then take the legal divisible block-growth branch, drafter + mamba
|
||
layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
|
||
passes, torch.compile passes, and the boot reaches cudagraph capture — the
|
||
NotImplementedError is gone.
|
||
2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
|
||
`kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
|
||
`assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
|
||
`[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
|
||
`Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
|
||
page (8192 B) *divided* the raised max page, so the unifier's divisible branch
|
||
grew its `block_size` by the ratio and the view became
|
||
`[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
|
||
promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
|
||
so `disable_hybrid_kv_cache_manager` is unset and
|
||
`unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
|
||
(= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
|
||
growing its block) **verified 2026-09-30**: the assert is gone and the boot
|
||
captures the DFlash2 drafter's FULL cuda graphs on both TP workers
|
||
(`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
|
||
|
||
3. **The capacity wall: the generic path cannot alias the mamba states.**
|
||
Past capture, the boot dies at `_check_enough_kv_cache_memory`:
|
||
`Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
|
||
65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
|
||
per-token KV). Two coupled causes, read out of the image:
|
||
- **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
|
||
(`get_kv_cache_config_from_groups` places every mamba layer at the MLA
|
||
offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
|
||
expert plan's KV registry is calibrated at 13,600 B/token). On the
|
||
generic path the mamba groups claim their OWN blocks, so ~30 GiB of
|
||
unaliased state pages appears.
|
||
- **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
|
||
a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
|
||
KV at the plan's reserve, so even a correct generic-path estimate would
|
||
be starved.
|
||
|
||
Learned en route: the engine AUTO-SETS the attention block size to
|
||
**2,432 tokens** (`interface.py:918: Setting attention block size to 2432
|
||
tokens to ensure that attention page size is >= mamba page size`), so every
|
||
attention group's block counts scale from that floor — a 16-token drafter
|
||
block would need 4x the shared `num_blocks`; with the global 2,432 all
|
||
groups align.
|
||
|
||
**`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
|
||
for the registry) keeps the aliased glm5 pool and extends it: the grouping
|
||
partitions the hidden-state specs (`HiddenStateCacheSpec` is an
|
||
`MLAAttentionSpec` subclass — excluded before the exact-type
|
||
`type(spec) is MLAAttentionSpec` partition) and the drafter attention
|
||
specs; the drafter layers form their own uniform group (per-group pages,
|
||
NO unification — the MLA block size stays at the engine-set 2,432 and the
|
||
kpool guard `block_size % 512 == 0` is never stressed); the layout returns
|
||
a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
|
||
shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
|
||
equal) as aliased pool + appended draft/hidden tensors with their own
|
||
strides; the memory estimator adds the extras. Patch 04b extends the
|
||
registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
|
||
5 x hidden_size x 2 B (the aux hidden-state cache) per token when
|
||
`speculative_config.use_aux_hidden_state_outputs` is set.
|
||
|
||
The pure logic is smoke-tested locally (CPU torch + the image's real spec
|
||
classes against synthetic GLM+draft+hidden spec sets): grouping shape,
|
||
layout recognition, builder block math and memory accounting all pass.
|
||
Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
|
||
65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
|
||
2,432 tokens ≈ 70,500 ≥ 65,536.
|
||
|
||
All of these mount over site-packages via the scratch launcher's
|
||
`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
|
||
`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
|
||
`expert_activation_registry.py`; apply-verified against the image's pristine
|
||
files). The window script's decision pipeline was also fixed for a SIGPIPE
|
||
death (`head -80` on a `podman logs` grep pipeline killed one run before
|
||
log-save and restore; the log is now captured to a file FIRST, then grepped).
|
||
|
||
|
||
## Measured arms (all 2026-09-30, one boot each)
|
||
|
||
| arm | decode | prefill | verdict |
|
||
| --- | --- | --- | --- |
|
||
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
|
||
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
|
||
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
|
||
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
|
||
|
||
### Why MTP is rejected here (it is not that it fails)
|
||
|
||
MTP **works** on GLM in this fork: the head loads, the architecture resolves to
|
||
`Glm5NextMTPModel`, and acceptance is excellent —
|
||
|
||
```
|
||
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
|
||
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
|
||
Avg Draft acceptance rate: 69-97%
|
||
```
|
||
|
||
— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
|
||
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
|
||
verification forward gathers the **union** of the batch's experts, so spec
|
||
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
|
||
of arena. On a compute-bound (fully-resident) shape this would invert; here it
|
||
does not.
|
||
|
||
## Served surface (verified 2026-09-30 against the live engine)
|
||
|
||
| capability | state | evidence |
|
||
| --- | --- | --- |
|
||
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
|
||
| completion | yes | `bench_glm.py` runs above |
|
||
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
|
||
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
|
||
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
|
||
|
||
Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
|
||
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
|
||
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
|
||
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
|
||
|
||
## Fork traps found (for the engine authors)
|
||
|
||
1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
|
||
nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
|
||
`_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
|
||
Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
|
||
op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
|
||
(`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
|
||
the MTP draft step demands an op that cannot exist here:
|
||
`RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
|
||
attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
|
||
(`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
|
||
CDNA check, not the env var.)
|
||
- **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
|
||
more tokens in the same reserve, and this fork's reserve model is
|
||
**dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
|
||
`KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
|
||
dominated by the per-sequence KDA state (34 linear-attention layers) and the
|
||
DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
|
||
cover — so halving the KV dtype cannot move it.
|
||
- **What a fix would take, if it were ever wanted**: a new fp8 path in the
|
||
rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
|
||
in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
|
||
in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
|
||
`_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
|
||
path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
|
||
**do not spend effort here** — the payoff is capacity, and the reserve is
|
||
dtype-blind.
|
||
2. **The plan's KV reserve under-models this model, and the cap enforces the
|
||
mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
|
||
carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
|
||
the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
|
||
`--max-model-len` becomes unbootable:
|
||
`ValueError: To serve at least one request with the model's max seq len
|
||
(131072), (2.07 GiB KV cache is needed, which is larger than the available KV
|
||
cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
|
||
tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
|
||
fixed) — where the reserve would eat a third of the arena.
|
||
`scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
|
||
request raise the cap; **it is not part of the deployed shape**, which boots
|
||
on the plan's own reserve.
|
||
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
|
||
what the reserve model requires for this payload (below ~92k the boot
|
||
refuses); it is not a capacity decision.
|
||
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
|
||
(36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
|
||
|
||
## Operations
|
||
|
||
```bash
|
||
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
|
||
# dirs are untouched on disk):
|
||
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
|
||
./scripts/deploy.sh dankserver --service frank-archon
|
||
|
||
# payload inventory (34 shards + 2 extras, weight-map index):
|
||
bash scripts/download-frank-archon.sh --verify-only
|
||
|
||
# live telemetry (needs the profiler window; with_stack=false is mandatory —
|
||
# with stack the stop path hangs and the trace buffer grows to tens of GB):
|
||
podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"
|
||
```
|
||
|
||
**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
|
||
**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
|
||
`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
|
||
liveness probe a lying oracle for a restore — twice a restore script printed
|
||
`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
|
||
still in place. The restore oracle must be a **conjunction**: `is-active` ==
|
||
`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
|
||
mask symlink **and** the container is owned by the unit and not `Stopping`. The
|
||
repair recipe, all four steps needed:
|
||
|
||
```bash
|
||
sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl reset-failed frank-archon.service # a failed unit refuses to start
|
||
podman rm -f frank-archon-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
|
||
sudo systemctl start frank-archon.service # then ~7 min to health 200
|
||
```
|
||
|
||
Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
|
||
glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
|
||
container, so one whole round produced no evidence at all (`no container with name
|
||
glm-probe found`) — check `proc://` for running jobs before opening a window.
|
||
|
||
**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
|
||
bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
|
||
name-map lookups and vLLM module forwards) produced a self-replicating chain of
|
||
`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
|
||
imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
|
||
that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
|
||
the hook) and spawned again. It was still alive 18 minutes later with the container
|
||
still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
|
||
identical interpreters, each child's PPID being the previous one. Guard (now in the
|
||
hook): arm once per environment via a marker in `os.environ`, which children inherit —
|
||
|
||
```python
|
||
if os.environ.get("VLLM_PROBE_ARMED") == "1":
|
||
pass # an ancestor already armed it; stay inert
|
||
else:
|
||
os.environ["VLLM_PROBE_ARMED"] = "1"
|
||
<install probes>
|
||
```
|
||
|
||
Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
|
||
generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
|
||
variable — **never** pattern-kill an interpreter name, because the media stack on this
|
||
host (romm, jellyfin, plex, glances) runs python in production.
|
||
|
||
```bash
|
||
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
|
||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
|
||
|
||
# experiment windows (production stopped + `systemctl mask --runtime`, which
|
||
# self-clears at reboot) ran from the host-side scratch launcher
|
||
# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
|
||
# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
|
||
# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
|
||
# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
|
||
# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
|
||
# word-split by the launcher's unquoted $EXTRA).
|
||
bash /home/dank/serve_glm_autoround.sh
|
||
```
|
||
|
||
Boot gate: the engine refuses a legal-looking shape in two measured ways — the
|
||
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
|
||
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
|
||
what killed the first two boots at util 0.95.
|
||
|
||
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
|
||
shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
|
||
the default allocator's fragmentation fails the profile/graph-capture phase
|
||
(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
|
||
MiB is free").
|
||
|
||
The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
|
||
walks every guarded vhost — including `http://192.168.0.112:8989/` on
|
||
FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
|
||
timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
|
||
healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
|
||
`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
|
||
and the WoL packet did not wake it (macOS "Wake for network access" needs
|
||
toggling on the mini itself).
|
||
|
||
## Rollback targets
|
||
|
||
| state | where |
|
||
| --- | --- |
|
||
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
|
||
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
|