frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory
Root-caused the moving failures: they were my own scratch launcher. Its compile branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to True — and with fullgraph=True EVERY break is an error, so each boot reported whichever break came first (cell -> workspace -> compiler-disable). I had blamed TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The launcher sets the env var again, and patch 04's header, the runbook and the work log now carry the true run history as a table instead of the bad theory. New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch — the indexer acquires its shared workspace INSIDE the forward (its profile-run branch literally says 'Reserve workspace for indexer during profiling run'), and v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper (narrower than decorating the workspace manager, which every model would go through; a no-op when not compiling). All five artifacts verified against the pinned image: applied in sequence with patch -p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…, 732363bb…, 84969993…, c154115a…). tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
@@ -4,24 +4,30 @@
|
||||
# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
|
||||
# decorator) to matter; not part of the deployed shape.
|
||||
#
|
||||
# MEASUREMENT NOTE — READ BEFORE TRUSTING ANY CLAIM BELOW. Two break-tolerance runs
|
||||
# disagree, and the difference is a logging flag:
|
||||
# * 01+02+03, AOT off, breaks tolerated, NO TORCH_LOGS -> gets PAST all graph-break
|
||||
# handling and dies on the dynamo name-map KeyError ('in_proj_qkvbfg_a').
|
||||
# * same + TORCH_LOGS=graph_breaks -> dies on the fatal
|
||||
# `Unsupported: Read uninitialized cell` at this very call site.
|
||||
# The likely mechanism is that `graph_breaks` logging makes dynamo build a break
|
||||
# *explanation* (walk the frames, name the reason), and that analysis is what trips on
|
||||
# the uninitialized cell; without the flag the break is taken silently. So the cell
|
||||
# error is PROBABLY an artifact of the diagnostic, not necessarily a blocker for a
|
||||
# break-tolerant compile. Whether this patch is therefore necessary — as opposed to
|
||||
# removing a call dynamo cannot *explain* — is being decided by a flag-free run of
|
||||
# 01+02+03+04. Do not repeat the earlier overclaim ("fatal even with fullgraph=False");
|
||||
# it was written from the confounded run.
|
||||
# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
|
||||
# launcher bug made four consecutive boots misleading. The scratch launcher
|
||||
# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
|
||||
# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
|
||||
# patch while `_FULLGRAPH` still defaulted to True. With fullgraph=True *every* break
|
||||
# is an error, which is why the reported failure kept moving as breaks were removed:
|
||||
#
|
||||
# fullgraph=False, AOT off -> breaks tolerated; dies later on the dynamo name-map
|
||||
# `KeyError: 'in_proj_qkvbfg_a'`
|
||||
# fullgraph=True, AOT off -> dies on whichever break comes first
|
||||
# (cell -> workspace -> compiler-disable)
|
||||
#
|
||||
# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
|
||||
# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
|
||||
# Launcher fixed; the env var is set again.
|
||||
#
|
||||
# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
|
||||
# break dynamo reported, and after this patch the report moved on to the workspace
|
||||
# acquisition — so it removes a real break. It is not by itself what unblocks a boot:
|
||||
# with breaks genuinely tolerated the blocker is the name-map KeyError.
|
||||
#
|
||||
# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
|
||||
# (@kernel_launcher, 39 call sites) is exactly what breaks dynamo's break analysis, and
|
||||
# the object it closes over is bound after use (see below).
|
||||
# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
|
||||
# object this one closes over is bound after use (see below).
|
||||
#
|
||||
# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30). With 01+02+03 the
|
||||
# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
|
||||
# acquisition must not be traced.
|
||||
#
|
||||
# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is the current
|
||||
# blocker; it was reached only after 04.
|
||||
#
|
||||
# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
|
||||
#
|
||||
# torch._dynamo.exc.Unsupported: Unsupported method call
|
||||
# user stack
|
||||
# glm5next/common/model.py:748 Glm5NextModel.forward
|
||||
# glm5next/common/model.py:526 Glm5NextDecoderLayer.forward
|
||||
# glm5next/common/attention.py:599 / :401
|
||||
# glm5next/amd/sparse_indexer.py:711 forward_native
|
||||
# glm5next/amd/sparse_indexer.py:132 sparse_attn_indexer_kpool
|
||||
# v1/worker/workspace.py:135 get_simultaneous
|
||||
# v1/worker/workspace.py:155 _ensure_workspace_size
|
||||
#
|
||||
# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
|
||||
#
|
||||
# if not isinstance(attn_metadata, dict):
|
||||
# # Reserve workspace for indexer during profiling run
|
||||
#
|
||||
# so the engine dies exactly where it sizes its memory. `_ensure_workspace_size`
|
||||
# reads ContextVar and ubatch state (`_workspace_lane.get()`,
|
||||
# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
|
||||
#
|
||||
# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
|
||||
# so dynamo stops analysing the call and executes it eagerly from inside the compiled
|
||||
# region. Chosen over decorating the workspace manager's own methods because that
|
||||
# would change workspace behaviour for every model, not just this one. The wrapper is
|
||||
# a no-op when nothing is compiling, so it needs no env gate and is safe in the
|
||||
# stock (uncompiled) path — which is why this patch can be reasoned about
|
||||
# independently of whether the changelist ever ships.
|
||||
#
|
||||
# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
|
||||
# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
|
||||
# can build a resume function around it is exactly what the next boot decides — the
|
||||
# difference between this failure and 04's `Read uninitialized cell`.
|
||||
#
|
||||
# TARGET: vllm/models/glm5next/amd/sparse_indexer.py (inside the pinned image)
|
||||
# base sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
|
||||
# patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
|
||||
#
|
||||
# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
|
||||
--- a/vllm/models/glm5next/amd/sparse_indexer.py
|
||||
+++ b/vllm/models/glm5next/amd/sparse_indexer.py
|
||||
@@ -35,6 +35,21 @@
|
||||
from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
|
||||
from vllm.v1.worker.workspace import current_workspace_manager
|
||||
|
||||
+@torch.compiler.disable
|
||||
+def _acquire_workspace(*shapes_and_dtypes):
|
||||
+ """Acquire shared workspace buffers, untraced.
|
||||
+
|
||||
+ Workspace acquisition reads ContextVar/ubatch state and may allocate
|
||||
+ (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
|
||||
+ ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
|
||||
+ ``Unsupported: Unsupported method call`` and kills the engine during the profile
|
||||
+ run. ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
|
||||
+ region inside the compiled graph. No-op when not compiling.
|
||||
+ """
|
||||
+
|
||||
+ return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
|
||||
+
|
||||
+
|
||||
logger = init_logger(__name__)
|
||||
|
||||
# kpool write helper: form pools from the current token batch and compress them
|
||||
@@ -129,7 +144,7 @@
|
||||
values_spec, scales_spec = _gather_workspace_shapes(
|
||||
total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
|
||||
)
|
||||
- current_workspace_manager().get_simultaneous(
|
||||
+ _acquire_workspace(
|
||||
values_spec,
|
||||
scales_spec,
|
||||
((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
|
||||
@@ -275,11 +290,10 @@
|
||||
# Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
|
||||
# MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
|
||||
# scales) based on use_fp4_cache.
|
||||
- workspace_manager = current_workspace_manager()
|
||||
values_spec, scales_spec = _gather_workspace_shapes(
|
||||
total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
|
||||
)
|
||||
- k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
|
||||
+ k_quant_full, k_scale_full = _acquire_workspace(
|
||||
values_spec,
|
||||
scales_spec,
|
||||
)
|
||||
Reference in New Issue
Block a user