frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory
Root-caused the moving failures: they were my own scratch launcher. Its compile branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to True — and with fullgraph=True EVERY break is an error, so each boot reported whichever break came first (cell -> workspace -> compiler-disable). I had blamed TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The launcher sets the env var again, and patch 04's header, the runbook and the work log now carry the true run history as a table instead of the bad theory. New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch — the indexer acquires its shared workspace INSIDE the forward (its profile-run branch literally says 'Reserve workspace for indexer during profiling run'), and v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper (narrower than decorating the workspace manager, which every model would go through; a no-op when not compiling). All five artifacts verified against the pinned image: applied in sequence with patch -p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…, 732363bb…, 84969993…, c154115a…). tests/run.sh: 362 seams pass.
This commit is contained in:
1 parent
fd864b5a42
commit
71db0debcc
4 files changed
+166
-63
No files matched your search
@@ -4,24 +4,30 @@
|
||||
# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
|
||||
# decorator) to matter; not part of the deployed shape.
|
||||
#
|
||||
# MEASUREMENT NOTE — READ BEFORE TRUSTING ANY CLAIM BELOW. Two break-tolerance runs
|
||||
# disagree, and the difference is a logging flag:
|
||||
# * 01+02+03, AOT off, breaks tolerated, NO TORCH_LOGS -> gets PAST all graph-break
|
||||
# handling and dies on the dynamo name-map KeyError ('in_proj_qkvbfg_a').
|
||||
# * same + TORCH_LOGS=graph_breaks -> dies on the fatal
|
||||
# `Unsupported: Read uninitialized cell` at this very call site.
|
||||
# The likely mechanism is that `graph_breaks` logging makes dynamo build a break
|
||||
# *explanation* (walk the frames, name the reason), and that analysis is what trips on
|
||||
# the uninitialized cell; without the flag the break is taken silently. So the cell
|
||||
# error is PROBABLY an artifact of the diagnostic, not necessarily a blocker for a
|
||||
# break-tolerant compile. Whether this patch is therefore necessary — as opposed to
|
||||
# removing a call dynamo cannot *explain* — is being decided by a flag-free run of
|
||||
# 01+02+03+04. Do not repeat the earlier overclaim ("fatal even with fullgraph=False");
|
||||
# it was written from the confounded run.
|
||||
# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
|
||||
# launcher bug made four consecutive boots misleading. The scratch launcher
|
||||
# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
|
||||
# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
|
||||
# patch while `_FULLGRAPH` still defaulted to True. With fullgraph=True *every* break
|
||||
# is an error, which is why the reported failure kept moving as breaks were removed:
|
||||
#
|
||||
# fullgraph=False, AOT off -> breaks tolerated; dies later on the dynamo name-map
|
||||
# `KeyError: 'in_proj_qkvbfg_a'`
|
||||
# fullgraph=True, AOT off -> dies on whichever break comes first
|
||||
# (cell -> workspace -> compiler-disable)
|
||||
#
|
||||
# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
|
||||
# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
|
||||
# Launcher fixed; the env var is set again.
|
||||
#
|
||||
# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
|
||||
# break dynamo reported, and after this patch the report moved on to the workspace
|
||||
# acquisition — so it removes a real break. It is not by itself what unblocks a boot:
|
||||
# with breaks genuinely tolerated the blocker is the name-map KeyError.
|
||||
#
|
||||
# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
|
||||
# (@kernel_launcher, 39 call sites) is exactly what breaks dynamo's break analysis, and
|
||||
# the object it closes over is bound after use (see below).
|
||||
# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
|
||||
# object this one closes over is bound after use (see below).
|
||||
#
|
||||
# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30). With 01+02+03 the
|
||||
# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
|
||||
|
||||
Reference in new issue
Block a user