serverAppConfigs/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	vLLM Porting and Compile Triage	Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.	vllm porting torch-compile dynamo eagle3 dflash glm	2026-10-05T00:00:00-07:00

vLLM Porting and Compile Triage

Part I — Forward-porting a model architecture

1. Check whether a port already exists; size the work honestly

Read what the target image already ships before assuming a port is needed: a backport's upstream PR may be merged in the target version, and the target may carry a native kernel path the older line lacks (e.g. rfi_hip for RDNA4 native quantized MoE). Porting around a native path that was already there is the expensive mistake.

An upstream backport ships two kinds of file: additive (the new model package, its config, its kernel modules — port these) and whole-file replacements of vLLM core (model_runner.py, scheduler.py, kv_cache_utils.py, platform files — these are version X's copies of files version Y restructured; copying them is a DOWNGRADE, not a port). Measure both (find … | xargs wc -l), and say the additive-only ratio out loud before starting — one real case: a 71,724-line overlay reduced to 10,385 additive, and the "missing" drafter was already registered in the target.

2. Enumerate the gap in ONE pass, not one ImportError at a time

Inventory the target tree first (model_executor/layers/, v1/attention/backends/, quantization backends, spec-decode drafters in registry.py, third_party/flash_linear_attention/ops/). Then walk the staged package with ast, resolving every from vllm… import X against the target, split into MODULE-missing vs NAME-missing.

• The double-prefix bug: the dotted module already starts with vllm., so joining onto the package dir needs rel = mod.split(".", 1)[1].replace(".", os.sep).
• Static scans over-report: bare annotations, TypeAlias and star re-exports resolve at runtime. Do a runtime check (importlib.import_module + getattr) before quoting numbers — one scan said 151/163 resolved; runtime said 10/13.
• Classify: redirects dominate (version moved a symbol to a submodule) — a ~60-line rewriter splitting from X import (a, b, c) blocks handles them; delete a block that empties (a bare from X import ( with an empty body is a SyntaxError) and gate the rewriter on ast.parse of every touched file.
• Port absent symbols with AST extraction, not copy-paste: locate the ClassDef, check every base/decorator resolves in the target, append the exact segment, re-ast.parse the target, roll back on failure, skip if already defined.

3. Silent-default traps and transformers gates

• Check every cfg_int(cfg, key, DEFAULT) against the new checkpoint: if the key was renamed, the DEFAULT silently survives and the model computes fluent nonsense with no error. Diff every config key the old parser reads against the new config.json. When the old port used a fallback, the CHECKPOINT's value must win (measured: num_experts_per_token fallback 7 vs checkpoint num_experts_per_tok=8).
• model type X but Transformers does not recognize this architecture = pinned transformers predates the model. Upgrading transformers inside a pinned vLLM image breaks things hard to attribute — prefer registering a genuine PretrainedConfig subclass: CONFIG_MAPPING.register("glm5_next", Glm5NextConfig), gated by a real round-trip (AutoConfig.from_pretrained(<ckpt>) returns the right class). If the target image's transformers is below vLLM's declared minimum, upgrading inside the image beats a shim.
• Hardware-gate flags from the port's own docs: search the README for the arch name and "do not" — e.g. a native quant MoE backend that is CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at ~5x slower; pin those flags in the serve script with a comment citing the line.
• Backend selection: default to auto, then READ the chosen-backend line. TRITON_UNFUSED rejects some activation keys; TRITON_ATTN rejects MLA + sparse attention — both surface as an init rejection. "Using 'EMULATION' backend" is a FAILED configuration, not a success (weights dequantized per call: memory + throughput cost).
• Loading model from scratch… for many minutes usually means the port SUCCEEDED and you are in weight placement — verify all layers constructed, both TP workers alive, zero errors, before chasing memory. And if it does not fit, prove the band is empty: sweep the knob and show no setting works; a single OOM is not a conclusion. Keep MemAvailable guarded (abort on low RAM; do not let the OOM killer take co-tenants).

4. Shims that must survive worker spawn

vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone by weight-load time. In order of preference:

1. Patch the installed library's source file on disk (reliable). Locate the real install path at runtime: python -c "import transformers,os;print(os.path.dirname(transformers.__file__))".
2. .pth in site-packages — works but runs at interpreter start and can import a package mid-import (circular import through accelerate); expect failure.
3. sitecustomize.py — often silently DEAD: Debian ships its own at /usr/lib/python3.12/sitecustomize.py. Verify it is actually imported before debugging why it has no effect. (A .pth on PYTHONPATH is never executed by site.py; sitecustomize.py is.)

Always confirm with a FRESH process — never with the process that set the patch up (is_moe, num_experts_per_token, mhc_num_residual_streams).

Misleading OOM names on ROCm: torch.AcceleratorError: CUDA error: out of memory from pin_memory() is usually the HOST memlock rlimit, not VRAM. Cannot be raised from inside a container or root's own shell; only PAM at login: /etc/security/limits.d/*.conf + a NEW session. Confirm by pinning 2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the failure you are chasing.

5. Ship as an idempotent patch-at-start applier

One apply.sh at container start, each step guarded by a grep-for-marker (re-entry safe), ending in a verification block that imports every installed symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute load, and it turns "I think it works" into evidence.

---

Part II — torch.compile / dynamo bring-up triage

The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)

1. suppress_errors does NOT cover graph breaks — fullgraph=True + suppress_errors=True still raises Unsupported. Only fullgraph=False tolerates a break.
2. A warm lru_cache is not trace-safe — dynamo re-traces the wrapped body to build its guard even on a hit ("Dynamo ignores the cache wrapper and directly traces the wrapped function"). Resolve the value ONCE into a plain module global and read that; a global read folds, a cached call does not. Verified in-image: warm lru_cache-calling function still fails fullgraph=True; a plain global read compiles.
3. AOT compile refuses breaks outright ("Graph breaks are not supported with aot compile") — set VLLM_USE_AOT_COMPILE=0 for a break-tolerant run.
4. Fatal vs tolerated breaks: tolerated breaks get a resume function (untraceable part runs eagerly, numerics unchanged). Fatal ones cannot: Unsupported: Read uninitialized cell (a traced callee closes over a name bound after its definition) and Skip calling torch.compiler.disable()'d function under fullgraph=True. torch.compiler.disable is the documented escape hatch for host-side/stateful work (ContextVar reads, lazy allocation, workspace grab) and is a NO-OP when nothing compiles — safe to ship in the stock path.
5. Trust the frame chain, not the line number — dynamo INLINES user callees; an inlined failure reports the enclosing function's def line. "KeyError in forward at the def line" is really "a callee raised". TORCHDYNAMO_VERBOSE=1 adds nothing for a plain Python exception (only for dynamo-internal errors).

The tell that the trunk lacks compile support at all:

Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
FULL_DECODE_ONLY.

Diff against the family that DOES compile and copy its @support_torch_compile decorator including dynamic_arg_dims and any enable_if hook; env-gate every change so the file can stay mounted for an A/B.

Procedure

0. Establish what configuration is ACTUALLY in effect: podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …. A launcher that "sets" an env var in a branch may silently drop it — four consecutive boots ran fullgraph=True while the operator believed otherwise, which makes the reported failure MOVE as breaks are removed and invites entirely wrong theories.

1. Boot with the break list, not the first error: TORCH_LOGS=graph_breaks

   • VLLM_USE_AOT_COMPILE=0 + VLLM_COMPILE_FULLGRAPH=0. Re-verify each candidate break under the flag-free config you actually intend — a break that appears only under the logging flag is a diagnostic artifact.

2. Fix in priority order: (a) host syncs in the traced region (.item()/.cpu()/.tolist() — check reachability first, a dead fallback branch is not your blocker); (b) capability probes doing I/O in forward (os.path.exists, .so/ctypes lookups — resolve once into globals, warm at import); (c) lazy stateful work (hoist, or torch.compiler.disable a thin wrapper); (d) new kernel specializations — check the vendor ships SOURCES (nm -D/strings): a binary-only templated kernel<128|256|512> makes the 288 you need unreachable. Fix at the PATTERN level (one launcher/decorator family breaking all 39 call sites), not site level.

3. KeyError on a bare module name = dynamo's name→object map miss (OutputGraph.nn_modules: registered output_graph.py:1259/:1384, read back :3000, _buffers/_parameters/_modules stripped at :1225). Ship a probe via sitecustomize.py that patches OutputGraph.example_value_from_input_node and keeps weakrefs of live graphs, printing node.target, key count, near-miss keys — separates a cross-graph reference from a name-derivation mismatch.

4. Instrumenting a startup hook is a FIREARM — arm it once:

   if os.environ.get("PROBE_ARMED") == "1":
       pass                     # an ancestor already armed it -> stay inert
   else:
       os.environ["PROBE_ARMED"] = "1"
       ...install probes...
   

   A sitecustomize.py on PYTHONPATH imports in EVERY interpreter and children inherit PYTHONPATH: an unguarded hook that imports the engine self-replicates (observed: 19+ levels, 33 processes alive 18 min later). Kill the chain by removing the container whose cgroup owns every generation, then verify with a precise /proc/<pid>/environ count — NEVER pattern-kill an interpreter name on a host running python in production containers. Also chain the image's own sitecustomize so its hook survives.

5. Cheap in-image classification (no boot): g = torch.compile(fn, backend="eager", fullgraph=True) inside the container (with --device /dev/kfd --device /dev/dri — torch.compile queries a device even on the eager backend). ~5 s per question instead of ~7 min boots.

6. Package as real patch artifacts: hand-written @@ headers are not diffs and patch(1) rejects them. Generate diff -u --label a/<path> --label b/<path> orig new > name.patch with --- a/ / +++ b/ paths so -p1 works; verify by applying to a fresh extraction of the pinned image and comparing sha256 (sha256sum must equal the b/ hash recorded in the header). Header carries: STATUS, the measured evidence, base+patched sha256, how to apply, the NEXT blocker.

Cost model before investing

Profile first — parse the rank0 chrome trace (POST /start_profile, device-activity rows) for the kernel table. Measured case: the long-suspected expert-gather was 1.05 % of device time while 44 % was unfused pointwise glue (708,500 vectorized_elementwise_kernel launches = 1,428/token) — a months-old "structural" conclusion died in one afternoon. Speculative decoding does NOT help a shape whose per-token cost scales with the token count, and it hurts prefill. Decide the ceiling up front: if the fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.

---

Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port

Goal: make a trunk satisfy SupportsEagle3 so a DFlash2 drafter (DFlash2DraftModel, method dflash) boots. Built on the GLM-5.3-Flash port (tcclaviger fork, 2026-09-30; four failed boots produced the laws below).

The four laws

1. Patch the instance the runner calls. set_eagle3_aux_hidden_state_layers(self.model) (model_runner.py, right after model_loader.load_model) runs on the LOADED instance — for a multimodal model that is the …ForConditionalGeneration WRAPPER (the APIServer log line Resolved architecture: … names it), NOT the text model it holds as self.language_model. Patching only the inner …ForCausalLM reproduces the identical RuntimeError: Model does not support EAGLE3 interface forever. Patch BOTH wrappers.
2. The check is Protocol isinstance, NOT duck typing. supports_eagle3(model) = isinstance(model, SupportsEagle3) (interfaces.py ~1728; caller v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20). SupportsEagle3 is @runtime_checkable over base SupportsEagleBase (:1486, :1659); typing._ProtocolMeta.__instancecheck__ does getattr_static over ALL __protocol_attrs__ — including BASE-protocol members — and breaks on the FIRST miss. Full member set: supports_eagle3, has_own_lm_head, has_own_embed_tokens, set_aux_hidden_state_layers, get_eagle3_default_aux_hidden_state_layers. Class attributes count as presence; a methods-only implementation FAILS because the three attrs are missing. Verified in-image with stock typing, no boot: flag-only → False; flag + two base members → True.
3. MRO forbids inheriting the Protocol. Appending SupportsEagle3 to a wrapper already carrying Protocol-derived bases dies at class creation: Cannot create a consistent method resolution order (MRO) for bases … — MRO simulation (stub Protocols + type("C", bases, {})) confirmed no position works. This failure also PROVES the isinstance-semantics question is moot: class creation precedes any check.
4. Write the two methods by hand — without inheritance the Protocol defaults do not exist. Text model: unwrap to the trunk (self.model.aux_hidden_state_layers = layers); multimodal wrapper: delegate to self.language_model (the pixtral.py:498 pattern). get_eagle3_default_aux_hidden_state_layers returns (2, L//2, L-3); the drafter config's target_layer_ids wins when present (GLM DFlash2 ships [5, 14, 24, 33, 42] of 45).

The fix shape (on EACH wrapper class):

supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False          # False when embed/lm_head are delegated
has_own_embed_tokens: bool = False
def set_aux_hidden_state_layers(self, layers): ...
def get_eagle3_default_aux_hidden_state_layers(self): ...

Trunk mechanics

The trunk (Glm5NextModel) must be EagleModelMixin; add aux_hidden_state_layers: set[int] (default empty) + start_layer handling. Capture aux states in the layer loop: the state ENTERING each requested layer, BEFORE its input norm (GLM's loop carries the single-stream state; deepseek_v2's equivalent is hidden_states + residual). use_aux_hidden_state_outputs=True is set in model_runner.py (~:287) for methods dflash, dspark, extract_hidden_states; the forward returns (hidden_states, aux_hidden_states) conditionally after self.norm (tuple consumed at v1/worker/gpu/cudagraph_utils.py:621).

Window discipline and verification

• Scratch boots on dankserver MUST stop + runtime-mask production frank-archon first — the expert-offload planner reads MemAvailable at plan time, and beside-prod boots die at plan with ExpertPlanError: host holds N of M swappable experts. After stop+mask, loop until MemAvailable ≥ 230 GiB before launching (a stop against a mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to return — a new window started during restore inherits the false failure).
• Never launch a window while the previous window's RESTORE is in flight (systemctl start returns immediately; the engine loads ~6 min).
• Save full probe logs BEFORE teardown (podman logs <probe> > full.log); decision greps truncate lines at … and tracebacks die with the container.
• Cheap verification, no boot: AST-parse the patched file for class bases + member presence; MRO simulation with stub Protocols; in-image typing replication of the isinstance walk (pure stdlib — importing vllm itself crashes on GPU-less inspection). Boot check: the worker line Using Eagle3 auxiliary layers from config: (…) = gate cleared.
• Honest expectations: a target-side DFlash2/EAGLE3 port on a pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain — the verification pass costs ∝ its token count. Port it for capability, not speed.

Related

• vLLM Boot Forensics and Config Interrogation — capture and contracts
• RDNA4 Container Serving — the serving stack being ported onto
• frank-archon — the live deployment
• frank-dragoon — the retired w4b-era endpoint and its profiling notes