serverAppConfigs/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md

---
type: reference
title: vLLM Porting and Compile Triage
description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
timestamp: 2026-10-05T00:00:00-07:00
---

# vLLM Porting and Compile Triage

## Part I — Forward-porting a model architecture

### 1. Check whether a port already exists; size the work honestly

Read what the target image already ships before assuming a port is needed: a
backport's upstream PR may be merged in the target version, and the target may
carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
native quantized MoE). Porting around a native path that was already there is
the expensive mistake.

An upstream backport ships two kinds of file: **additive** (the new model
package, its config, its kernel modules — port these) and **whole-file
replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
`kv_cache_utils.py`, platform files — these are version X's copies of files
version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
(`find … | xargs wc -l`), and say the additive-only ratio out loud before
starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
and the "missing" drafter was already registered in the target.

### 2. Enumerate the gap in ONE pass, not one ImportError at a time

Inventory the target tree first (`model_executor/layers/`,
`v1/attention/backends/`, quantization backends, spec-decode drafters in
`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
staged package with `ast`, resolving every `from vllm… import X` against the
target, split into MODULE-missing vs NAME-missing.

- **The double-prefix bug:** the dotted module already starts with `vllm.`,
  so joining onto the package dir needs
  `rel = mod.split(".", 1)[1].replace(".", os.sep)`.
- **Static scans over-report:** bare annotations, TypeAlias and star
  re-exports resolve at runtime. Do a runtime check
  (`importlib.import_module` + `getattr`) before quoting numbers — one scan
  said 151/163 resolved; runtime said 10/13.
- **Classify:** redirects dominate (version moved a symbol to a submodule) —
  a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
  them; **delete a block that empties** (a bare `from X import (` with an
  empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
  touched file.
- **Port absent symbols with AST extraction**, not copy-paste: locate the
  `ClassDef`, check every base/decorator resolves in the target, append the
  exact segment, re-`ast.parse` the target, roll back on failure, skip if
  already defined.

### 3. Silent-default traps and transformers gates

- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
  key was renamed, the DEFAULT silently survives and the model computes fluent
  nonsense with no error. Diff every config key the old parser reads against
  the new `config.json`. When the old port used a fallback, the CHECKPOINT's
  value must win (measured: `num_experts_per_token` fallback 7 vs
  checkpoint `num_experts_per_tok=8`).
- `model type X but Transformers does not recognize this architecture` =
  pinned transformers predates the model. Upgrading transformers inside a
  pinned vLLM image breaks things hard to attribute — prefer registering a
  genuine `PretrainedConfig` subclass:
  `CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
  round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
  If the target image's transformers is below vLLM's declared minimum,
  upgrading inside the image beats a shim.
- **Hardware-gate flags from the port's own docs**: search the README for the
  arch name and "do not" — e.g. a native quant MoE backend that is
  CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
  ~5x slower; pin those flags in the serve script with a comment citing the
  line.
- **Backend selection:** default to `auto`, then READ the chosen-backend
  line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
  MLA + sparse attention — both surface as an init rejection. **"Using
  'EMULATION' backend" is a FAILED configuration, not a success** (weights
  dequantized per call: memory + throughput cost).
- `Loading model from scratch…` for many minutes usually means the port
  SUCCEEDED and you are in weight placement — verify all layers constructed,
  both TP workers alive, zero errors, before chasing memory. And if it does
  not fit, **prove the band is empty**: sweep the knob and show no setting
  works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
  (abort on low RAM; do not let the OOM killer take co-tenants).

### 4. Shims that must survive worker spawn

vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
by weight-load time. In order of preference:

1. **Patch the installed library's source file on disk** (reliable). Locate
   the real install path at runtime:
   `python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
2. `.pth` in site-packages — works but runs at interpreter start and can
   import a package mid-import (circular import through `accelerate`);
   expect failure.
3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
   `/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
   before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
   executed by `site.py`; `sitecustomize.py` is.)

Always confirm with a FRESH process — never with the process that set the
patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).

**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
Cannot be raised from inside a container or root's own shell; only PAM at
login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
failure you are chasing.

### 5. Ship as an idempotent patch-at-start applier

One `apply.sh` at container start, each step guarded by a grep-for-marker
(re-entry safe), ending in a verification block that imports every installed
symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
load, and it turns "I think it works" into evidence.

---

## Part II — torch.compile / dynamo bring-up triage

### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)

1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
   suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
   tolerates a break.
2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
   body to build its guard even on a hit ("Dynamo ignores the cache wrapper
   and directly traces the wrapped function"). Resolve the value ONCE into a
   plain module global and read that; a global read folds, a cached call does
   not. Verified in-image: warm lru_cache-calling function still fails
   fullgraph=True; a plain global read compiles.
3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
   with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
   run.
4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
   (untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
   `Unsupported: Read uninitialized cell` (a traced callee closes over a name
   bound after its definition) and `Skip calling torch.compiler.disable()'d
   function` under fullgraph=True. `torch.compiler.disable` is the documented
   escape hatch for host-side/stateful work (ContextVar reads, lazy
   allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
   ship in the stock path.
5. **Trust the frame chain, not the line number** — dynamo INLINES user
   callees; an inlined failure reports the enclosing function's `def` line.
   "KeyError in forward at the def line" is really "a callee raised".
   `TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
   for dynamo-internal errors).

**The tell** that the trunk lacks compile support at all:

```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
FULL_DECODE_ONLY.
```

Diff against the family that DOES compile and copy its
`@support_torch_compile` decorator including `dynamic_arg_dims` and any
`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.

### Procedure

0. **Establish what configuration is ACTUALLY in effect**:
   `podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
   A launcher that "sets" an env var in a branch may silently drop it — four
   consecutive boots ran `fullgraph=True` while the operator believed
   otherwise, which makes the reported failure MOVE as breaks are removed and
   invites entirely wrong theories.
1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
   + `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
   candidate break under the flag-free config you actually intend — a break
   that appears only under the logging flag is a diagnostic artifact.
2. **Fix in priority order:** (a) host syncs in the traced region
   (`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
   branch is not your blocker); (b) capability probes doing I/O in `forward`
   (`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
   warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
   a thin wrapper); (d) new kernel specializations — check the vendor ships
   SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
   makes the 288 you need unreachable. Fix at the PATTERN level (one
   launcher/decorator family breaking all 39 call sites), not site level.
3. **KeyError on a bare module name** = dynamo's name→object map miss
   (`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
   read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
   Ship a probe via `sitecustomize.py` that patches
   `OutputGraph.example_value_from_input_node` and keeps weakrefs of live
   graphs, printing `node.target`, key count, near-miss keys — separates a
   cross-graph reference from a name-derivation mismatch.
4. **Instrumenting a startup hook is a FIREARM — arm it once:**

   ```python
   if os.environ.get("PROBE_ARMED") == "1":
       pass                     # an ancestor already armed it -> stay inert
   else:
       os.environ["PROBE_ARMED"] = "1"
       ...install probes...
   ```

   A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
   children inherit `PYTHONPATH`: an unguarded hook that imports the engine
   self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
   Kill the chain by removing the container whose cgroup owns every
   generation, then verify with a precise `/proc/<pid>/environ` count —
   NEVER pattern-kill an interpreter name on a host running python in
   production containers. Also chain the image's own sitecustomize so its
   hook survives.
5. **Cheap in-image classification (no boot):**
   `g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
   container (with `--device /dev/kfd --device /dev/dri` — torch.compile
   queries a device even on the eager backend). ~5 s per question instead of
   ~7 min boots.
6. **Package as real patch artifacts:** hand-written `@@` headers are not
   diffs and `patch(1)` rejects them. Generate
   `diff -u --label a/<path> --label b/<path> orig new > name.patch` with
   `--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
   extraction of the pinned image and comparing sha256 (`sha256sum` must
   equal the `b/` hash recorded in the header). Header carries: STATUS, the
   measured evidence, base+patched sha256, how to apply, the NEXT blocker.

### Cost model before investing

**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
device-activity rows) for the kernel table. Measured case: the long-suspected
expert-gather was **1.05 %** of device time while **44 %** was unfused
pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
1,428/token) — a months-old "structural" conclusion died in one afternoon.
Speculative decoding does NOT help a shape whose per-token cost scales with
the token count, and it hurts prefill. Decide the ceiling up front: if the
fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.

---

## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port

Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).

### The four laws

1. **Patch the instance the runner calls.**
   `set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
   after `model_loader.load_model`) runs on the LOADED instance — for a
   multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
   APIServer log line `Resolved architecture: …` names it), NOT the text
   model it holds as `self.language_model`. Patching only the inner
   `…ForCausalLM` reproduces the identical `RuntimeError: Model does not
   support EAGLE3 interface` forever. Patch BOTH wrappers.
2. **The check is Protocol isinstance, NOT duck typing.**
   `supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
   (interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
   `SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
   (`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
   `getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
   members — and breaks on the FIRST miss. Full member set:
   `supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
   `set_aux_hidden_state_layers`,
   `get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
   presence; a methods-only implementation FAILS** because the three attrs
   are missing. Verified in-image with stock typing, no boot: flag-only →
   False; flag + two base members → True.
3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
   wrapper already carrying Protocol-derived bases dies at class creation:
   `Cannot create a consistent method resolution order (MRO) for bases …` —
   MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
   position works. This failure also PROVES the isinstance-semantics question
   is moot: class creation precedes any check.
4. **Write the two methods by hand** — without inheritance the Protocol
   defaults do not exist. Text model: unwrap to the trunk
   (`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
   delegate to `self.language_model` (the pixtral.py:498 pattern).
   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
   drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
   `[5, 14, 24, 33, 42]` of 45).

The fix shape (on EACH wrapper class):

```python
supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False          # False when embed/lm_head are delegated
has_own_embed_tokens: bool = False
def set_aux_hidden_state_layers(self, layers): ...
def get_eagle3_default_aux_hidden_state_layers(self): ...
```

### Trunk mechanics

The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
Capture aux states in the layer loop: the state ENTERING each requested
layer, BEFORE its input norm (GLM's loop carries the single-stream state;
deepseek_v2's equivalent is `hidden_states + residual`).
`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
consumed at `v1/worker/gpu/cudagraph_utils.py:621`).

### Window discipline and verification

- Scratch boots on dankserver MUST stop + runtime-mask production
  `frank-primary` first — the expert-offload planner reads `MemAvailable` at
  plan time, and beside-prod boots die at plan with
  `ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
  loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
  mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
  return — a new window started during restore inherits the false failure).
- Never launch a window while the previous window's RESTORE is in flight
  (`systemctl start` returns immediately; the engine loads ~6 min).
- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
  decision greps truncate lines at `…` and tracebacks die with the container.
- Cheap verification, no boot: AST-parse the patched file for class bases +
  member presence; MRO simulation with stub Protocols; in-image `typing`
  replication of the isinstance walk (pure stdlib — importing vllm itself
  crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
  auxiliary layers from config: (…)` = gate cleared.
- Honest expectations: a target-side DFlash2/EAGLE3 port on a
  pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
  the verification pass costs ∝ its token count. Port it for capability, not
  speed.

## Related

- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[RDNA4 Container Serving]] — the serving stack being ported onto
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes