docs: vault link integrity restored after rogue-session doc WIP landed
The parallel session's stash pop left GGUF Compatibility, Multi-Device Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving, vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9 unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link, and two multi-line-wrapped link tokens). All repointed to real pages; six rows added to the vault index; seam green.
This commit is contained in:
Frank committed
1 parent
380aafa1cc
commit
a6ad9af708
10 files changed
+2013
-1
No files matched your search
@@ -0,0 +1,330 @@
|
||||
---
|
||||
type: reference
|
||||
title: vLLM Porting and Compile Triage
|
||||
description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
|
||||
tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# vLLM Porting and Compile Triage
|
||||
|
||||
## Part I — Forward-porting a model architecture
|
||||
|
||||
### 1. Check whether a port already exists; size the work honestly
|
||||
|
||||
Read what the target image already ships before assuming a port is needed: a
|
||||
backport's upstream PR may be merged in the target version, and the target may
|
||||
carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
|
||||
native quantized MoE). Porting around a native path that was already there is
|
||||
the expensive mistake.
|
||||
|
||||
An upstream backport ships two kinds of file: **additive** (the new model
|
||||
package, its config, its kernel modules — port these) and **whole-file
|
||||
replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
|
||||
`kv_cache_utils.py`, platform files — these are version X's copies of files
|
||||
version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
|
||||
(`find … | xargs wc -l`), and say the additive-only ratio out loud before
|
||||
starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
|
||||
and the "missing" drafter was already registered in the target.
|
||||
|
||||
### 2. Enumerate the gap in ONE pass, not one ImportError at a time
|
||||
|
||||
Inventory the target tree first (`model_executor/layers/`,
|
||||
`v1/attention/backends/`, quantization backends, spec-decode drafters in
|
||||
`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
|
||||
staged package with `ast`, resolving every `from vllm… import X` against the
|
||||
target, split into MODULE-missing vs NAME-missing.
|
||||
|
||||
- **The double-prefix bug:** the dotted module already starts with `vllm.`,
|
||||
so joining onto the package dir needs
|
||||
`rel = mod.split(".", 1)[1].replace(".", os.sep)`.
|
||||
- **Static scans over-report:** bare annotations, TypeAlias and star
|
||||
re-exports resolve at runtime. Do a runtime check
|
||||
(`importlib.import_module` + `getattr`) before quoting numbers — one scan
|
||||
said 151/163 resolved; runtime said 10/13.
|
||||
- **Classify:** redirects dominate (version moved a symbol to a submodule) —
|
||||
a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
|
||||
them; **delete a block that empties** (a bare `from X import (` with an
|
||||
empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
|
||||
touched file.
|
||||
- **Port absent symbols with AST extraction**, not copy-paste: locate the
|
||||
`ClassDef`, check every base/decorator resolves in the target, append the
|
||||
exact segment, re-`ast.parse` the target, roll back on failure, skip if
|
||||
already defined.
|
||||
|
||||
### 3. Silent-default traps and transformers gates
|
||||
|
||||
- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
|
||||
key was renamed, the DEFAULT silently survives and the model computes fluent
|
||||
nonsense with no error. Diff every config key the old parser reads against
|
||||
the new `config.json`. When the old port used a fallback, the CHECKPOINT's
|
||||
value must win (measured: `num_experts_per_token` fallback 7 vs
|
||||
checkpoint `num_experts_per_tok=8`).
|
||||
- `model type X but Transformers does not recognize this architecture` =
|
||||
pinned transformers predates the model. Upgrading transformers inside a
|
||||
pinned vLLM image breaks things hard to attribute — prefer registering a
|
||||
genuine `PretrainedConfig` subclass:
|
||||
`CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
|
||||
round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
|
||||
If the target image's transformers is below vLLM's declared minimum,
|
||||
upgrading inside the image beats a shim.
|
||||
- **Hardware-gate flags from the port's own docs**: search the README for the
|
||||
arch name and "do not" — e.g. a native quant MoE backend that is
|
||||
CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
|
||||
~5x slower; pin those flags in the serve script with a comment citing the
|
||||
line.
|
||||
- **Backend selection:** default to `auto`, then READ the chosen-backend
|
||||
line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
|
||||
MLA + sparse attention — both surface as an init rejection. **"Using
|
||||
'EMULATION' backend" is a FAILED configuration, not a success** (weights
|
||||
dequantized per call: memory + throughput cost).
|
||||
- `Loading model from scratch…` for many minutes usually means the port
|
||||
SUCCEEDED and you are in weight placement — verify all layers constructed,
|
||||
both TP workers alive, zero errors, before chasing memory. And if it does
|
||||
not fit, **prove the band is empty**: sweep the knob and show no setting
|
||||
works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
|
||||
(abort on low RAM; do not let the OOM killer take co-tenants).
|
||||
|
||||
### 4. Shims that must survive worker spawn
|
||||
|
||||
vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
|
||||
by weight-load time. In order of preference:
|
||||
|
||||
1. **Patch the installed library's source file on disk** (reliable). Locate
|
||||
the real install path at runtime:
|
||||
`python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
|
||||
2. `.pth` in site-packages — works but runs at interpreter start and can
|
||||
import a package mid-import (circular import through `accelerate`);
|
||||
expect failure.
|
||||
3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
|
||||
`/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
|
||||
before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
|
||||
executed by `site.py`; `sitecustomize.py` is.)
|
||||
|
||||
Always confirm with a FRESH process — never with the process that set the
|
||||
patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).
|
||||
|
||||
**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
|
||||
memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
|
||||
Cannot be raised from inside a container or root's own shell; only PAM at
|
||||
login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
|
||||
2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
|
||||
failure you are chasing.
|
||||
|
||||
### 5. Ship as an idempotent patch-at-start applier
|
||||
|
||||
One `apply.sh` at container start, each step guarded by a grep-for-marker
|
||||
(re-entry safe), ending in a verification block that imports every installed
|
||||
symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
|
||||
load, and it turns "I think it works" into evidence.
|
||||
|
||||
---
|
||||
|
||||
## Part II — torch.compile / dynamo bring-up triage
|
||||
|
||||
### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)
|
||||
|
||||
1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
|
||||
suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
|
||||
tolerates a break.
|
||||
2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
|
||||
body to build its guard even on a hit ("Dynamo ignores the cache wrapper
|
||||
and directly traces the wrapped function"). Resolve the value ONCE into a
|
||||
plain module global and read that; a global read folds, a cached call does
|
||||
not. Verified in-image: warm lru_cache-calling function still fails
|
||||
fullgraph=True; a plain global read compiles.
|
||||
3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
|
||||
with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
|
||||
run.
|
||||
4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
|
||||
(untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
|
||||
`Unsupported: Read uninitialized cell` (a traced callee closes over a name
|
||||
bound after its definition) and `Skip calling torch.compiler.disable()'d
|
||||
function` under fullgraph=True. `torch.compiler.disable` is the documented
|
||||
escape hatch for host-side/stateful work (ContextVar reads, lazy
|
||||
allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
|
||||
ship in the stock path.
|
||||
5. **Trust the frame chain, not the line number** — dynamo INLINES user
|
||||
callees; an inlined failure reports the enclosing function's `def` line.
|
||||
"KeyError in forward at the def line" is really "a callee raised".
|
||||
`TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
|
||||
for dynamo-internal errors).
|
||||
|
||||
**The tell** that the trunk lacks compile support at all:
|
||||
|
||||
```
|
||||
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
|
||||
provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
|
||||
FULL_DECODE_ONLY.
|
||||
```
|
||||
|
||||
Diff against the family that DOES compile and copy its
|
||||
`@support_torch_compile` decorator including `dynamic_arg_dims` and any
|
||||
`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.
|
||||
|
||||
### Procedure
|
||||
|
||||
0. **Establish what configuration is ACTUALLY in effect**:
|
||||
`podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
|
||||
A launcher that "sets" an env var in a branch may silently drop it — four
|
||||
consecutive boots ran `fullgraph=True` while the operator believed
|
||||
otherwise, which makes the reported failure MOVE as breaks are removed and
|
||||
invites entirely wrong theories.
|
||||
1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
|
||||
+ `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
|
||||
candidate break under the flag-free config you actually intend — a break
|
||||
that appears only under the logging flag is a diagnostic artifact.
|
||||
2. **Fix in priority order:** (a) host syncs in the traced region
|
||||
(`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
|
||||
branch is not your blocker); (b) capability probes doing I/O in `forward`
|
||||
(`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
|
||||
warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
|
||||
a thin wrapper); (d) new kernel specializations — check the vendor ships
|
||||
SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
|
||||
makes the 288 you need unreachable. Fix at the PATTERN level (one
|
||||
launcher/decorator family breaking all 39 call sites), not site level.
|
||||
3. **KeyError on a bare module name** = dynamo's name→object map miss
|
||||
(`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
|
||||
read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
|
||||
Ship a probe via `sitecustomize.py` that patches
|
||||
`OutputGraph.example_value_from_input_node` and keeps weakrefs of live
|
||||
graphs, printing `node.target`, key count, near-miss keys — separates a
|
||||
cross-graph reference from a name-derivation mismatch.
|
||||
4. **Instrumenting a startup hook is a FIREARM — arm it once:**
|
||||
|
||||
```python
|
||||
if os.environ.get("PROBE_ARMED") == "1":
|
||||
pass # an ancestor already armed it -> stay inert
|
||||
else:
|
||||
os.environ["PROBE_ARMED"] = "1"
|
||||
...install probes...
|
||||
```
|
||||
|
||||
A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
|
||||
children inherit `PYTHONPATH`: an unguarded hook that imports the engine
|
||||
self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
|
||||
Kill the chain by removing the container whose cgroup owns every
|
||||
generation, then verify with a precise `/proc/<pid>/environ` count —
|
||||
NEVER pattern-kill an interpreter name on a host running python in
|
||||
production containers. Also chain the image's own sitecustomize so its
|
||||
hook survives.
|
||||
5. **Cheap in-image classification (no boot):**
|
||||
`g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
|
||||
container (with `--device /dev/kfd --device /dev/dri` — torch.compile
|
||||
queries a device even on the eager backend). ~5 s per question instead of
|
||||
~7 min boots.
|
||||
6. **Package as real patch artifacts:** hand-written `@@` headers are not
|
||||
diffs and `patch(1)` rejects them. Generate
|
||||
`diff -u --label a/<path> --label b/<path> orig new > name.patch` with
|
||||
`--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
|
||||
extraction of the pinned image and comparing sha256 (`sha256sum` must
|
||||
equal the `b/` hash recorded in the header). Header carries: STATUS, the
|
||||
measured evidence, base+patched sha256, how to apply, the NEXT blocker.
|
||||
|
||||
### Cost model before investing
|
||||
|
||||
**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
|
||||
device-activity rows) for the kernel table. Measured case: the long-suspected
|
||||
expert-gather was **1.05 %** of device time while **44 %** was unfused
|
||||
pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
|
||||
1,428/token) — a months-old "structural" conclusion died in one afternoon.
|
||||
Speculative decoding does NOT help a shape whose per-token cost scales with
|
||||
the token count, and it hurts prefill. Decide the ceiling up front: if the
|
||||
fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.
|
||||
|
||||
---
|
||||
|
||||
## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port
|
||||
|
||||
Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
|
||||
(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
|
||||
(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).
|
||||
|
||||
### The four laws
|
||||
|
||||
1. **Patch the instance the runner calls.**
|
||||
`set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
|
||||
after `model_loader.load_model`) runs on the LOADED instance — for a
|
||||
multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
|
||||
APIServer log line `Resolved architecture: …` names it), NOT the text
|
||||
model it holds as `self.language_model`. Patching only the inner
|
||||
`…ForCausalLM` reproduces the identical `RuntimeError: Model does not
|
||||
support EAGLE3 interface` forever. Patch BOTH wrappers.
|
||||
2. **The check is Protocol isinstance, NOT duck typing.**
|
||||
`supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
|
||||
(interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
|
||||
`SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
|
||||
(`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
|
||||
`getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
|
||||
members — and breaks on the FIRST miss. Full member set:
|
||||
`supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
|
||||
`set_aux_hidden_state_layers`,
|
||||
`get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
|
||||
presence; a methods-only implementation FAILS** because the three attrs
|
||||
are missing. Verified in-image with stock typing, no boot: flag-only →
|
||||
False; flag + two base members → True.
|
||||
3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
|
||||
wrapper already carrying Protocol-derived bases dies at class creation:
|
||||
`Cannot create a consistent method resolution order (MRO) for bases …` —
|
||||
MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
|
||||
position works. This failure also PROVES the isinstance-semantics question
|
||||
is moot: class creation precedes any check.
|
||||
4. **Write the two methods by hand** — without inheritance the Protocol
|
||||
defaults do not exist. Text model: unwrap to the trunk
|
||||
(`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
|
||||
delegate to `self.language_model` (the pixtral.py:498 pattern).
|
||||
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
|
||||
drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
|
||||
`[5, 14, 24, 33, 42]` of 45).
|
||||
|
||||
The fix shape (on EACH wrapper class):
|
||||
|
||||
```python
|
||||
supports_eagle3: ClassVar[Literal[True]] = True
|
||||
has_own_lm_head: bool = False # False when embed/lm_head are delegated
|
||||
has_own_embed_tokens: bool = False
|
||||
def set_aux_hidden_state_layers(self, layers): ...
|
||||
def get_eagle3_default_aux_hidden_state_layers(self): ...
|
||||
```
|
||||
|
||||
### Trunk mechanics
|
||||
|
||||
The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
|
||||
`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
|
||||
Capture aux states in the layer loop: the state ENTERING each requested
|
||||
layer, BEFORE its input norm (GLM's loop carries the single-stream state;
|
||||
deepseek_v2's equivalent is `hidden_states + residual`).
|
||||
`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
|
||||
methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
|
||||
`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
|
||||
consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
|
||||
|
||||
### Window discipline and verification
|
||||
|
||||
- Scratch boots on dankserver MUST stop + runtime-mask production
|
||||
`frank-primary` first — the expert-offload planner reads `MemAvailable` at
|
||||
plan time, and beside-prod boots die at plan with
|
||||
`ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
|
||||
loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
|
||||
mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
|
||||
return — a new window started during restore inherits the false failure).
|
||||
- Never launch a window while the previous window's RESTORE is in flight
|
||||
(`systemctl start` returns immediately; the engine loads ~6 min).
|
||||
- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
|
||||
decision greps truncate lines at `…` and tracebacks die with the container.
|
||||
- Cheap verification, no boot: AST-parse the patched file for class bases +
|
||||
member presence; MRO simulation with stub Protocols; in-image `typing`
|
||||
replication of the isinstance walk (pure stdlib — importing vllm itself
|
||||
crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
|
||||
auxiliary layers from config: (…)` = gate cleared.
|
||||
- Honest expectations: a target-side DFlash2/EAGLE3 port on a
|
||||
pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
|
||||
the verification pass costs ∝ its token count. Port it for capability, not
|
||||
speed.
|
||||
|
||||
## Related
|
||||
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
|
||||
- [[RDNA4 Container Serving]] — the serving stack being ported onto
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
|
||||
- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes
|
||||
Reference in new issue
Block a user