diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index 2b0da93..270873a 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -348,30 +348,49 @@ flat decode with a prefill regression like MTP's until the pointwise mass is fus **The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP, independent of the compile changelist — it patches the same file as compile-01, so -compile-01 goes first if both are applied; base `008eeaa2…` → patched `5eea89c5…`). It +compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It implements exactly the contract read out of this image: `EagleModelMixin` on `Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and -`SupportsEagle3` among `Glm5NextForCausalLM`'s bases (capture shape from -`deepseek_v2.py:1551-1585`; tuple consumption at `v1/worker/gpu/cudagraph_utils.py:621`). +the three `SupportsEagle3` member attributes plus both interface methods on the +wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at +`v1/worker/gpu/cudagraph_utils.py:621`). -**The first version of that patch was wrong in an instructive way, and it is the trap to -avoid on any future spec-decode port: `supports_eagle3` is a Protocol `isinstance` -check, not duck typing.** It hand-wrote `set_aux_hidden_state_layers` and -`get_eagle3_default_aux_hidden_state_layers` on the outer wrapper — the two methods the -traceback's call site reads — and the boot reproduced the *identical* -`RuntimeError: Model does not support EAGLE3 interface`. The reason is -`interfaces.py:1663`: `SupportsEagle3` also declares -`supports_eagle3: ClassVar[Literal[True]] = True`, and a runtime-checkable Protocol -requires every member to be *present* on the instance. **Inherit the Protocol** — the -idiom every model in the image uses (`LagunaForCausalLM`, `MiMoV2FlashForCausalLM`, -`DFlashLagunaForCausalLM` all list `SupportsEagle3` in their bases) — which supplies the -flag, `SupportsEagleBase`'s `has_own_lm_head` / `has_own_embed_tokens`, *and* working -default implementations of both methods (they unwrap -`holder = getattr(parent_ref, "model", parent_ref)`, assert it is an `EagleModelMixin`, -and default to `(2, L // 2, L - 3)`). The lesson generalizes: when a runtime error names -an interface, read the interface's declaration first — the methods a call site happens to -invoke are not necessarily the whole contract. +**The patch went through five versions, and the decisive discovery was where the +instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)` +is called on the model the runner LOADED — and for GLM-5.3-Flash that is +`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it), +which holds the text model as `self.language_model = Glm5NextForCausalLM` +(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed +with the identical error because the `isinstance` ran against the wrapper. **v5 fixes +both wrappers** and is the first version to clear the gate (both TP workers log +`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`). + +The full evolution, each an instructive trap: + +1. **v1** hand-wrote the two methods the traceback names → identical error. The methods + are not the whole contract. +2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`, + `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method + resolution order (MRO) for bases …`. GLM's wrappers already carry four/five + Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering + cannot take `SupportsEagle3` in *any* position. +3. **v3** declared the members as class attributes (MRO-safe) — but only on + `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above). +4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`, + `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL + `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified + experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error. +5. **v5** adds the three attributes to BOTH wrappers + the two methods on both + (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk; + `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the + `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**. + +**The lesson generalizes twice over**: when a runtime error names an interface, read the +interface's declaration first — and establish WHICH INSTANCE the check runs against +before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded +models, not pass-throughs, and the text model behind `self.language_model` is invisible +to a check run on the wrapper. **Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An `op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 00d055e..f246d28 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -66,14 +66,22 @@ capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py` demonstrates). No speedup is budgeted: a verification pass pays pointwise cost proportional to its token count. - - **DFlash2 tried, and the gap is exactly that one interface.** Payload fetched - and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`, 2,342,169,800 B, - sha256 `b038e1d9…` = the Hub's LFS oid). Booting with + - **DFlash2 tried, and the gap was exactly that one interface — now closed.** + Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`, + 2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'` - accepts the config and the drafter and dies at worker init with - `RuntimeError: Model does not support EAGLE3 interface` — nothing about the - drafter, the method or the payload is wrong; the trunk simply does not - implement the aux-hidden-state contract. + accepted the config and the drafter and died at worker init with + `RuntimeError: Model does not support EAGLE3 interface`. The port + (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions — + hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the + text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH + wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal + `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`; + `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP + workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the + DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice + by experiment-window RAM contention (see the window discipline below), so the + drafter-serving end state was still unproven when this entry was written. - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.** With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots