diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index dd79bde..fb43186 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -436,15 +436,63 @@ construction — where the next two blockers live, both in so `disable_hybrid_kv_cache_manager` is unset and `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`** (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever - growing its block) is the fix; supersedes 02 in the repo. + growing its block) **verified 2026-09-30**: the assert is gone and the boot + captures the DFlash2 drafter's FULL cuda graphs on both TP workers + (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo. -Both patches mount `v1/core/kv_cache_utils.py` over site-packages via the -scratch launcher's `KVUNIFY_PATCH=1` knob (host -`/home/dank/dflash-patch2/new/vllm/v1/core/kv_cache_utils.py`, patched sha -`5093e4b7…`; apply-verified against the image's pristine file). The window -script's decision pipeline was also fixed for a SIGPIPE death (`head -80` on a -`podman logs` grep pipeline killed one run before log-save and restore; the log -is now captured to a file FIRST, then grepped). +3. **The capacity wall: the generic path cannot alias the mamba states.** + Past capture, the boot dies at `_check_enough_kv_cache_memory`: + `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len + 65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not + per-token KV). Two coupled causes, read out of the image: + - **The glm5 tensor layout aliases the mamba states INTO the MLA pages** + (`get_kv_cache_config_from_groups` places every mamba layer at the MLA + offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the + expert plan's KV registry is calibrated at 13,600 B/token). On the + generic path the mamba groups claim their OWN blocks, so ~30 GiB of + unaliased state pages appears. + - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`, + a flat 13,600 B/token) knows nothing about the drafter.** The engine caps + KV at the plan's reserve, so even a correct generic-path estimate would + be starved. + + Learned en route: the engine AUTO-SETS the attention block size to + **2,432 tokens** (`interface.py:918: Setting attention block size to 2432 + tokens to ensure that attention page size is >= mamba page size`), so every + attention group's block counts scale from that floor — a 16-token drafter + block would need 4x the shared `num_blocks`; with the global 2,432 all + groups align. + + **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`** + for the registry) keeps the aliased glm5 pool and extends it: the grouping + partitions the hidden-state specs (`HiddenStateCacheSpec` is an + `MLAAttentionSpec` subclass — excluded before the exact-type + `type(spec) is MLAAttentionSpec` partition) and the drafter attention + specs; the drafter layers form their own uniform group (per-group pages, + NO unification — the MLA block size stays at the engine-set 2,432 and the + kpool guard `block_size % 512 == 0` is never stressed); the layout returns + a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE + shared backing allocation (`allocate_kv_cache` asserts all tensor sizes + equal) as aliased pool + appended draft/hidden tensors with their own + strides; the memory estimator adds the extras. Patch 04b extends the + registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) + + 5 x hidden_size x 2 B (the aux hidden-state cache) per token when + `speculative_config.use_aux_hidden_state_outputs` is set. + + The pure logic is smoke-tested locally (CPU torch + the image's real spec + classes against synthetic GLM+draft+hidden spec sets): grouping shape, + layout recognition, builder block math and memory accounting all pass. + Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len + 65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x + 2,432 tokens ≈ 70,500 ≥ 65,536. + +All of these mount over site-packages via the scratch launcher's +`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas +`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for +`expert_activation_registry.py`; apply-verified against the image's pristine +files). The window script's decision pipeline was also fixed for a SIGPIPE +death (`head -80` on a `podman logs` grep pipeline killed one run before +log-save and restore; the log is now captured to a file FIRST, then grepped). ## Measured arms (all 2026-09-30, one boot each)