docs: dflash KV journey — 03 verified (drafter graphs capture), 04 capacity/aliasing wall

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -436,15 +436,63 @@ construction — where the next two blockers live, both in
    so `disable_hybrid_kv_cache_manager` is unset and
    `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
    (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
-   growing its block) is the fix; supersedes 02 in the repo.
+   growing its block) **verified 2026-09-30**: the assert is gone and the boot
+   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
+   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
 
-Both patches mount `v1/core/kv_cache_utils.py` over site-packages via the
+3. **The capacity wall: the generic path cannot alias the mamba states.**
-scratch launcher's `KVUNIFY_PATCH=1` knob (host
+   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
-`/home/dank/dflash-patch2/new/vllm/v1/core/kv_cache_utils.py`, patched sha
+   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
-`5093e4b7…`; apply-verified against the image's pristine file). The window
+   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
-script's decision pipeline was also fixed for a SIGPIPE death (`head -80` on a
+   per-token KV). Two coupled causes, read out of the image:
-`podman logs` grep pipeline killed one run before log-save and restore; the log
+   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
-is now captured to a file FIRST, then grepped).
+     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
+     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
+     expert plan's KV registry is calibrated at 13,600 B/token). On the
+     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
+     unaliased state pages appears.
+   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
+     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
+     KV at the plan's reserve, so even a correct generic-path estimate would
+     be starved.
+
+   Learned en route: the engine AUTO-SETS the attention block size to
+   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
+   tokens to ensure that attention page size is >= mamba page size`), so every
+   attention group's block counts scale from that floor — a 16-token drafter
+   block would need 4x the shared `num_blocks`; with the global 2,432 all
+   groups align.
+
+   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
+   for the registry) keeps the aliased glm5 pool and extends it: the grouping
+   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
+   `MLAAttentionSpec` subclass — excluded before the exact-type
+   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
+   specs; the drafter layers form their own uniform group (per-group pages,
+   NO unification — the MLA block size stays at the engine-set 2,432 and the
+   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
+   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
+   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
+   equal) as aliased pool + appended draft/hidden tensors with their own
+   strides; the memory estimator adds the extras. Patch 04b extends the
+   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
+   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
+   `speculative_config.use_aux_hidden_state_outputs` is set.
+
+   The pure logic is smoke-tested locally (CPU torch + the image's real spec
+   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
+   layout recognition, builder block math and memory accounting all pass.
+   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
+   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
+   2,432 tokens ≈ 70,500 ≥ 65,536.
+
+All of these mount over site-packages via the scratch launcher's
+`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
+`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
+`expert_activation_registry.py`; apply-verified against the image's pristine
+files). The window script's decision pipeline was also fixed for a SIGPIPE
+death (`head -80` on a `podman logs` grep pipeline killed one run before
+log-save and restore; the log is now captured to a file FIRST, then grepped).
 
 
 ## Measured arms (all 2026-09-30, one boot each)