frank-small: reslice to 2 x 262,144 native slots (was 4 x 151,552)

The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.

Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.

No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.
This commit is contained in:
Frank
2026-10-01 14:03:25 -07:00
parent 389b605baf
commit 35e24db7e6
4 changed files with 35 additions and 21 deletions
@@ -39,11 +39,15 @@ timestamp: 2026-10-01
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
tower (27 layers) in the safetensors — no separate mmproj file.
- **Context:** four 151,552-token slots (`--context-length 151552`,
`--max-running-requests 4`) against a **pinned** 606,208-token pool
(`--max-total-tokens`). A single request may grow to the whole pool —
a 170,019-token prompt is verified — the grid is the four-way
concurrency contract.
- **Context:** two 262,144-token slots at the model's native window
(`--context-length 262144`, `--max-running-requests 2`) against a
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
~82k tokens of pool slack, and a compaction request (frank-primary's
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
one request. A single request may grow to the whole pool — a
170,019-token prompt was verified at the 4-slot shape — the grid is
the two-way concurrency contract.
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet