docs: LLM index + work log for frank-primary shape D, harness window

index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and
frank-small at 2 x 262144; the active-deployment list had no frank-primary
entry at all. Fixed both, plus a malformed GLM list item.

log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever
measurement, the 2x2 probe table, the measured 1.99x), and the frank-small
entry's stale consequence bullet now points at the resolution.

frank-small runbook: the compaction-fit note is corrected - the 200704 window
moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
This commit is contained in:
Frank
2026-10-02 12:05:39 -07:00
parent 295d5784d3
commit 656012a0fa
4 changed files with 115 additions and 26 deletions
+80 -3
View File
@@ -971,9 +971,10 @@ the 3090", then **"frank-small should have vision"**.
- **Three contexts cost window, and that is the tradeoff Frank was told:**
180,224 tokens per conversation, not the native 262,144. A longer prompt is
a hard `400 exceed_context_size_error`, so `models.yml` sets
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
compaction request no longer fits here with summary room; the two-slot
rollback is the way back.
`contextWindow: 180224`. Consequence at the time: frank-primary's then
183,501-token compaction request did not fit here with summary room —
resolved later the same day when frank-primary's window moved to 200,704
and its trigger fell to 140,492 (see the shape-D entry below).
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
@@ -993,3 +994,79 @@ the 3090", then **"frank-small should have vision"**.
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
Frank's order: shrink frank-primary to a 200704-token window with about two
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
went live as `e460bbb`, and the shape was then re-applied one law at a time on
top of the proven-good base, root cause first.
- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM,
never MNBT, never the KV dtype.** Four single-variable boot probes against
the then-deployed shape (production stopped for each, candidate booted AS the
service):
| MAXLEN | SEQS | MNBT | result |
|---|---|---|---|
| 262144 | 2 | 1024 | **Engine core initialization failed** |
| 262144 | 2 | 2048 | **Engine core initialization failed** |
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
`torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints
violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant
(3)`, raised from AOT `build_guards(strict_error=True)` during
`determine_available_memory -> profile_run -> _dummy_run`, ~7 min in.
`model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run`
sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is
**SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the
constant in the message. MNBT only moves the compile RANGE ((1,1024) vs
(1,2048)), which is why the identical error appears at both rungs.
`--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01
bf16 2 x 262144 failure was this same error class; its message named "KV
dtype and SEQS" together and nobody bisected, so the dtype was blamed - it
was a red herring.
- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The
planner sizes expert cache as `budget - non-expert - KV reserve - graphs -
overhead`, and the KV reserve is registry-priced at **9190 B/token x
max_model_len**, independent of the pool override:
| MAXLEN | KV reserve | arena |
|---|---|---|
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's
premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool
cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes
free` on a real 995-token request).
- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV,
`--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no
speculative decode, prefetch off, no `--expert-precise-memory`. The argv
keeps the only slot count that compiles; the POOL caps resident
conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33.
- **Measured live, and one prediction corrected.** `expert cache space 14.87
GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache
size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request:
1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted:
**the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN
262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 %
dearer. Never extrapolate a concurrency figure across geometries.
- **Smoke-verified on the live endpoint:** three concurrent 67-token requests
answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls`
(`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean
`content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers
400 for anything else (re-measured; the fork's message names the set).
- **Harness contract followed the window:** `models.yml` `contextWindow`
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
200,704). Side effect: frank-primary's compaction request now fits inside a
frank-small slot again, which it had stopped doing when frank-small went to
3 x 180,224.
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
`frank-primary-fp8-mtp-20261001`. Runbook
[[dankserver-qwen38-fp8-expert-offload-tuning]].