docs: LLM index + work log for frank-primary shape D, harness window
index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and frank-small at 2 x 262144; the active-deployment list had no frank-primary entry at all. Fixed both, plus a malformed GLM list item. log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever measurement, the 2x2 probe table, the measured 1.99x), and the frank-small entry's stale consequence bullet now points at the resolution. frank-small runbook: the compaction-fit note is corrected - the 200704 window moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
This commit is contained in:
@@ -971,9 +971,10 @@ the 3090", then **"frank-small should have vision"**.
|
||||
- **Three contexts cost window, and that is the tradeoff Frank was told:**
|
||||
180,224 tokens per conversation, not the native 262,144. A longer prompt is
|
||||
a hard `400 exceed_context_size_error`, so `models.yml` sets
|
||||
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
|
||||
compaction request no longer fits here with summary room; the two-slot
|
||||
rollback is the way back.
|
||||
`contextWindow: 180224`. Consequence at the time: frank-primary's then
|
||||
183,501-token compaction request did not fit here with summary room —
|
||||
resolved later the same day when frank-primary's window moved to 200,704
|
||||
and its trigger fell to 140,492 (see the shape-D entry below).
|
||||
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
|
||||
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
|
||||
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
|
||||
@@ -993,3 +994,79 @@ the 3090", then **"frank-small should have vision"**.
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
|
||||
## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
|
||||
|
||||
Frank's order: shrink frank-primary to a 200704-token window with about two
|
||||
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
|
||||
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
|
||||
went live as `e460bbb`, and the shape was then re-applied one law at a time on
|
||||
top of the proven-good base, root cause first.
|
||||
|
||||
- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM,
|
||||
never MNBT, never the KV dtype.** Four single-variable boot probes against
|
||||
the then-deployed shape (production stopped for each, candidate booted AS the
|
||||
service):
|
||||
|
||||
| MAXLEN | SEQS | MNBT | result |
|
||||
|---|---|---|---|
|
||||
| 262144 | 2 | 1024 | **Engine core initialization failed** |
|
||||
| 262144 | 2 | 2048 | **Engine core initialization failed** |
|
||||
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
|
||||
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
|
||||
|
||||
`torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints
|
||||
violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant
|
||||
(3)`, raised from AOT `build_guards(strict_error=True)` during
|
||||
`determine_available_memory -> profile_run -> _dummy_run`, ~7 min in.
|
||||
`model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run`
|
||||
sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is
|
||||
**SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the
|
||||
constant in the message. MNBT only moves the compile RANGE ((1,1024) vs
|
||||
(1,2048)), which is why the identical error appears at both rungs.
|
||||
`--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01
|
||||
bf16 2 x 262144 failure was this same error class; its message named "KV
|
||||
dtype and SEQS" together and nobody bisected, so the dtype was blamed - it
|
||||
was a red herring.
|
||||
- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The
|
||||
planner sizes expert cache as `budget - non-expert - KV reserve - graphs -
|
||||
overhead`, and the KV reserve is registry-priced at **9190 B/token x
|
||||
max_model_len**, independent of the pool override:
|
||||
|
||||
| MAXLEN | KV reserve | arena |
|
||||
|---|---|---|
|
||||
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
|
||||
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
|
||||
|
||||
d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's
|
||||
premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool
|
||||
cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes
|
||||
free` on a real 995-token request).
|
||||
- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV,
|
||||
`--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no
|
||||
speculative decode, prefetch off, no `--expert-precise-memory`. The argv
|
||||
keeps the only slot count that compiles; the POOL caps resident
|
||||
conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33.
|
||||
- **Measured live, and one prediction corrected.** `expert cache space 14.87
|
||||
GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache
|
||||
size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request:
|
||||
1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted:
|
||||
**the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN
|
||||
262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 %
|
||||
dearer. Never extrapolate a concurrency figure across geometries.
|
||||
- **Smoke-verified on the live endpoint:** three concurrent 67-token requests
|
||||
answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls`
|
||||
(`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean
|
||||
`content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers
|
||||
400 for anything else (re-measured; the fork's message names the set).
|
||||
- **Harness contract followed the window:** `models.yml` `contextWindow`
|
||||
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
|
||||
200,704). Side effect: frank-primary's compaction request now fits inside a
|
||||
frank-small slot again, which it had stopped doing when frank-small went to
|
||||
3 x 180,224.
|
||||
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
|
||||
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
|
||||
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
|
||||
`frank-primary-fp8-mtp-20261001`. Runbook
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
Reference in New Issue
Block a user