docs: frank-large vLLM MiMo-BF16 runbook; retire the gufo runbook

New runbook carries the roof math, the measured three-arm spec table, the
greedy-text numerics trap, the reasoning-field contract change and the
rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook
index follow; the frank-small index entries catch up to the SGLang cutover
(entry was missed on 2026-10-01). Work log records the cutover.
This commit is contained in:
Frank committed 2026-10-01 18:48:51 -07:00
1 parent dab483c79f
commit 1127a52078
5 files changed
+330 -133

No files matched your search

+37
View File
@@ -875,3 +875,40 @@ any repo service.
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
contract probes (reasoning split, tool_calls). Rollback target for the
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
geometry) works. Frank's call: ship that as frank-large.
- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
text (bf16 batched verification reorders reductions; near-tie argmaxes
cascade). vLLM spec decode is distribution-preserving, not
bitwise-greedy-preserving.
- **Client-contract delta wired:** vLLM emits `reasoning`, not
`reasoning_content` - `models.yml` frank-large block now sets
`reasoningContentField: "reasoning"`, and the extraBody moved to the
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
(a top-level `preserve_thinking` is ignored by vLLM).
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
`model.safetensors` (the dflash layout) and would have rejected the
deployed MTP arm (4 shards behind an index); fixed to accept both
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
0.30 (verified in-image); the drafter stays on disk as the alternative.
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
Q4_K_XL payload and DFlash2 drafter all remain on the host.