docs: frank-large vLLM MiMo-BF16 runbook; retire the gufo runbook
New runbook carries the roof math, the measured three-arm spec table, the greedy-text numerics trap, the reasoning-field contract change and the rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook index follow; the frank-small index entries catch up to the SGLang cutover (entry was missed on 2026-10-01). Work log records the cutover.
This commit is contained in:
1 parent
dab483c79f
commit
1127a52078
5 files changed
+330
-133
No files matched your search
@@ -875,3 +875,40 @@ any repo service.
|
||||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||||
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
|
||||
|
||||
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||||
|
||||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||||
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
|
||||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||||
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
|
||||
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
|
||||
geometry) works. Frank's call: ship that as frank-large.
|
||||
- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
|
||||
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
|
||||
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
|
||||
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
|
||||
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
|
||||
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
|
||||
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
|
||||
text (bf16 batched verification reorders reductions; near-tie argmaxes
|
||||
cascade). vLLM spec decode is distribution-preserving, not
|
||||
bitwise-greedy-preserving.
|
||||
- **Client-contract delta wired:** vLLM emits `reasoning`, not
|
||||
`reasoning_content` - `models.yml` frank-large block now sets
|
||||
`reasoningContentField: "reasoning"`, and the extraBody moved to the
|
||||
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
|
||||
(a top-level `preserve_thinking` is ignored by vLLM).
|
||||
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
|
||||
`model.safetensors` (the dflash layout) and would have rejected the
|
||||
deployed MTP arm (4 shards behind an index); fixed to accept both
|
||||
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
|
||||
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
|
||||
0.30 (verified in-image); the drafter stays on disk as the alternative.
|
||||
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
|
||||
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
|
||||
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
|
||||
Q4_K_XL payload and DFlash2 drafter all remain on the host.
|
||||
Reference in new issue
Block a user