docs: frank-large vLLM MiMo-BF16 runbook; retire the gufo runbook
New runbook carries the roof math, the measured three-arm spec table, the greedy-text numerics trap, the reasoning-field contract change and the rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook index follow; the frank-small index entries catch up to the SGLang cutover (entry was missed on 2026-10-01). Work log records the cutover.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
description: "The RETIRED frank-large runbook (2026-09-28 → 2026-10-01, rollback target via tag frank-large-gufo-q4kxl-20261001) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
@@ -18,7 +18,11 @@ timestamp: 2026-09-28
|
||||
|
||||
# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-28.** Replaces
|
||||
> **RETIRED 2026-10-01** — replaced by [[frank-large-vllm-mimo-v26-bf16]]
|
||||
> (vLLM TheRock + MiMo-V2.6 BF16). Rollback: git tag
|
||||
> `frank-large-gufo-q4kxl-20261001`; the gufo build (`build-gufo.sh`), the
|
||||
> Q4_K_XL payload and the DFlash2 drafter all remain on the host.
|
||||
> **Originally ACTIVE 2026-09-28 → 2026-10-01.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP"
|
||||
description: "The active frank-large runbook — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU: identity, the measured roof math, the cross-model Qwen3.5-9B MTP drafter and its acceptance curve, the greedy-text numerics trap, the client-contract field change, deployed flags, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- dankclaw
|
||||
- vllm
|
||||
- strix-halo
|
||||
- mimo
|
||||
- mtp
|
||||
- spec-decode
|
||||
- bf16
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||||
|
||||
> **ACTIVE since 2026-10-01.** Replaces [[frank-large-gufo-qwen38-27b]]
|
||||
> (gufo + Qwen3.8-27B GGUF, now the rollback target, git tag
|
||||
> `frank-large-gufo-q4kxl-20261001`). Same port (7770), same served alias
|
||||
> (`frank-large`). This is the fleet's second vLLM endpoint and the first
|
||||
> CROSS-MODEL speculative-decode deployment: the target is Xiaomi's distill,
|
||||
> the draft head is Qwen's.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
|
||||
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
|
||||
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
|
||||
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
|
||||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||||
are loaded into the draft pass).
|
||||
|
||||
## Why this shape (the roof math)
|
||||
|
||||
Decode speed for a dense model on Strix Halo is bandwidth-bound:
|
||||
BF16 weights x one pass per token against the LPDDR5x aperture. Measured
|
||||
2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
|
||||
a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
|
||||
because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
|
||||
wall is generating more than one token per weight pass: **speculative decode
|
||||
is the performance feature of this endpoint**, not a garnish.
|
||||
|
||||
The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
|
||||
attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
|
||||
the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
|
||||
sequence. `--max-model-len 262144` (the native window) with
|
||||
`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
|
||||
four-session contract the gufo shape ran, with the pool shared by the
|
||||
scheduler instead of carved per session.
|
||||
|
||||
## Spec method: cross-model MTP — measured, not assumed
|
||||
|
||||
| arm | prefill | decode | acceptance |
|
||||
|---|---|---|---|
|
||||
| BF16 base | 17,876 t/s | 13.1 t/s | — |
|
||||
| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
|
||||
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
|
||||
|
||||
One session, temperature 0, 28,647-token prefill / 512-token decode, same
|
||||
client for all three arms (2026-10-01 bench window, frank-large stopped per
|
||||
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
|
||||
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
|
||||
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
|
||||
`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
|
||||
fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
|
||||
the compute queues) — this endpoint's thinking-heavy traffic spends its wall
|
||||
clock in decode, which is the regime where that trade wins.
|
||||
|
||||
**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
|
||||
Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||||
4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
|
||||
per-position acceptance is the direct measurement that the distillation
|
||||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||||
why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||||
separate path. That difference is the whole trick:
|
||||
|
||||
```
|
||||
--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
|
||||
```
|
||||
|
||||
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
|
||||
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
|
||||
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||||
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
|
||||
|
||||
## The numerics trap (greedy text is NOT preserved by spec decode)
|
||||
|
||||
Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
|
||||
`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
|
||||
reduction order, argmax near-ties diverge, and the divergence cascades.
|
||||
vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
|
||||
The gufo-era "greedy == autoregressive text" law does not carry over; do not
|
||||
chase text drift between spec and no-spec arms as a regression.
|
||||
|
||||
## Payload
|
||||
|
||||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||||
byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh`
|
||||
(`--verify-only` re-checks; no token on the curl command line).
|
||||
- MTP source: `scripts/download-frank-large-qwen35.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||||
the engine boots from exactly this tree).
|
||||
- DFlash drafter: `scripts/download-frank-large-dflash.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||||
- The serve script gates the image by REPO DIGEST (rootless podman resolves
|
||||
tags from the network; the pin exists to boot exactly these bytes
|
||||
offline), then config + index + exact shard count, then the spec draft
|
||||
(either weight layout: single `model.safetensors` or a sharded index),
|
||||
then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
|
||||
Dankserver 2026-09-29 lesson).
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
|
||||
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
|
||||
`models.yml` compat block for this provider sets
|
||||
`reasoningContentField: "reasoning"`. Miss that and thinking silently
|
||||
vanishes from transcripts.
|
||||
- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
|
||||
- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
|
||||
serve script passes `--language-model-only` — the endpoint stays
|
||||
text-only by decision; dropping the flag brings vision back.
|
||||
- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
|
||||
(temp 0.7, top_p 0.8) via `compat.extraBody`, and
|
||||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
|
||||
`chat_template_kwargs` form is what vLLM reads; a top-level
|
||||
`preserve_thinking` is ignored there.
|
||||
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
|
||||
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
|
||||
the env file with `set -a`.
|
||||
|
||||
## Ops
|
||||
|
||||
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
|
||||
- The serve script sweeps stale container identities before starting:
|
||||
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
|
||||
- Container death → `podman wait` returns non-zero from ExecStart →
|
||||
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
|
||||
frank-primary lesson).
|
||||
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
|
||||
relinked unit; the deploy restores).
|
||||
- First-deploy fact to record here: the boot-time KV pool line
|
||||
(`Maximum concurrency for 262,144 tokens per prompt: N.Nx`).
|
||||
|
||||
## Rollback
|
||||
|
||||
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||||
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
|
||||
payload + DFlash2 drafter, all still on the host. Restoring the tag's
|
||||
unit/env/serve trio and redeploying returns the 27B endpoint.
|
||||
|
||||
# Citations
|
||||
|
||||
- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
|
||||
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
|
||||
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
|
||||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||||
[[frank-small-sglang-mimo-v26]]
|
||||
@@ -10,12 +10,12 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints —
|
||||
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
|
||||
unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
|
||||
on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
|
||||
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
|
||||
port 7771).
|
||||
runs two endpoints on the same arch —
|
||||
vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large,
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP
|
||||
spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx,
|
||||
port 7770) and SGLang (frank-small, the same model as an INT8 W8A16
|
||||
AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -39,14 +39,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[stock llama.cpp (unsloth b11030, gfx1151)
|
||||
E[vLLM TheRock gfx1151, digest-pinned
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
|
||||
262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head
|
||||
4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers]
|
||||
G[SGLang venv cu130
|
||||
frank-small, port 7771]
|
||||
H[Gemma4-12B QAT Q4_K_M
|
||||
4 x 262144 ctx, MTP + vision + audio]
|
||||
H[MiMo-V2.6 INT8 W8A16 AutoRound
|
||||
2 x 262144 ctx, pinned pool 606208, in-checkpoint vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -162,14 +162,23 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
|
||||
Strix Halo inference engine, built from source on Dankclaw, serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
|
||||
Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
|
||||
262,144-token conversation, greedy, text-only, the two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode trade
|
||||
against llama.cpp, troubleshooting
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since
|
||||
2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B
|
||||
(the fleet's first; 71%/44% per-position acceptance measured) —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics
|
||||
trap, the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git
|
||||
tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on
|
||||
the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine,
|
||||
built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL
|
||||
with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two
|
||||
from-source build workarounds, the client-contract delta, the measured
|
||||
prefill/decode trade against llama.cpp
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -203,12 +212,18 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since
|
||||
2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained
|
||||
python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-
|
||||
AutoRound on the RTX 3090 — the same architecture frank-large now serves
|
||||
in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool,
|
||||
NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision.
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]]; rollback target, payload and binary
|
||||
intact on disk): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
|
||||
vLLM detour (2026-09-19) was reverted the same day; see below.
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
|
||||
@@ -875,3 +875,40 @@ any repo service.
|
||||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||||
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
|
||||
|
||||
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||||
|
||||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||||
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
|
||||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||||
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
|
||||
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
|
||||
geometry) works. Frank's call: ship that as frank-large.
|
||||
- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
|
||||
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
|
||||
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
|
||||
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
|
||||
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
|
||||
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
|
||||
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
|
||||
text (bf16 batched verification reorders reductions; near-tie argmaxes
|
||||
cascade). vLLM spec decode is distribution-preserving, not
|
||||
bitwise-greedy-preserving.
|
||||
- **Client-contract delta wired:** vLLM emits `reasoning`, not
|
||||
`reasoning_content` - `models.yml` frank-large block now sets
|
||||
`reasoningContentField: "reasoning"`, and the extraBody moved to the
|
||||
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
|
||||
(a top-level `preserve_thinking` is ignored by vLLM).
|
||||
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
|
||||
`model.safetensors` (the dflash layout) and would have rejected the
|
||||
deployed MTP arm (4 shards behind an index); fixed to accept both
|
||||
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
|
||||
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
|
||||
0.30 (verified in-image); the drafter stays on disk as the alternative.
|
||||
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
|
||||
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
|
||||
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
|
||||
Q4_K_XL payload and DFlash2 drafter all remain on the host.
|
||||
|
||||
Reference in New Issue
Block a user