Frank
1127a52078
New runbook carries the roof math, the measured three-arm spec table, the greedy-text numerics trap, the reasoning-field contract change and the rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook index follow; the frank-small index entries catch up to the SGLang cutover (entry was missed on 2026-10-01). Work log records the cutover.
177 lines
9.0 KiB
Markdown
177 lines
9.0 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP"
|
||
description: "The active frank-large runbook — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU: identity, the measured roof math, the cross-model Qwen3.5-9B MTP drafter and its acceptance curve, the greedy-text numerics trap, the client-contract field change, deployed flags, and what to check when it misbehaves."
|
||
tags:
|
||
- llm
|
||
- frank-large
|
||
- dankclaw
|
||
- vllm
|
||
- strix-halo
|
||
- mimo
|
||
- mtp
|
||
- spec-decode
|
||
- bf16
|
||
timestamp: 2026-10-01
|
||
---
|
||
|
||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||
|
||
> **ACTIVE since 2026-10-01.** Replaces [[frank-large-gufo-qwen38-27b]]
|
||
> (gufo + Qwen3.8-27B GGUF, now the rollback target, git tag
|
||
> `frank-large-gufo-q4kxl-20261001`). Same port (7770), same served alias
|
||
> (`frank-large`). This is the fleet's second vLLM endpoint and the first
|
||
> CROSS-MODEL speculative-decode deployment: the target is Xiaomi's distill,
|
||
> the draft head is Qwen's.
|
||
|
||
## Identity
|
||
|
||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
|
||
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
|
||
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
|
||
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
|
||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||
(17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
|
||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||
are loaded into the draft pass).
|
||
|
||
## Why this shape (the roof math)
|
||
|
||
Decode speed for a dense model on Strix Halo is bandwidth-bound:
|
||
BF16 weights x one pass per token against the LPDDR5x aperture. Measured
|
||
2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
|
||
a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
|
||
because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
|
||
wall is generating more than one token per weight pass: **speculative decode
|
||
is the performance feature of this endpoint**, not a garnish.
|
||
|
||
The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
|
||
attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
|
||
the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
|
||
sequence. `--max-model-len 262144` (the native window) with
|
||
`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
|
||
four-session contract the gufo shape ran, with the pool shared by the
|
||
scheduler instead of carved per session.
|
||
|
||
## Spec method: cross-model MTP — measured, not assumed
|
||
|
||
| arm | prefill | decode | acceptance |
|
||
|---|---|---|---|
|
||
| BF16 base | 17,876 t/s | 13.1 t/s | — |
|
||
| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
|
||
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
|
||
|
||
One session, temperature 0, 28,647-token prefill / 512-token decode, same
|
||
client for all three arms (2026-10-01 bench window, frank-large stopped per
|
||
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
|
||
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
|
||
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
|
||
`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
|
||
fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
|
||
the compute queues) — this endpoint's thinking-heavy traffic spends its wall
|
||
clock in decode, which is the regime where that trade wins.
|
||
|
||
**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
|
||
Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||
4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
|
||
per-position acceptance is the direct measurement that the distillation
|
||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||
why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||
separate path. That difference is the whole trick:
|
||
|
||
```
|
||
--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
|
||
```
|
||
|
||
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
|
||
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
|
||
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
|
||
|
||
## The numerics trap (greedy text is NOT preserved by spec decode)
|
||
|
||
Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
|
||
`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
|
||
reduction order, argmax near-ties diverge, and the divergence cascades.
|
||
vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
|
||
The gufo-era "greedy == autoregressive text" law does not carry over; do not
|
||
chase text drift between spec and no-spec arms as a regression.
|
||
|
||
## Payload
|
||
|
||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||
byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh`
|
||
(`--verify-only` re-checks; no token on the curl command line).
|
||
- MTP source: `scripts/download-frank-large-qwen35.sh` →
|
||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||
the engine boots from exactly this tree).
|
||
- DFlash drafter: `scripts/download-frank-large-dflash.sh` →
|
||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||
- The serve script gates the image by REPO DIGEST (rootless podman resolves
|
||
tags from the network; the pin exists to boot exactly these bytes
|
||
offline), then config + index + exact shard count, then the spec draft
|
||
(either weight layout: single `model.safetensors` or a sharded index),
|
||
then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
|
||
Dankserver 2026-09-29 lesson).
|
||
|
||
## Client contract
|
||
|
||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
|
||
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
|
||
`models.yml` compat block for this provider sets
|
||
`reasoningContentField: "reasoning"`. Miss that and thinking silently
|
||
vanishes from transcripts.
|
||
- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
|
||
- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
|
||
serve script passes `--language-model-only` — the endpoint stays
|
||
text-only by decision; dropping the flag brings vision back.
|
||
- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
|
||
(temp 0.7, top_p 0.8) via `compat.extraBody`, and
|
||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
|
||
`chat_template_kwargs` form is what vLLM reads; a top-level
|
||
`preserve_thinking` is ignored there.
|
||
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
|
||
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
|
||
the env file with `set -a`.
|
||
|
||
## Ops
|
||
|
||
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
|
||
- The serve script sweeps stale container identities before starting:
|
||
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
|
||
- Container death → `podman wait` returns non-zero from ExecStart →
|
||
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
|
||
frank-primary lesson).
|
||
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
|
||
relinked unit; the deploy restores).
|
||
- First-deploy fact to record here: the boot-time KV pool line
|
||
(`Maximum concurrency for 262,144 tokens per prompt: N.Nx`).
|
||
|
||
## Rollback
|
||
|
||
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
|
||
payload + DFlash2 drafter, all still on the host. Restoring the tag's
|
||
unit/env/serve trio and redeploying returns the 27B endpoint.
|
||
|
||
# Citations
|
||
|
||
- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
|
||
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
|
||
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
|
||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||
[[frank-small-sglang-mimo-v26]]
|