serverAppConfigs/serverAppConfigs-docs/LLM/frank-dragoon-vllm-mimo-v26-bf16.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-dragoon on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)	The RETIRED frank-dragoon runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to frank-dragoon-gufo-qwen38-27b by Frank the same night.	llm frank-dragoon dankclaw vllm strix-halo mimo mtp spec-decode bf16	2026-10-01

frank-dragoon — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)

RETIRED 2026-10-01 night, one day active. Frank's call after the autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft) against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only; torchao config unwired). Back to frank-dragoon-gufo-qwen38-27b (Q4_K_XL, 1 x 262144). The measured content below stands as the record.

Identity

• Endpoint: frank-dragoon at 192.168.0.143:7770 (Dankclaw).
• Runtime: vLLM 0.30.0 in the digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 image (image ID 2d24a8c5a3fd, digest sha256:98c1f3ac9833… — ROCm 10.0.0 / torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM images are CUDA-only; this fork is the fleet's gfx1151 path. Plain podman run driven by scripts/serve-frank-dragoon.sh under a root unit (scripts/frank-dragoon.service, User=dank, Restart=always, ExecStop=podman stop -t 120, TimeoutStopSec=180 > the stop grace).
• Model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in full BF16 (17.53 GiB, 4 shards) — the same checkpoint frank-zealot serves as an INT8 W8A16 AutoRound conversion on the 3090, here unquantized.
• Draft: Qwen/Qwen3.5-9B (18 GiB on disk; only the 15 mtp.* tensors are loaded into the draft pass).

Why this shape (the roof math)

Decode speed for a dense model on Strix Halo is bandwidth-bound: BF16 weights x one pass per token against the LPDDR5x aperture. Measured 2026-10-01 on this host: 13.1 t/s is ~94% of the roof — the payload, not a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090 because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth wall is generating more than one token per weight pass: speculative decode is the performance feature of this endpoint, not a garnish.

The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B); the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running sequence. --max-model-len 262144 (the native window) with --max-num-seqs 4 at --gpu-memory-utilization 0.45 boots clean — the four-session contract the gufo shape ran, with the pool shared by the scheduler instead of carved per session.

Spec method: cross-model MTP — measured, not assumed

arm	prefill	decode	acceptance
BF16 base	17,876 t/s	13.1 t/s	—
BF16 + DFlash 8 tok	7,976 t/s	17.4 t/s	pos0 59% pos1 26% (token rate 14.8%)
BF16 + MTP 2 tok	7,380 t/s	19.8 t/s	pos0 71% pos1 44% (token rate 57.4%)

One session, temperature 0, 28,647-token prefill / 512-token decode, same client for all three arms (2026-10-01 bench window, frank-dragoon stopped per Benchmarking Policy). MTP wins (+51% vs +33%) and needs only 2 draft tokens — acceptance past position 1 is 44% and falling; DFlash's own curve decays 59/26/14/7, so 8 proposals are mostly wasted verification. num_speculative_tokens beyond 2 is UNMEASURED; do not raise it without a fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share the compute queues) — this endpoint's thinking-heavy traffic spends its wall clock in decode, which is the regime where that trade wins.

Why a Qwen head accepts on a MiMo target: MiMo-V2.6 is a distillation of Qwen3.5-9B that preserves the hidden-state geometry — identical 4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44% per-position acceptance is the direct measurement that the distillation preserved more than geometry. MiMo itself ships ZERO mtp.* tensors (verified in both the BF16 base and the INT8 conversion) — which is exactly why frank-zealot's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang reads the head from the TARGET checkpoint, vLLM reads the draft from a separate path. That difference is the whole trick:

--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'

The alternative drafter (z-lab/Qwen3.5-9B-DFlash, 6 layers, target_layer_ids [1,5,…,29]) stays fetched on disk; flip FRANK_DRAGOON_SPEC=dflash and repoint FRANK_DRAGOON_SPEC_DIR to use it. Its card's "vLLM PR #40898 unmerged" claim is FALSE — vllm/v1/spec_decode/dflash.py ships in 0.30.

The numerics trap (greedy text is NOT preserved by spec decode)

Measured: temperature-0 completions DIFFER across arms (sha 1318aeb6 base, 0208b45d dflash, 20c41c38 mtp). Batched verification changes the bf16 reduction order, argmax near-ties diverge, and the divergence cascades. vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving. The gufo-era "greedy == autoregressive text" law does not carry over; do not chase text drift between spec and no-spec arms as a regression.

Payload

• Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under /home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/, each shard byte-count + LFS sha256-verified by old/download-frank-dragoon-mimo.sh (--verify-only re-checks; no token on the curl command line).
• MTP source: old/download-frank-dragoon-qwen35.sh → /home/dank/models/Qwen3.5-9B/ (4 shards behind model.safetensors.index.json; the shards carry the odd HF asset names model.safetensors-0000N-of-00004.safetensors — the index resolves them, the engine boots from exactly this tree).
• DFlash drafter: old/download-frank-dragoon-dflash.sh → /home/dank/models/Qwen3.5-9B-DFlash/.
• Payload status 2026-10-02: all three trees moved to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ (59 GiB staged with the other retired frank payloads) and the fetch trio moved to old/ — the endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
• The serve script gates the image by REPO DIGEST (rootless podman resolves tags from the network; the pin exists to boot exactly these bytes offline), then config + index + exact shard count, then the spec draft (either weight layout: single model.safetensors or a sharded index), then /dev/kfd + whole /dev/dri (never individual renderD nodes — Dankserver 2026-09-29 lesson).

Client contract

• Alias: frank-dragoon (the id at /v1/models).
• Reasoning: --reasoning-parser qwen3; vLLM emits the thinking text in reasoning, NOT llama.cpp/gufo's reasoning_content — the harness models.yml compat block for this provider sets reasoningContentField: "reasoning". Miss that and thinking silently vanishes from transcripts.
• Tools: --enable-auto-tool-choice --tool-call-parser qwen3_coder.
• Vision: the checkpoint carries the qwen3_5_vision tower but the serve script passes --language-model-only — the endpoint stays text-only by decision; dropping the flag brings vision back.
• Sampling: request-side; the harness sends the Qwen3.5-family recipe (temp 0.7, top_p 0.8) via compat.extraBody, and chat_template_kwargs.{enable_thinking,preserve_thinking} — the chat_template_kwargs form is what vLLM reads; a top-level preserve_thinking is ignored there.
• Env hygiene: every tunable is FRANK_DRAGOON_-prefixed because HIP reads bare TEMP/HOST/GPU_MAX_HEAP_SIZE and the serve script exports the env file with set -a.

Ops

• Boot ~200–240 s warm; health 503 while loading (do not kill it).
• The serve script sweeps stale container identities before starting: frank-dragoon-mimo itself and the bench-window container vllm-mimo.
• Container death → podman wait returns non-zero from ExecStart → Restart=always self-heals (a clean exit-0 drain also restarts, the frank-archon lesson).
• Bench windows: systemctl stop frank-dragoon is enough (mask fails on the relinked unit; the deploy restores).
• First-deploy facts (2026-10-01, deploy #3): healthy after 230 s; boot log GPU KV cache size: 819,332 tokens, Maximum concurrency for 262,144 tokens per request: 3.13x — four sessions share the pool (the scheduler, not a carve). Live battery green: alias frank-dragoon, reasoning split into reasoning, tool_calls with correct args, and /metrics acceptance pos0 69% pos1 44% (54/86 draft tokens accepted), matching the bench window.

Rollback

Git tag frank-dragoon-gufo-q4kxl-20261001 — gufo v0.1.1 (built from source, scripts/build-gufo.sh stays on disk) + the Q4_K_XL Qwen3.8-27B payload + DFlash2 drafter, all still on the host. Restoring the tag's unit/env/serve trio and redeploying returns the 27B endpoint.

Citations

• Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
• Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
• DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
• Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
• Serve script: scripts/serve-frank-dragoon.sh; tunables: scripts/frank-dragoon.env
• Predecessor: frank-dragoon-gufo-qwen38-27b; sibling on the same arch: frank-zealot-sglang-mimo-v26